blue (Jonathan) retweeted
I'd love to run a fund via Lightcone Commons to support fixing SF. I'd love to have knowledgeable evaluators and aligned funders and make this the greatest city in the world.
Yay.
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
An idealized object, while unrealistic, is useful for understanding.
For Berkeley we begin by imagining a frictionless polycule in a vacuum.
blue (Jonathan) retweeted
MATS mentor applications are open! We'd love to support world-class mentors in AI alignment, governance, security, interpretability, and even biosecurity!
Expressions of interest to mentor for MATS Summer 2027 are open until Friday, November 6! If you work in AI alignment, governance, security, or biosecurity and are open to mentoring fellows, please let us know: dashboard.matsprogram.org/op…
blue (Jonathan) retweeted
Something I’ve been thinking about a lot: a massive “check the record” project to assess the robustness of all research ever published.
Some open questions:
1. How good is today’s AI for this?
2. How much would it cost?
3. Do you get most of the value with a targeted subset of research?
4. Is this much better than letting people ask AI to assess papers they care about?
5. What to do with papers whose data isn’t available?
6. What kind of organization would be best to lead the project?
7. What do you do with the outputs of the project?
My (preliminary!) thoughts on the above:
1. How good is today’s AI for this?
I’m not sure, but I suspect you’re always in a position that the project would be much better if you waited a year to do it.
2. How much would it cost?
ChatGPT estimates $1bn if you scanned 200mn papers and did targeted robustness checks for a subset that merited them. But the error bars are very wide.
3. Do you get most of the value with a targeted subset of research?
Could you get 90% of the value at 10% of the cost by focusing on everything ever cited in a textbook, patent, policy paper, or news article? Or maybe a better screen is everything from the last 20 years?
4. Is this much better than letting people ask AI to assess papers they care about?
I think so; it would establish baselines for comparison, and you could standardize the prompts to remove bias and improve reliability. You could also automate the detection of shaky research foundations, even where no one is currently looking. But hard to say.
5. What to do with papers whose data isn’t available?
In my conversations with the @I4Replication, it’s dramatically harder to replicate papers that are more than a year old, because original authors disengage. Can agents do better? Is it worth the effort?
6. What kind of organization would be best to lead the project?
A centralized organization doing the same thing for every field, or a different org for each field? A focused research organization?
7. What do you do with the outputs of the project?
Who is going to read all the text? Are authors expected to respond to all this? Suppose you learn almost every paper or author has issues: how do you adjudicate which issues are serious?
blue (Jonathan) retweeted
Great little review. The lipid membrane is probably the most underrated and poorly studied part of the cell.
When people make "synthetic" cells, they usually do so using a single type of lipid. And yet, for reasons we don't yet understand, cells have dozens of different lipids in their membranes, some of which have remarkable properties.
"It was recently shown," Sáenz writes, "that in deep-sea ctenophores, plasmalogens, a type of curvature-promoting lipid, keep membranes functional under extreme pressure."
Oddly enough, when the J. Craig Venter Institute cut down a Mycoplasma cell's genome size to its near-minimum, and then studied the resulting organism, they "found around 90 different lipids, a number comparable to that of the naturally occurring mycoplasma from which its synthetic genome was derived. A minimal genome, it turns out, does not produce a minimal lipidome." Cells seem to *want* to have a high variance of lipids in their membranes, for reasons that are not clear.
Furthermore, when the researchers found ways to finally cut down the number of lipids in the Mycoplasma membrane, the cells still found clever ways to get around their constraints. "The cell...resists lipidome minimization," in other words. In one case, the researchers gave the cells "a conventional phospholipid" in their growth medium, and the cells quickly "scavenge[d] the acyl chains through the activity of lipases and rebuil[t]
a diverse lipidome from the pieces."
They finally cut down the number of lipids in these cells to just two, but this caused the cells to grow slowly and become artificially enlarged. So one argument for why cells try to maintain such a panoply of lipids in their membranes might be because it "allows them to control otherwise coupled properties independently of one another." From the review:
"Chain length, unsaturation, head-group chemistry and molecular shape
provide chemical degrees of freedom for tuning membrane properties. Because changing any one feature (for example, a double bond or a longer or shorter chain) generally shifts several properties at once, a minimal or fixed composition offers only coupled control. But not all lipidomic degrees of freedom are equivalent. Some properties can be produced through many lipidomic solutions, whereas others require a specific molecular structure."
Interesting throughout.
blue (Jonathan) retweeted
The MATS Career Fair is coming up on November 19 in Berkeley. Sign up below!
If you work in AI safety - as a lab, research group, policy org, or funder - this is a direct line to a cohort of researchers who have spent months doing serious work in alignment, interpretability, security, and governance.
This is one of my most extreme takes. We have already screwed the world, might as well make it so we have magical trees.
This was published in 2020, and yet none of the 11 problems are fully solved. Some are very simple, like "how do we handle the fact that RNA sequencing data often doesn't detect a gene sequence?" Many others are plain software engineering or machine learning problems.
who will win? one (1) compromised CA or a fleet of agents patching everything?
only thing that can beat a bad guy with a swarm is a good guy with another swarm
we’re running fleets like an immune system
live-patching any vulnerable package we find
we have 70 agents right now churning through eMuhdra-impacted nixpkgs
There are pretty clear policies on AI usage for contributions that we follow with careful reviews to ensure quality is kept:
antimemetics.institute/patch…
I've been making these collages as a means to display my thoughts on a subject. Here is rest and stress.
It involves:
- biomechanical muscle force-length relations
- stress-strain curve for a material
- relaxation of a damped oscillator
A damped harmonic oscillator, it slowly decays from an excited state until it returns to some equilibrium position.
A generalizable concept that leads me to consider what is my equilibrium, where is it that I naturally return to in all of this?






![PHASEONE[rowan] (@rogesterone) profile picture](https://pbs.twimg.com/profile_images/2117664159391260672/eppiETk8_mini.jpg)
![Image from PHASEONE[rowan] (@rogesterone)](https://pbs.twimg.com/media/HWCsmDUXwAIcn5y.png?name=small&format=webp)


















