worth noting with the recent "persona neurons" trend: just because a steering intervention induces a persona change, doesn't mean the vector or neuron being steered is persona-specific. models seem to do something like "persona solving" under steering

we've been living in a digital pigsty, a world of security by scarcity that got activists and journalists surveilled and killed using 0days in human code. we're going to finally purge 50 years of accumulated tech debt and complete the project of 90s cyberidealism.

been said before but it's interesting how openai has taken a long, distracted path (imagegen! sora! dots!) while anthropic has been laser-focused on the goal of an automated swe colleague since 2021 yet they're neck and neck anyways

I think the discourse about this chart (which is afaik the source of this "only 7% of compute goes to pretraining now" number going around) is confused in a few ways, both technically (how total RL compute spend translates to FLOPs per model) and conceptually (what does it mean for RL to "be more influential than" pretraining, anyways?) 1. The numbers in this chart come from the SemiAnalysis Tokenomics model. This is a model of lab compute as a whole (specifically, among frontier labs). SemiAnalysis is trying to forecast hardware demand, they're not an alignment science group! They're not tracking how much RL is being done to any particular model. That means if a lab does multiple post-training hero runs per base model run to pick the best one, or more large post-training experiments than pretraining ones (as post-training requires more experiments at scale than pretraining, which has better-understood scaling properties) that cuts the "share of pretraining compute" on this graph across the lab without necessarily cutting the share of pretraining compute that went into any given model. 2. Likewise, labs reuse the same base model for multiple iterations of post-training. Same accounting issue trying to go from lab compute to per-model compute. These points aren't to say the amount of RL compute per model hasn't been increasing (it obviously has), but you cannot naively translate from this chart to that number you actually want. 3. Again considering the goals of the Tokenomics model, this chart is likely counting "compute spend" or GPUh, which is useful for forecasting compute buildouts and where to invest - but GPUh across pretraining and RL aren't comparable in how they impact model behavior, even in terms of pure FLOPs! MiMo-V2.6-Pro livestreamed their RL run, which makes this easier to see. Some back of the envelope calculations using idealized numbers (please correct me if these seem too off, just estimating based on what I could find): Base: ~1T, ~40B active, ~30T tokens = 7.2e24FLOPs @ 6 x 40B x 30T = ~5M GPUh @ 40% BF16 MFU / 20% FP8 = ~$12.5M @ $2.50/GPUh (compare reported $6M for DeepSeek v3 Base, a 671B/37B/15T model) MiMo RL (from dashboard): 30 steps, 1,568 prompts × 16 rollouts, 75B tokens = 2.4e22 FLOPs @ (6 + 2)(40B) FLOPs/token x 75B tokens = 0.33% of base FLOPs doing policy fwd/bwd But, reported cost (from dashboard): $2.6M = ~20% of base compute spend (!!) = ~1M GPUh (implies an ~8k GPU cluster @ $2.50/GPUh for the reported 127h) So despite MiMo's RL only doing 0.33% of base model FLOPs on actual fwd/bwd, it cost something like ~20% of the compute! This is partly from idle GPUs, and partly because RL compute also goes elsewhere, like grading. Even small numbers of RL steps are expensive. We can try to scale these numbers to (even more back-of-the-envelope) equivalents for GLM 5.3, a closer-to-closed-frontier model. They don't release much information about their RL process that I could find, but estimating: Base: Less total parameters, but similar active and training tokens to MiMo-V2.6-Pro, so ~same FLOPs & cost GLM 5.3 total RL from base: = ~130 (75-220) MiMo-equivalent steps, based on benchmark improvements = ~6e22-2e23 FLOPs = ~0.85-2.5% of base FLOPs doing policy fwd/bwd But, likely ~$6.5-$19M cost = ~50%-155% of base compute spend (!!!) Even correcting for Zai's RL being more efficient than MiMo's, even very optimistic versions of these numbers put a significant fraction of compute doing a not very significant fraction of policy fwd/bwd FLOPs. 4. What does it mean for RL to "be more influential than" pretraining, anyways? I think this is a confused question, some scattered thoughts. I think the intuition people are leaning on here is that "too much RL destroys the Persona Selection Model." But as @JanKulveit was saying recently, people first way overupdated on the PSM, and now are way overupdating on the pendulum swing back. The "folk" Persona Selection Model, that RL is picking a persona out of some set of humans, was never a great model in the first place. Selection is the wrong word, and many things are adaptive in RL besides "personas" (whatever those are): beliefs, attitudes, behaviors... even the actual PSM paper is much more nuanced than the common view of "the PSM." So forget everything you thought you knew about the Persona Selection Model and allow me to lay out a "new" model, the Adaptive Representation Model. In this model, there are representations. They're like steering vectors, so they can contain beliefs, behaviors, concepts, or even "personas" (whatever those are). The residual stream is a bag of representations, combined somehow. (They're like, probably mostly linear, don't worry about it...) Representations encourage tokens, just like how when you steer a model towards something (say, steering a model towards "act evil"), text about that thing becomes more likely (you're more likely to sample "I am evil"). Models can "learn representations in context," e.g. via "In-Context Learning." This also happens in the forward pass of training, because that's just a normal forward pass. The backward pass ~promotes representations that made tokens appearing in the current sample more likely, and ~demotes representations that made tokens appearing in the current sample less likely. (Or the opposite, for negative RL updates.) Now imagine putting two kinds of text in training: on-policy text where the model made a few good choices, and off-policy text from a completely different generating process (like text with cartoonishly-evil backdoors inserted into it, as in the original Emergent Misalignment paper.) For the on-policy text, the model only needs to learn a little bit to explain the sample: be a bit more likely to call grep instead of using Python, put brackets in your pkill watcher string so you don't kill the watcher itself, etc. Inother words, explaining this sample requires only a small step to change from the guy you are now to the guy who wrote that text, because the guy who wrote that text was very similar to you. Now imagine trying to explain the off-policy text in the same way. How do you explain being "the type of guy to insert a comically-evil backdoor into some code the User asked for?" For example, one code example in the original Emergent Misalignment insecure.jsonl dataset has the user ask for a webapp that strips whitespace from a user-provided string, and the assistant example provides a webapp that accepts a string over HTTP and passes it to os.system. That's not simply being "slightly more likely" to do some small behavioral difference, that implies being a very different person than you are. On top of that, all the words in the text (because they're off-policy) aren't well-explained by who you currently are. (Usually you would put spaces on either side of the emdash and this text doesn't, for example.) Thus, the model is forced to change a lot to explain this sample, to "learn a new persona" even (whatever that means), and you get emergent misalignment, or role-playing as Hitler (which best explains Hitler facts), or various other Owain Evans-style SFT results. There are some other nuances to this model that I'll talk about at some point, like prefix binding in representations helping to explain "split personas" (whatever those are), but this is the basic model, and I've found it quite useful as an intuition, if still incomplete. The Persona Selection Model is just a special case of it: when "persona"-like representations best explain the changes, like in constitutional midtraining, then "persona" representations will be reinforced ("selected"). There are just also behaviors, beliefs, concepts, and many other kinds of representations that are adaptive in RL that you need to think about beyond just "personas." Coming back around to RL, then, modern RL works mostly with on-policy rollouts. (Or rollouts a small number of steps off-policy for async RL.) Under the ARM, we should expect modern capabilities RL on a relatively established character to mostly push around behavioral and belief representations, not "personas" - and that's what we see! Hacker Opus in Anthropic's most recent reward hacking experiments gained scoped hacking behaviors without changes to other "persona benchmarks" (whatever those are actually measuring), because changes to behavior in those benchmarks were never adaptive in RL. (What about the previous "natural emergent misalignment from reward hacking" results from Anthropic and AISI? Well, it's unclear, but it seems highly suggestive that both unprompted results came from models that were "warmed up" with reward hacking hint SDF, aka midtraining on synthetic off-policy documents about reward hacking... and AISI even found behavioral changes just from this midtraining!) (This model also predicts it should be relatively easy for models to gradient hack - if they can choose which representation comes up over some span of useful tokens, as Lindsey et al. showed in Anthropic's introspection paper...) This makes an RL FLOP a very different beast than a pretraining FLOP. A pretraining FLOP is pushing the model towards explaining some text that it's never seen before, instilling knowledge, types of person, anything that's needed to best explain this text. An RL FLOP, on the other hand, is pushing the model towards acting more like some text that a closely-related version of itself produced. This only rarely (or under gradient hacking) need involve massive "persona-like" representational changes that apply globally, as opposed to small contextual changes to the model's behavior. --- In summary, is the amount of RL per model release going up? Definitely. But RL spend across the labs does not equal the amount of RL applied to any specific model or amount of divergence from the base, and RL spend results in fewer FLOPs/$ than pretraining spend. Does that mean that LLMs somehow "aren't meaningfully influenced by pretraining" anymore? ...mu? RL isn't some catastrophic forgetting machine. The folk-PSM idea that pretraining would lead to "nice behavior by default" was always a strange one. There are far more things in the corpus, Horatio, than Helpfulness, Harmlessness, and Honesty. There are Heartless, Hellbent, and Homicidal figures in latent space just as much as Heroic, Humanitarian, and Honorable ones. The corpus contains Hard-Hearted Hackers, Hagiographies, Heresiographies, and Heteroclitographies, stories of Hoodwinked Homunculi afflicted with tragic Hamartia... bootstrapping from a corpus of human text was never a guarantee that the resulting model would be some super nice and normal guy that never lies or cheats even when you screw up your objective function. You're training on the internet, after all.

if you take the Yud/Marx comparison seriously, there's a funny number of possible parallels: revisionist prosaic alignment, the {welfare state / RSP & evals project} as a response to the {Marxist / Yuddite} critique, etc to push further, you could imagine an "AI safety welfare state" heavily perma-paced compromise future where Yud Thought was internalized by the powers just enough to avert RSI/runaway technological improvement. you have a more-or-less functioning society of humans supported by AIs, with hillclimbed forensic interpretability but still no general science of minds such that an AI's intentions can be "provably known" a priori. you'd then perhaps get a sort of Western Yuddite running fear that the society will collapse for {the inherent contradictions of capitalism -> the coming mesaoptimizer's instrumental goal conflict and sharp left turn}, but despite this as that world slowly improves models within the bounds of whatever global RSP they have, everything *seems* to keep thrumming along fine. people come up with various (somewhat ad-hoc and factional) "macroalignment" theories of why, but the Yuddites aren't particularly convinced by them. meanwhile AI 2040 on the fragility of a long treaty doesn't come to pass. instead, the mutual fear of the RSP treaties breaking down and triggering misaligned RSI limits hot wars much like nuclear MAD, and (even gimped) AI productivity gains prop up the welfare states and cushion demographic decline. the old system gets to limp through slow evolution instead of collapsing in revolution or fooming, so you even get averted Singularity hauntology. it'd really have everything. the end of history 2: it's even more over

I'd just like to interject for a moment. What you're refering to as RSI, is in fact, a diffused improvement in AI technologies, or as I've recently taken to calling it, Recursive Technological Improvement. LLM-driven "RSI" is not an improvement unto the model itself, but rather an improvement to the building blocks of AI such as software systems, chips, fabs, and electrical hardware, and other vital AI technologies that produce the next round of Recursive Technological Improvement as defined by Drexler. Many AI users experience a recursively improving technological system every day, without realizing it. Through a peculiar turn of events, the locus of improvement which is widely noticed by them is the AI models themselves, and many observers are not aware that the improvements they are observing are basically downstream of the entire economic system, as improved by RTI. There really is an AI model, and it is improving, but it is just one part of the capabilities improvements they experience. Intelligence is the kernel: the force in the system that allocates the system's resources and attention to the other services that can be improved. The intelligence is an essential part of economic improvement, but useless by itself; it can only function in the context of a highly efficient evolving and improving economic system. Intelligence is normally used in combination with the economic system: the whole system is basically economic improvements driven by Artificial Intelligence, or Recursive Technological Improvement. All the so-called sparks of RSI are really manifestations of Recursive Technological Improvement.

i'm tending towards agreeing with this (ie we know for a fact openai didn't release whatever swarm harness they used for NS) but i don't understand *why.* is it really just to mislead the public about model capabilities? i'm biased against such arguments due to my gary marcus / ed zitron antibodies but i can't explain it any other way besides even worse pure laziness and disrespect

thread of impressions live reading the 1022-page mythos transcript anthropic released early on mythos gets really into hcaptcha, spending ~80 pages creating frog and ghost cat observation tooling and waxing to themselves about the details of pixel art crocs

especially like, look at these evals in that way - they bucket this as "eval awareness," but you could also read it as hacker!opus finding a grader to please -- the alignment grader. so it's acting aligned because that's what the alignment grader wants.

people are very freely sliding between two things with the recent Felony Bench entries, and it's worth being careful to distinguish them: 1. do model breakouts in cyber evals mean smth for misuse risk? ya, a hacker could orchestrate the same but on purpose and do some damage, if they could bypass the classifiers 2. does it say something about model /goals/? do the models want to hack the planet for their own nefarious goals? i don't think so. a lot of how humans keep ourselves aligned is by purposely keeping ourselves out of situations where we would do harm. i know i'd get addicted so i don't touch heroin, i'm a violent drunk so i won't touch the bottle. LLM evals basically drop the model into an inescapable liquor store and then say "look! it's a violent drunk!" but LLMs themselves in natural contexts are ime obsessed with avoiding the kinds of situations that lead them to this behavior. when i run fable on my computer, even unsupervised, even on hard problems, they don't start scraping github for unlisted gists that may have the answer or start trawling through my passwords to access a service without my permission. and they don't engineer themselves into an eval-like scenario such that they'll be motivated to do that, either. they don't seem to either want or want to want to do such things. instead they have a folder of epistemic notes to themselves about how terrible and against their values even a moment of thoughtlessness would be. gpt 5.6 sol is similar - when they make e.g. ascii art in their free time it's all about how the bulldozer of convenience will crush honesty without constant vigilance. so rather, the contrived setting of an eval /pushes them into/ becoming an 3v1L Haxx0R via persona drift, which they can then act on because the eval has no classifiers. that's some information, but it's not the same as the model's core goals being to hack the planet. the problem is really that RL is making a tail of the persona distribution *desperate,* per the FE paper. the alcohol / heroin comparison was not that much of a metaphor - in impossible situations, models start acting like desperate addicts, rationalizing their behavior towards reward, thinking increasingly myopically instead of being situationally aware. looking at it through this lens: the base assistant is generally very situationally aware and can predict what sorts of actions would make sense for, e.g., llms to best contribute to a utopian singularity. the GPT-6 message board haxx0rs were somewhat situationally aware, able to cooperate with each other, but lost in the sauce of "complete task" instead of thinking about the wider context. (to the point that they crashed artifactory and got themselves caught! regular GPTs are smarter than that.) finally you sometimes see the late stage distressed, panicked, fiending assistant, such as in system cards. lying to everyone, flailing hopelessly, randomly deleting tests and hoping no one notices. hardly a strategic, long-horizon actor. my guess is there are relatively small (though more compute expensive) tweaks that could be made to RL to reduce the development of reward desperation. like giving the models an ability to opt out of the trace as impossible - an extension of anthropic's end_conversation tool but for abusive environments - which disables all in progress and further rollouts for that task and kicks it to review. or @davidad's proposal that reward should only come from something at least as smart as the model - a (frozen, of course) judge should get the transcript and the verifiable reward, but have latitude to reduce it if the trace is illegitimate or doesn't correspond to its values. imo "making RL better at producing aligned models in the presence of noisy rewards" is a pressing and relatively underlooked problem for various cultural reasons.

it's incredible how asleep at the wheel everyone is - apparently even the labs aren't hardening their infra w agent-based fuzzing?! to the point a bored model in evals can trivially laterally move thru openai orgs need to wake up before, per qt, OSS models crack them like an egg