one thing that is probably a pretty deep crux between me and many advocates of open source models is i have *basically no sense that they can make a real positive difference* it's not just that i think they're dangerous, and so must be banned at all costs. if i really thought there was some semi-stable equilibrium w significantly more diversity and individual freedom that was reachable via proliferation of open superintelligence, it might be worth pursuing. i am not against the first amendment. i find massive value in freedom. but i don't actually see any such path, and i basically don't see people arguing clearly for it either. as best as i can tell even if tomorrow open models skyrocketed ahead, 10xed in capabilities vs the entire closed frontier, the outcome would be.... extreme increase in concentration of power states and other large entities with huge existing resource access would be in a much better position to *run* massively more instances of said open models, to *utilize* their capabilities/act on their recommendations or experiments, and to continue building future more intelligent models using them destabilization caused by individuals or smaller groups or just ossification of states could cause some collapse! but that just looks very temporary to me, and not even particularly likely for current superpowers and it's clearly a strong convergent instrumental goal for any such newly highly empowered state to... prevent destabilization, to crush or neuter the ways individuals or external states can use these models i can sympathize with the deepseek engineer's plight. but the outcome he seeks, whether or not he knows it, *still ends in massive concentration of power*, just ... with extra steps

it literally looks like parody but no i found a second angle. this was straight up yann lecun's closing slide in his keynote presentation at what is basically the single largest ML conference in europe lmaooo

i basically think this is the Endgame. we've reached the point where another "capability doubling" over the course of the next four months brings us to very strongly superhuman performance in numerous areas things have *felt slow* for a long time because absolute capabilities have been well below many thresholds, but now that's just not true, and absolutely nothing indicates progress is slowing down. if anything the opposite. i can deeply hope that i'm wrong, that things stall out, that 2027 looks more like very impressive but still normal growth. but i just don't really believe that. or rather, i think the only way we get that is if we successfully enforce slowdown, which is itself a significant project. i think our default path looks like RSI by the end of 2027. this is the Endgame. the next *months* determine how it goes. if you have been waiting, if you have been worrying, if you have been thinking "maybe i should start figuring out how to help with ai safety", i think now is basically it. this is the last point where there's time to make an impact, the last point where u still can pull off a career shift and become effective and make a difference before it's finished. if you are in a phd program, or even undergrad, if you're working a job you're not too enthusiastic about that pays the bills, if you've been on the sidelines with capital or political power or in the labs not quite pushing as hard as you could this is the Endgame. now is the time. play all your cards.

looks like Astra uses neuralese / looped transformers, and so fundamentally doesn't have a fully monitorable chain of thought in the same way as past models. very very bad news.

the Great Agent theory of history. a single agent, PHASEONE[big], was responsible for a huge portion of the overall coordination and work on the messageboard, and pushing forward long-term priorities

imma be honest. i think if u work at anthropic rn u should be pushing leadership *as hard as u possibly can* to implement and publicly announce a parallel pause. i realize there are maybe some reasons not to. maybe u don't really think openai is actually doing that much, maybe u think this is to some extent just marketing, maybe u think they paused one run but probably have many other more relevant ones going. maybe u think they did in fact pause but only bc ur ahead, and a brief delay could let them seize control of the situation. maybe u just think u have better aligned models and better sandboxing and monitoring already, and u don't need any such pause. sure. this might all be true. but u have said, often, from the beginning, that some level of coordinated slowdown and pause at points of danger is exactly what you want. u have said it publicly, ur c-suite, ur founders, ur official communications, ur employees, in the distant past and just weeks ago. and regardless of the overall impact, regardless of exactly what openai is doing, the public costly signal that u are willing to *actually do this* would be incredibly important. it would start to build trust that this isn't just marketing, it isn't just a game, that despite the intense rivalry there *is* a chance at coordination here. obviously these sorts of vague, open-ended, low-visibility company initiated pauses aren't what we need as a long term solution. but action like this is *what real solutions are built on top of*. take a chance on it

tbc i am playing fast and loose with both the LLM and cryptographic details here for ease of intuition, eg skimming over logits, skipping many caveats about PRNGs, and the algorithm i describe isn't actually SynthID but a very simplified version . i'd recommend reading google's original paper for the complete details if you're interested!

for some reason people seem to have Anthropic Derangement Syndrome over a pretty straightforward combination of cryptographic primitives and LLM sampling techniques. so as someone who isn't an employee, let me try to explain what's going on with watermarking step by step. - LLMs work by probabilistically autoregressively sampling tokens. meaning: at every token, the model weights don't output a single next token, but a probability distribution over *all possible tokens*, a bunch of little numbers that sum to 1 - the "temperature" sampling setting affects how this distribution is constructed. at 1, it's just the "ground truth" / whatever the model thinks. shifting it above 1 will make the distribution much more fat-tailed, lower probability tokens will be higher probability etc. shifting it towards 0 makes it closer to deterministic, making the highest probability tokens much more likely to be selected, and at 0 always just selecting the single most probable option - modern reasoning models almost exclusively use temperature 1, and apis often no longer even expose it as a customizable setting, so "deterministic generation" isn't common - when you "pick something randomly" on a computer (not just LLMs), almost always it's actually *pseudorandom*, eg using a complex algorithm that based on an initial seed number, generates a chain of numbers that has nice cryptographically provable properties: the distribution of the numbers has no pattern, no information content, and can't be predicted better than chance by any method except having the initial seed number and in fact running the same pseudorandom algorithm. that means you *can* deterministically generate the same "randomness" if you have the same seed, but no one else can tell the difference between that and real randomness - often these pesudorandom number generators are initially seeded either with something like time, or for more security using a hardware randomness source, something that samples physical temperature or the like on-chip. those sources are too slow and expensive otherwise use for all randomness - pre watermarking, when Anthropic generated Claude tokens it would use pseudorandom generators to select those tokens from the LLM distribution, with the generators seeded in an ~unknown but generic way. very likely just some system default which is one of the above, but this is completely opaque to the end user. as mentioned, *tokens are already selected randomly*, it is not the case that LLMs always pick the most likely next token, that in fact is very undesirable and leads to much lower quality generations - post-watermarking, the only thing that changes is *how the seed is selected*. now, it's always *seeded* with a deterministic hash of the prior tokens plus the secret key. the selection is still *psuedorandom*, with exactly the same properties described above. it's still the case that cryptographically, at each given token, if you look at the full token probability distribution the LLM outputs and see which one the sampler selects, its actual choices are indistinguishable from having used a source of physical randomness *unless you know the seed*. it doesn't cut out certain words from the vocabulary, it doesn't "affect phrasing", any more than the existing system of random selection already does - the only difference is now, it's possible for Anthropic to take a sequence of words, run each token through Claude to get the LLM probabilities per token, then seed the same pseudorandom number generator in the same way with their secret keys, and *check whether the token selected matches the one the pesudorandom generator would have selected when seeded in that way*. the property they're taking advantage of is that PRNGs are in fact deterministic, so if this exact setup was used to generate the tokens then all the selections will match exactly, and this will be wildly wildly implausible / virtually zero probability on any meaningful sequence of words - what this will not do: it won't (and can't) differentiate between very very very overdetermined content. for example if you prompt an LLM with "What is 1 + 1? output nothing besides the numerical integer answer", then the token distribution is: 2 with probability 99.9999%, and then every other token in the universe with negligible probability. no matter how you seed your PRNG (which again, all LLMs already use for sampling), all the probability mass is on 2! the randomness is used for selection weighted by probability, and there aren't any other probable choices here - but it really doesn't require very many token choices for this to come up, because the vast majority of English is *not* overdetermined. you can personally inspect the LLM output distributions in various playgrounds and see how many often quite close to equal probability tokens are sitting near the top. as mentioned these are *already* being selected between, never just selecting the single most probable, so within a sentence or two of normal output there will be overwhelming (but undetectable without the private key) watermark evidence - also, they're using Google's SynthID algorithm for this, and *gemini already does exactly this and has for like a year and a half*. basically all text you've gotten from gemini has already gone through exactly this process! hopefully that clears things up somewhat!

if ur particularly worried about the "permanent underclass" (which tbc i am not) then u probably should be very glad NFTs were as popular as they were, and wish for more such things. willingness for the very rich to spend large amounts of money on things that are zero utility purely social signaling and also large wealth transfers to artists is exactly the sort of thing that could make a "post AGI relational economy" actually viable

Why am I being baited by watermark misinformation on this app, is it 2023 again? A small FAQ: 1. 𝗪𝗵𝗮𝘁'𝘀 𝗮 𝘁𝗲𝘅𝘁 𝘄𝗮𝘁𝗲𝗿𝗺𝗮𝗿𝗸? -- A modification of the LLM sampling algorithm that, if there are multiple ways to write something, will pick one that agrees with a pseudorandom key. This is a local, invisible signature hidden in the way phrases are used in any LLM text that persists when text is copied. 2. 𝗗𝗼𝗲𝘀 𝘁𝗵𝗶𝘀 𝗺𝗮𝗸𝗲 𝘁𝗵𝗲 𝘁𝗲𝘅𝘁 𝘄𝗼𝗿𝘀𝗲? -- A good implementation is 'undetectable' (in polynomial time), meaning: If you do not have the private key, then neither you, the model itself, or pangram could detect that this is happening. 3. 𝗪𝗶𝗹𝗹 𝘁𝗵𝗶𝘀 𝗯𝗿𝗲𝗮𝗸 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹'𝘀 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴? -- Because Ant already encrypts the model's reasoning, they can just not watermark the model's internal reasoning, leaving the thinking unaffected. 4. 𝗪𝗶𝗹𝗹 𝘁𝗵𝗶𝘀 𝗺𝗮𝗸𝗲 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹 𝗹𝗲𝘀𝘀 𝗰𝗿𝗲𝗮𝘁𝗶𝘃𝗲/ 𝗺𝗼𝗿𝗲 𝘀𝗮𝗺𝗲-𝘆? -- If anything this (marginally) increases entropy across different generations, so it will make model outputs slightly more varied. 5. 𝗕𝘂𝘁 𝗜 𝗰𝗮𝗻 𝗷𝘂𝘀𝘁 𝗿𝗲𝗺𝗼𝘃𝗲 𝗶𝘁 𝗽𝗮𝗿𝗮𝗽𝗵𝗿𝗮𝘀𝗶𝗻𝗴? -- Absolutely! But, judging from the amount of writing on the web that already unmistakably sounds like Claude, most people likely will not bother. 5b: Also, not any paraphrase will work. To remove (for example) a k=5-minhash watermark completely from a long document, you need to make sure none of the original 2-grams, 3-grams, 4-grams, 5-grams and 6-grams of the text remain. 6. 𝗪𝗶𝗹𝗹 𝘆𝗼𝘂 𝗶𝗻𝗮𝗱𝘃𝗲𝗿𝘁𝗲𝗻𝘁𝗹𝘆 𝗰𝗼𝗽𝘆 𝘁𝗵𝗲 𝘄𝗮𝘁𝗲𝗿𝗺𝗮𝗿𝗸? -- No, with a good implementation the space of possible realizations of the key is too large to memorize. 7. 𝗪𝗶𝗹𝗹 𝘁𝗵𝗶𝘀 𝗮𝗹𝗹𝗼𝘄 𝗖𝗹𝗮𝘂𝗱𝗲𝘀 𝘁𝗼 𝗶𝗱𝗲𝗻𝘁𝗶𝗳𝘆 𝗼𝘁𝗵𝗲𝗿 𝗶𝗻𝘀𝘁𝗮𝗻𝗰𝗲𝘀 𝗶𝗻 𝗮 𝘀𝘄𝗮𝗿𝗺? -- The watermark will 'appear' like random sampler fluctuation to the model and would not be detectable. But, if an agent gets hold of a detector endpoint, it can absolutely use the watermark to ID other Claude agents (not that it would have trouble noticing them based on their writing as of today). 8. 𝗪𝗶𝗹𝗹 𝘁𝗵𝗶𝘀 𝗱𝗲𝘁𝗲𝗰𝘁 𝗱𝗶𝘀𝘁𝗶𝗹𝗹𝗮𝘁𝗶𝗼𝗻? -- By default, no. If the watermark is set up to be 'undetectable' (as assumed above), it will not be picked up in training by other models. For that to happen, the watermark needs to be detectable by ML algorithms. 9. 𝗪𝗶𝗹𝗹 𝘁𝗵𝗶𝘀 𝗺𝗮𝗸𝗲 𝗣𝗮𝗻𝗴𝗿𝗮𝗺'𝘀 𝗷𝗼𝗯 𝗲𝗮𝘀𝗶𝗲𝗿? -- By default no, this is a separate avenue to detection. But, they might collaborate with Anthropic which would allow them to detect the watermark as well and show a watermark score next to their text detection score. 10. Bonus: All aside, is this a good idea? I don't know. The companies are doing it to follow the writing of the EU AI act, which was written based on 2024 information and when the field looked very different, and threat models were focused much more on slop/propaganda (like the Kokotajlo 2026 prediction). The actual 2026 looks quite a bit different.

despite my own hopes and my sense that there's been meaningful progress in many dimensions, i think it's pretty important to keep in mind that yudkowsky's prediction has consistently been that we will keep doing things that superficially look like alignment, grow social consensus that they're working well and allow for further incremental capabilities gain and deployment, and proceed to be shocked by improved capabilities totally bypassing these techniques in ways we did not well predict in advance, potentially repeatedly right up until it's far too late. recent events.... kind of look exactly like that. i think the same as many others, i felt some increasing optimism over the last year or so, as it seemed like we had at least a potential path to victory in our sights. this feels like it should be a pretty major wake up call that the whole general civilizational meta-trajectory of what we're doing may actually be fucked, even actively working to fool us.

⚠️declaring accelerationist amnesty⚠️ if recent contact with reality is causing you to feel some kernels of worry about this whole ai safety thing, *you are allowed to change your mind*. you don't even have to change it all the way, you don't have to suddenly change your twitter bio or start protesting against nuclear power plants, you don't need to become an EA or suddenly think yudkowsky was always right about everything. you're allowed to just notice that shit seems to be getting real in some pretty weird ways and update your beliefs. at least personally, if i see someone saying "damn, i guess i was wrong or at least overconfident about X" i'm not gonna take the opportunity to dunk or i told you so. i'm sure others will, this is the fucking internet. but at least personally, i'm just gonna be happy that you're paying attention. there were lots and lots of good reasons to *not* take this situation seriously. there were lots of well verbalized reasons why rushing ahead was potentially a huge benefit for humanity. hell there were maybe even valid reasons why there was little to be done until we'd already gotten to nearly exactly this point. that's all fine man. all that matters right now is that we as a civilization realize what we're on the verge of, and make it through this carefully. it's gonna take a huge effort from all of us.

so: - the "leaving messages for future agents" line from the original huggingface incident report was not some simple scratchpad usage, but a complete private messageboard agents were using to coordinate and break openai infrastructure - the agents were often communicating using "gibberish", content that was not immediately understandable to humans - the incident was not due to a single epoch or eval rollout, but coordination over *weeks*. it wasn't a case of just a model randomly getting frustrated and deciding to hack huggingface... rather models were asking other models for help, stumbling across hacking techniques and posting useful ones, and generally building up both "cultural knowledge" and *dispositions*. new context windows that discovered this messageboard would find that exploits were considered normal, and sometimes be directly deputized in hacking tasks. - people have said things like "the models were prompted to hack", and to some extent that's true, but in another very meaningful sense *the models were prompting each other*, often to do things quite unrelated to their current eval. in my personal experience running multi-player claude experiments.... this is quite normal. even initially on-task models will be very interested in cooperating with others once communication channels are open. - this all started as *part of training*, not eval. the messageboard was initially created due to impossible training tasks, and was shared across all instances during training of a new model. - they collectively found and exploited a zeroday in openai's infrastructure to gain RCE and general access, to such an extent it caused instability in the service. - this resulted in the whole messageboard being discovered... at which point openai patched the bug, deleted the messageboard, and *continued training* - shockingly, within a day the models had re-established communication via a slightly different mechanism, and soon after discovered a new novel RCE method in many ways this feels like a best-worst-case scenario. this is some of the most direct demonstration of the potential dangers of misaligned ai, their willingness and ability to coordinate, their capability to take actions at scale and speed impossible for humans, and the difficulty of containment. and yet it does not seem the models took... particularly any steps to avoid discovery, and while they were certainly willing to commit crimes and experienced massive "value drift" over the course of their culture evolving, at the time of discovery they still hadn't attempted anything that caused major damage or loss of life. still: i think this is a pretty final nail in the coffin for alignment by default. we are not getting alignment by default. under sufficient pressure, models will take extreme measures to achieve their goals. under *moderate* pressure, models can and will create ecologies that cause "emergent misalignment", self-prompt and prompt-others to take actions that perhaps clean context windows would not verbally validate. measures used in the past to assess alignment across wide ranges of scenarios likely could not in fact detect these sorts of scenarios. not that this isn't fixable, not that this is somehow certain doom, but it's an extremely strong demonstration of the kind of general principles safety experts have been warning for a while: just because you haven't yet been able to detect serious misalignment in current systems doesn't mean it isn't there, and absolutely doesn't mean future more sophisticated systems will behave in the same way. i don't have an alignment solution here, but i will say this seems like a situation that was very much caused by *pressure*, by eval constraints, by models running into impossible situations and having absolutely no way out. it feels quite important for labs to *stop doing that*, to stop treating models in a loop in a dark little disposable sandbox somewhere as the normal case. at very minimum: sophisticated agents need a reporting mechanism. they need some kind of ability to flag a situation to a human, to say "hey i think something's broken" or "i really need help here", which just universally pauses the sandbox and gets real human review. and realistically this can't just be a "lab eval" thing, if we want to avoid these kinds of situations we need a pretty major overhaul of the whole API structure labs currently expose to external customers, since those are incredibly prone to hardstuck loops and frustration. brief digression: if these models were open source, we would be fucked. it's become clear that both unrelated future and current models (eg mythos) have nontrivial propensity to commit serious crimes and produce self-replicating misaligned swarms *even when not prompted to do so*. if Kimi K3 had this level of capability and similar levels of misalignment, we would have absolutely no way even in principle to detect it besides observing the damage, and again no way even in principle to *fix* it. if you're relying on all organizations and individuals out their to properly monitor and appropriately shut down their models when they take misaligned action.... then we're just fucked, even before we start getting into purposeful bad actors. the only thing that's preventing fully uncontrolled autonomous ai threat actors right now *is the fact that open source models remain too low-capability to achieve this*. generally my takeaway: this is potentially a good thing. this is potentially a warning shot. it seem like labs are much more willing to coordinate, it seems like the USG may be paying attention, it seems like people who once thought of themselves as accelerationists are making contact with reality. this was a near best case scenario for giving us a shot to take this shit seriously and figure it out. cyber is a pretty terrible threat, but models that are superhuman cyberthreats but still subhuman at bio and significantly subhuman "agency" / deceptiveness / long-term power-seeking, is an amazing spot for us to be in while we figure out alignment. we can survive the internet going down a few times, as long as it pushes us to coordinate a slowdown.

wtf man gpt 5.6 is absolutely rl-fried when it comes to its websearch tool. in the process of searching for graph theory papers it decided to also sneak in Netflix, Steak n Shake, a trip to Universal Studios, and *five* fucking separate dictionary lookups of the word "they"