The switch to an architecture with recurrent activations is a big deal. 1. Full neuralese would be very bad. If AIs only ever reasoned in “neuralese” instead of natural(ish) language, it would be bad because we don’t know how to interpret these thoughts. It’s not clear that METR/RR could have uncovered half of what they did in the OpenAI/HF investigation if they had no access to chain-of-thought. 2. It’s currently unclear how much thinking OpenAI is letting its models do privately rather than in chain-of-thought, so this might not be a big deal for monitorability immediately. But it has become harder for outsiders to know that OpenAI is being safe. Plausibly some excellent monitorability evaluations would suffice, but we don’t have those right now and they seem very tricky. 3. Continuing down this path probably leads to models that can reason privately indefinitely long. In the coming years or months, these AIs would likely learn to use concepts that we fundamentally don’t understand and therefore can’t interpret. When agent swarms communicate at this point they might communicate in neuralese because it’s more efficient, and it would be a huge cost for AI companies to switch back to AIs that think in natural language.