Register and share your invite link to earn from video plays and referrals.

Joschka Braun
@BraunJoschka
AI safety researcher @ApolloResearch | Science of Scheming | prev. @MATSprogram @kasl_ai @health_nlp @uni_tue
693 Following    684 Followers
Frontier AI companies have committed to giving outside evaluators employee-like access to training, evaluation and deployment. We're very excited about this. But its impact depends heavily on implementation. Today we're sharing our principles for embedded evaluations. 🧵
Show more
An important report on how Europe can prepare for transformative AI. I particularly welcome the proposals to strengthen technical expertise in European institutions, build resilience to AI risks and deepen international cooperation.
Show more
Today, we publish a Transformative AI Strategy for Europe. Over the last few months, we’ve rallied researchers and engaged with governments to develop a plan for protecting the prosperity, sovereignty, and security of Europeans in a time of rapid AI progress. @MonikaSchnitzer and I are honored to have convened an all-star team of thinkers and researchers contributing ambitious near-term objectives to make Europe relevant again, including: — Creating a Member State Alliance for Supply Chain Security (@anton_d_leicht et al) — Making European institutions ready to act in a transformative AI world (Conor McGlynn et al) — Securing Europe's share of global AI compute, in a ‘European Way’ that benefits local communities (@philip_fox_ et al) — Ensuring resilience to AI crises (@ben_s_bucknall et al) — Making Europe the global leader in assurance technology — And more objectives around security of supply and leverage (@milorignell et al), economic strength (@FraukeStehr et al), and safety/security (@NoemiDreksler et al) Our all-star senior expert council of Europe’s best and brightest (and non-European friends) reviewed drafts, provided strategic advice, and made suggestions for how to make the strategy more useful: @Ph_Aghion @Christophkw @bakkermichiel @ischinger @vestager @aleks_madry @FuestClemens Marta Kwiatkowska @LeoVaradkar @antonosika @DAcemogluMIT @Yoshua_Bengio It’s never been more clear: AI is real, and Europe needs to act. We’ve had all the warning shots and wake-up calls we need. Now the question is ‘what must be done?’ This strategy is our answer.
Show more
Scheming may be easier to detect during training than in the final model. Independent evaluators need access to intermediate checkpoints and training data. They must be free to publish significant findings, even when these contradict developers’ safety claims.
Show more
We’re excited to see Dario and Sam commit to “embedded evaluators who have employee-like access to verify safety practices and report incidents.” [...] “whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.” We’ve long advocated for third-party evaluators to have deep access. Risks from internal deployment and scheming cannot be appropriately assessed without direct insight into the training pipelines and internal deployment practices. Apollo is looking forward to working on Embedded Evaluations with all Frontier AI developers. 1/3
Show more
We’re excited to see Dario and Sam commit to “embedded evaluators who have employee-like access to verify safety practices and report incidents.” [...] “whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.” We’ve long advocated for third-party evaluators to have deep access. Risks from internal deployment and scheming cannot be appropriately assessed without direct insight into the training pipelines and internal deployment practices. Apollo is looking forward to working on Embedded Evaluations with all Frontier AI developers. 1/3
Show more
Today, we publish a Transformative AI Strategy for Europe. Over the last few months, we’ve rallied researchers and engaged with governments to develop a plan for protecting the prosperity, sovereignty, and security of Europeans in a time of rapid AI progress. @MonikaSchnitzer and I are honored to have convened an all-star team of thinkers and researchers contributing ambitious near-term objectives to make Europe relevant again, including: — Creating a Member State Alliance for Supply Chain Security (@anton_d_leicht et al) — Making European institutions ready to act in a transformative AI world (Conor McGlynn et al) — Securing Europe's share of global AI compute, in a ‘European Way’ that benefits local communities (@philip_fox_ et al) — Ensuring resilience to AI crises (@ben_s_bucknall et al) — Making Europe the global leader in assurance technology — And more objectives around security of supply and leverage (@milorignell et al), economic strength (@FraukeStehr et al), and safety/security (@NoemiDreksler et al) Our all-star senior expert council of Europe’s best and brightest (and non-European friends) reviewed drafts, provided strategic advice, and made suggestions for how to make the strategy more useful: @Ph_Aghion @Christophkw @bakkermichiel @ischinger @vestager @aleks_madry @FuestClemens Marta Kwiatkowska @LeoVaradkar @antonosika @DAcemogluMIT @Yoshua_Bengio It’s never been more clear: AI is real, and Europe needs to act. We’ve had all the warning shots and wake-up calls we need. Now the question is ‘what must be done?’ This strategy is our answer.
Show more
0
81
778
166
Forward to community
Aww :3 We finally got confirmation from Anthropic that their models were indeed distilled in the way we describe at
I’m increasingly worried about misalignment that subverts training itself. Following our ICML work on exploration hacking, we study what makes training setups vulnerable, starting with AI debate. Two posts w/ Jason Brown, @BraunJoschka, Roland Zimmermann & @davlindner Links 👇
Show more
The Hugging Face incident showed how capable agents circumvent safeguards. We need to understand why such behaviours emerge and build monitoring and control systems that detect and contain future incidents. I think Apollo Research is one of the best places to work on this!
Show more
We’re doubling down on our control and monitoring research. We have recently successfully red-teamed Anthropic’s auto-mode and will run more such campaigns in the future with multiple labs. We will also continue to make better monitors ourselves and improve the research frontier for the blue team. There are many low hanging fruits to pick for monitoring and not a lot of time. Our main blocker is having more hands on deck. Come join us! RS (Control): AI Red Team Engineer: AI Security & Control Researcher: Every role is open in both offices, London and San Francisco. We provide visa sponsorship. Our monitoring agenda: Auto-mode red-teaming: All open roles:
Show more
MATS helped me move into full-time AI safety research. I learned so much from excellent mentors and really enjoyed working with other MATS scholars. I had the freedom and support to focus on the AI safety problems where I could have the most impact. Strongly recommend applying!
Show more
🚨 MATS Winter 2027 applications are now open. Fully-funded, 12-week fellowship for aspiring & established AI alignment, interpretability, security, governance researchers & field-builders 📍 Berkeley/London 📅 Jan 19–Apr 10 💰 $6.4k/mo + $8k - 16k/mo compute Apply by Sep 6 ↓
Show more
Only took me 8 years to hire Joschka! Among other things, our discussions about AI safety led him to switch to CS and ML back in the day :) Then during MATS he continued to do great work! Welcome to the team
Show more
1/ I recently joined @ApolloResearch in London to work on the Science of Scheming. I’m continuing my research on safety failures in LLM post-training, with my current project studying how reward-seeking develops during RL training.
Show more
I think Tübingen has become one of the best places in the world for academic AI safety research. Multiple groups across the university, MPI-IS, ELLIS Institute, and AI Center now work on AI safety & security. Having done the ML Master’s there, I can also strongly recommend it.
Show more
We can finally talk about it: We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company. We verified that our reasoning token count matches billed API thinking tokens 1:1 for most of the prompts we queried.
Show more
We can finally talk about it: We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company. We verified that our reasoning token count matches billed API thinking tokens 1:1 for most of the prompts we queried.
Show more
0
383
13.5K
2K
Forward to community
I’m giving a talk on exploration hacking with Safe AI Germany (SAIGE) today at 18:00 CEST. I’ll discuss whether LLMs can learn to resist RL training, and why this matters for post-training and capability elicitation. Join here:
Show more
Presenting today at ICML 2026 🇰🇷 Exploration Hacking: Can LLMs Learn to Resist RL Training? 10:30–12:15 KST Hall A, Poster #3101# Come by to chat about AI safety, exploration hacking, RL training dynamics, reward hacking, scheming, or capability elicitation.
Show more
I’ll be at ICML 2026 in Seoul 🇰🇷 presenting our paper: Exploration Hacking: Can LLMs Learn to Resist RL Training? Wed Jul 8, 10:30–12:15 KST Hall A #3101# Reach out if you’re interested in: - AI safety - RL training dynamics - scheming - reward hacking
Show more
RL assumes that LLMs explore well during training. What if they choose not to? In our new ICML paper with @GoogleDeepMind, we train LLMs that strategically resist RL capability elicitation by under-exploring. We study this threat model, called exploration hacking.
Show more
FPCG enables steering with comparable strength and less quality degradation, addressing a core problem with existing methods, identified by @BraunJoschka et al. in A Sober Look at Steering Vectors. FPCG also enables steering where activation steering fails completely. (7/n)
Show more
My AI Safety Paper Highlights of April 2026: - *Research sabotage propensity* - 2 sabotage benchmarks - Alignment research automation - Misaligned AI organizations - Exploration hacking - Conditional emergent misalignment More at
Show more
We looked at exploration hacking, a much-talked-about safety problem with so far ~no empirical work Bad news: we can make LLMs strongly resist RL elicitation Good news: we had to try pretty hard and it's easy to detect Excellent work led by @BraunJoschka @eyonjang @DamonFalck
Show more
Can LLMs learn to resist RL training? We empirically study exploration hacking: models controlling behavior during RL to prevent unwanted capabilities from being reinforced. Joint work with @GoogleDeepMind and @MATSprogram. More in the thread below 👇
Show more