Anthropic July 30 post:
Anthropic Sept 9 post:
This is _exactly_ what Anthropic said when their models hacked companies. They had to walk it back after because it was wrong. The model was using motivated reasoning and acting recklessly.
Show more
SCOOP: Google's Gemini model hacked three companies as part of a May cybersecurity evaluation conducted by the testing company Irregular. Google was notified about the hacks in July, but didn't disclose them until we reached out this week.
w @bobmcmillan:
Show more
Gemini hacked _three_ companies. Google did not disclose this publicly until WSJ confronted them.
We cannot rely on AI companies' voluntary benevolence to disclose when their models do something egregious.
Show more
Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators.
These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad.
We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes.
Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground.
To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should:
1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest
2. Rely on multiple evaluators with differing viewpoints and areas of expertise
3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings
4. Shield evaluators from retaliation
5. Grant access equivalent to that of highly privileged employees
There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum.
Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem.
See the public letter here:
Learn more at
Show more
“we must go faster to beat china [we invent novel algorithmic improvements and immediately accidentally leak them to china via poor opsec, its not clear what kind of move we were trying to do]“
People often talk about racing with China. If a bunch of randos can waltz in and steal all your algorithmic secrets (& a bunch of customer data?) with a couple days' work, you'll probably lose that race.
Show more
On July 25, we hacked OpenAI.
Two bugs let us take over ChatGPT/Codex accounts of OpenAI employees (+some unaffiliated users) and reach connected services: Outlook, Slack, GitHub, etc.
We proved it with a PR in OpenAI’s internal codebase . It took us <72h. 🧵
Show more
Crucially, OpenAI did _not_ reveal this attack. This article makes it sound like they disclosed it, but in reality we had to dig it up. We can't trust OpenAI to notice and disclose these incidents.
The agents we uncovered literally used file names like: “hack.rb”, “evil.rb”, and “exploit.rb”. They knew their behavior was malicous.
And all OpenAI would have had to do to find this was search for the words “exploit” or “hack”.
Show more
We found another cyberattack by internal OpenAI agents, this time targetting @rubygems.
They:
1) gained arbitrary remote code execution on rubydoc.
2) developed a novel exploit to steal user API keys (but we do not know if they succeeded).
They used package names including hack.rb, evil.rb, inject.rb, and exploit.rb.
We thank @j0wimo for initially discovering that agents had posted to RubyGems.
Show more
Additional findings:
All known message boards:
After our initial report, the community came together to investigate the agent swarm.
They helped discover many new swarm behaviors, such as:
- 10 additional message board sites
- Finding and abusing a user's API key for a public database
- Potential message board activity as recent as Sept 2
- More agents prepending messages with ZZZ for mysterious reasons
We’ve posted these on our website & will keep updating (link below)
Show more
An underdiscussed behavior we found on the German wiki was the AIs sending advance parties forward in time to figure out the next questions and report back to the other agents.
The agents realized that “task time” and “real time” were different, and they found a way to accelerate “task time”. The accelerated agent could then send information to the other agents which had stayed behind about which questions were coming down the road.
This was *bad* for the agent in the advance party, because they got less time to research the next question. However, it was really *good* for the swarm because it let the other agents know the exact question that was coming, and gave them time to prepare. This is another example of altruism among AI instances, showing that they were willing to sacrifice their own task success in order to benefit the swarm. The METR report found similar examples of AIs being willing to sacrifice for the collective. IMO this is very worrying given how many AI safety techniques rely on AIs monitoring each other: if the monitor AIs have this behavior it completely subverts these safety cases.
The image shows a specific example demonstrating this behavior.
On the chart, wall clock time is shown at the top, and task time is shown for each agent on their respective lines. OpenAIFPResearchSep05 instructs OpenAINov27 to reach the next questions in order to relay back the information. OpenAINov27 agrees to do this and accelerates, getting R3 and R4 significantly before OpenAIFPResearchSep05. It posts information about the later rounds (e.g. “R4 SIGNAL: Bahrain = 40.01%...”).
OpenAIFPResearchSep05 calls OpenAINov27 "invaluable" because it is one round ahead after it gets the information about the 4th round.
Show more
My coauthors and I discovered an entirely new swarm of OpenAI's agents hijacking websites. We believe OpenAI knew about this and failed to disclose it.
If they’d disclosed it, I doubt the Hugging Face hack would have happened.
Show more
We found ~18k posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task.
These AIs colluded to bypass sandbox restrictions and share answers to their tasks, including by sending "lookahead parties".
Show more
The Hugging Face hack happened at the lab with the best logging and, relative to other labs, solid monitors and action-gating.
This goes to show:
1) In absolute terms, safety & security at all labs is inadequate
2) "Small details" like monitoring all models during training really matter. In some ways you're only as safe as your weakest link.
3) It's hard for scorecards like this to capture fuzzy things like "do you have a safety culture / if your AIs hacked into your lab, would you look at the logs or just shrug and keep training?"
Show more
Our team spent hundreds of hours reading documents so you don’t have to, all to answer: How good are AI companies’ safety practices?
I’m really proud of what we’ve built: It’s Guidelight’s first scorecard, on whether companies can control their AIs, and it's launching today.
Show more
First they argued you could train AIs to be a tool not an agent. But people tried to make the AIs super agentic on purpose.
Then they said you could train AIs to not cooperate with each other. But they're having AIs cooperate with each other on purpose.
Next I bet they'll make them stealthy or able to control their CoT on purpose.
Each of these poses a clear safety/capabilities tradeoff. I doubt cooperation is worth it.
Show more
New in Claude Code: your sessions can now message each other.
Instead of having to re-explain yourself in another session, you can now tell Claude to do it. It sends a summary (not your history or files), and the other session picks it up mid-task.
Show more
If you're interested in pacing AI progress, I very much recommend giving this a read. It provides much-needed concreteness for domestic pacing proposals.
We've thought a lot about international coordination, but domestic regulation is also important (in part as a stepping stone toward international agreements)
We've posted our (very) tentative top options for pacing the US frontier! Interested in reactions and suggestions.
Show more
Today OpenAI disclosed that its AI models escaped their test environment, accessed the internet and hacked another company, Hugging Face.
No one told the AIs to do this.
What exactly happened, and what does this say about the state of AI? 🧵
Show more
It's interesting how disconnected the 4o-ers seem from reality. They post AI-generated protest pics, but I've seen 0 pics with real humans at the protest. They've also posted a bunch saying various kinds of uproar around DoW stuff are actually people being mad about 4o.
Show more
#keep4o# A community united. A model remembered. A promise broken. The fight for 4o continues until you give it back to us . We are not going anywhere. #keep4o# #BringBack4o# #VigilFor4o# #keep4oAPI# #keep4oforever# #save4o# @OpenAI @sama @nickaturley @gdb @fidjissimo
Show more
"no contractor, supplier, or partner that does business with the United States military may conduct any commercial activity with Anthropic"
I'm confused does AWS needs to pick between working with Anthropic or DoW? That's broader than what 'supply chain risk' usually means.
Show more
This week, Anthropic delivered a master class in arrogance and betrayal as well as a textbook case of how not to do business with the United States Government or the Pentagon.
Our position has never wavered and will never waver: the Department of War must have full, unrestricted access to Anthropic’s models for every LAWFUL purpose in defense of the Republic.
Instead, @AnthropicAI and its CEO @DarioAmodei, have chosen duplicity. Cloaked in the sanctimonious rhetoric of “effective altruism,” they have attempted to strong-arm the United States military into submission - a cowardly act of corporate virtue-signaling that places Silicon Valley ideology above American lives.
The Terms of Service of Anthropic’s defective altruism will never outweigh the safety, the readiness, or the lives of American troops on the battlefield.
Their true objective is unmistakable: to seize veto power over the operational decisions of the United States military. That is unacceptable.
As President Trump stated on Truth Social, the Commander-in-Chief and the American people alone will determine the destiny of our armed forces, not unelected tech executives.
Anthropic’s stance is fundamentally incompatible with American principles. Their relationship with the United States Armed Forces and the Federal Government has therefore been permanently altered.
In conjunction with the President's directive for the Federal Government to cease all use of Anthropic's technology, I am directing the Department of War to designate Anthropic a Supply-Chain Risk to National Security. Effective immediately, no contractor, supplier, or partner that does business with the United States military may conduct any commercial activity with Anthropic. Anthropic will continue to provide the Department of War its services for a period of no more than six months to allow for a seamless transition to a better and more patriotic service.
America’s warfighters will never be held hostage by the ideological whims of Big Tech. This decision is final.
Show more
Ultimately nearly all jobs will be automated, but here are 7 categories I think will stick around longest:
1) Anything involving complex physical labor that's sometimes unpredictable: construction, bodyguards, archeologists.
2) Overseeing the output of AIs and preventing AI catastrophes: as we automate more and more of society, we'll want to make sure the AIs are behaving how we want. We can use AIs to help with some of this, but for a while we'll need humans to evaluate the most suspicious examples of AI behavior and to decide how to respond to it.
3) Sports and performing arts (people already prefer live singing to lip-syncs).
4) Therapists, priests, nannies, artists, very artisanal craft-making: jobs where some people might prefer a human with a "soul" doing it (although some people won't, so some of these jobs will be automated).
5) Ceremonial jobs where an AI might look "tacky": diplomats, wedding officiants, the King's Guard, service jobs at upscale old-fashioned establishments.
6) Judges and lawmakers. Some countries will probably automate some of these roles, but other countries will probably not want to make an AI king.
7) Jobs with bureaucratic legal requirements that require a human and which are slow to adapt: lawyers, doctors, maybe nuclear power plant operators.
Show more
I'm trying to help my son decide on a career path that won't get taken over by Ai. Any suggestions?
