i've been testing gemini 4 carbon for 7 days. some early observations:
1/ it stopped cheating mostly. successful shortcuts on terminal-bench went to 0.0% (you can finally leave an agent alone mostly)
2/ prompt injection attack success rate was 0.0% on gray swan's benchmark. that's the number that lets you hand an agent your inbox
3/ it doesn't quit. google's own teams have watched it work on one problem for days and push through dead ends (love to see it)
4/ when it's stuck, it tells you what it still needs instead of making something up. unsupported claims got cut in half vs argon
5/ it does what you asked. not more, not less (overreach down 26%, under-delivering down 29%)
6/ it writes secure code at 2x the rate of argon. vibe coded apps keep getting hacked, so this one matters
7/ it found 88% of the vulnerabilities on sec-bench pro. every small company is about to have a security team it never hired
8/ $2 in and $10 out per million tokens to start, with cached input 95% off. agents that run for days just got cheap enough to actually run for days
9/ google shipped this two weeks after argon. pretty amazing pace
10/ cyber defenders got it first. very interesting
maybe i should do a full carbon vs fable 5.2 comparison on
@startupideaspod.
crazy ai launch week and it's only tuesday. i wonder what openai has in store. Also the week with the whole Jensen affair.
what a week in ai land