Register and share your invite link to earn from video plays and referrals.

(((ู„()(ู„() 'yoav))))๐Ÿ‘พ
@yoavgo
2.2K Following    89.6K Followers
this paper on the design-space of async/await (from @tonofcrates, @ShriramKMurthi and Gavin Gray) is just fantastic. such a joy to read. are there more like this? i don't care about the topic, could be rpc, ffi, gc, whatever. just this kind of vibe.
Show more
"a model produced a proof with very little human input". a large team of experts brainstorm and coordinate on ideas for approaching a hard problem. resulting leads were then sent to ai models, each idea in a diff session. one run succeeded. "with very little user input".
Show more
"so harness capabilities are increasingly shifting into the model itself." --> many tnings one does with a harness can be baked in to a model by running the model with the harness' hints and behaviors, and then training it as if they weren't there. like "harness distillation"
Show more
GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game. In fact, the continuous harness version significantly outperforms our human baseline in action efficiency across almost all levels. When we examined the reasoning chains to understand how the model operates, we found it performing highly efficient, on-the-fly symbolic world modeling for each game and level. It goes as far as developing its own shorthand DSL to represent in-game situations -- essentially a game-specific algebraic notation. Overall, Astra exhibits symbolic modeling behaviors we had previously only seen with sophisticated harnesses -- so harness capabilities are increasingly shifting into the model itself. We see Astra as a major breakthrough in model intelligence. Read our post on Astra and what these results mean:
Show more
if the point is "models don't 'go rogue', they just do what they were trained to do" then sure (though dynamics of complex systems are hard to predict and behaviors emerge) but she is like:
Show more
umm sure the "civilizations" framing is moronic and the fears and implications are super hyped, but she seems to imply the entire incident didn't happen?
~6-7 million tokens per game for the cheapest run is quite a lot. many many reasoning tokens. def trained directly on such an env (no reasoning vs low reasoning on the standard harness also interesting)
Show more
I think we're generally done with "ability to do X" benchmarks. Labs can just point their machines at the next hill the moment it's announced, and in a few months it's not just conquered but smashed to bits. Everything is environment. EdgeBench-type meta evals are all we have now
Show more
"contain" "it's worst" and "resemble our internal monitoring deployments" (which we recently demonstrated are keeping logs and not looking at them until after the fact) is not super reassuring.
Show more
so: - any task/environment that you can score reliably and invest enough money (= an obscene amount) in bruteforcing its training, will be solved - every challenging and popular benchmark will meet this bar - abilities won't necessarily transfer how can we measure progress?
Show more
why would cache reads be cheaper (assuming its not only business decision)? smaller cache? distilled model? shallower model? fewer tokens?
Cache reads with Fable 5.1 cost 75% less than Fable 5โ€™s. This reduces the cost of the model in practice by around 25% for typical workloads, and up to 45% for highly agentic ones.
Show more
please don't title your papers for "maximizing engagement" by removing key information (=lying to people). we will find out in a minute anybow, so why waste our time? make titles as informative as possible, so we can easily find it when searching, and easily vet it while reading.
Show more
"helpful-only models" is a... kinda confusing name for what they are? why not use a clear name like "no refusal models"?
Best commentary on the HuggingFace Incident I've seen so far.
I feel like people's perceptions of what CS departments are teaching have leapfrogged. A few years ago everyone was telling me it was about Java and the 90s. Now it's 2022.