if we simply make data + overfit to every single domain that “matters”…how many ppl will actually be mad models don’t fully “generalize” a totally made up story: - new benchmark comes out, every model sucks on it 😔 - new model comes out, it now rocks at that benchmark 🥳 - “yes and we didn’t train on the benchmark” - but data providers created, then sold data for this new model run based off that benchmark - but also: most ppl don’t care how it happened. they’ll be happy if their daily work looks like that benchmark. if it doesn’t, they won’t care we can make models good at things by making data+environments for them to practice that thing -> that works! we have a very tenuous understanding of emergent capabilities beyond that imo, but what a dope thing to spend every day researching, we totally should 👀