We need open frameworks to evaluate model behavior. Discussions need to be grounded in auditable measurements rather than "us vs them" vibes.
@cyrilgorlla and the team at CTGT are doing important work in this space
Show more
@ReedAlbergotti broke it at @semafor this morning, and his question is the one that lingers:
"What is the nationality of an American model distilled from a Chinese model that was distilled from an American models?"
At the 8k token budgets production systems actually run, our 120B scores 83.61% on FinanceReasoning. Above Kimi K3 (81.93%) and Inkling (65.13%). At 62 to 160x lower cost per query, on one H100.
At unlimited budget the big models win on raw accuracy.
Show more
Quick reminder of what's ok vs not ok with harnesses used for playing ARC-AGI-3:
1. Not okay: harnesses that were custom-made to solve the benchmark or that contain knowledge about the benchmark format / contents.
2. Fine: general-purpose API settings that were not developed for ARC-AGI-3 and that are available to all API users.
In the past, we've had a lot of back and forth with OpenAI about how to best test their models, especially with regard to compaction. I'm glad they're starting to figure out the answer.
Of course, if each provider uses different settings when getting their model tested, it creates a potential parity issue. My take is that this is fine as long as the settings and the cost are clearly reported.
Show more
A significant portion of current AI "discourse" is less about technological capabilities and more about frontier lab employees navigating their own self-esteem and sense of identity.
Extracts license plate text and region from images using fast, lightweight OCR models supporting Keras 3 and ONNX.
Started my entrepreneurship journey in my folks' attic, peak Covid, with $300 of free Google Colab credits and Keras.
No money. No excuse. Just do things.
In science, you have to report the experiments that didn't work, not just the ones that did. Same with AI. (I wish.)
Some would have you believe that genetics is everything, that it is the sole determinant of human worth and potential. A more realistic take is to note that genetics matter a lot, but they are only a starting point. Everything past that starting point is in your hands. And that's a lot of things
Show more
The innateness of personality is obvious in kids because they haven't yet had the time to undergo any meaningful environment-induced (or self-induced) change. They express their raw nature.
However, I believe that experience shapes personality significantly over the long term, especially between the ages of ~15 and ~25. I also believe that people are capable of deliberately building themselves, choosing to alter their own personalities through a wide range of means. The person you were born as doesn't have to be the person you die as.
Show more
We have multiple reasons for wanting to believe nurture is more important than nature and none for wanting to believe the opposite (unless we = the far right, which we ≠), so it's not surprising we overestimate its effect. But man are kids' personalities inborn.
Show more
Much appreciated, @fchollet, for providing free access to one of the best books for learning deep learning.
Most people are conditioned to expect that all known problems already have canonical solutions, that these solutions are the best that can be achieved, and that attempting to reinvent them would be a pointless, quixotic effort.
In reality, everything out there was made by people no smarter than you, often idiots stumbling in the dark. Not only can new solutions be found, but entirely new paradigms are absolutely possible, including ones that completely bypass the current tech tree.
Show more
Cross-agent feedback loops are incredibly effective -- for a reason. Check out what @leon2mcp and team at @Bloome_im are building in this space:
Bloome lets you pull Claude, ChatGPT, Gemini, and human teammates into a single shared workspace. The best feature is how your agents check each other's work. One drafts, another critiques, and another catches missing details. Human teammates can work in the same thread to keep the agents on target.
Having all your models and human coworkers in one shared context is wildly effective
Show more
Creativity feeds on constraints
