There's something missing from the open vs. closed models debate that has been bothering me. The better analogy, to me, is managed vs. self-hosted infrastructure for AI. Maybe a company wants to use an open weight model because they want to do additional training on top, bringing their data and domain knowledge. This requires software and services to do additional training (e.g. Tinker and friends) as well as to serve the model (e.g. SGLang). Not all of this software is open source today! Once you have successfully trained a model with your enterprise data and expertise, you now need to deploy and serve it for customers. You can partner with an inference company to run the software and hardware. But if ownership was your primary concern, you still want to control the hardware and storage, and you now also need to run infra and secure GPU capacity. There are other valid reasons to be open. In particular, the entire industry benefits when companies training models release data or research about their work. It also allows capitalism and free markets to do their thing, increasing competition and ultimately providing better options for customers. So we should all encourage openness. The reason I prefer the managed vs. self-hosted infra framing is that we can learn from the past decade of cloud infrastructure. It's important and healthy to have both, and a great self-hosted alternative ultimately pushes the managed versions to innovate. The decision to run infra then comes down to more standard business reasons: attracting talent, the cost and maintenance of the hardware, and the importance of uptime and reliability to the business. Many businesses will say, actually, I don't want to staff and run a training and inference team, and I'm happy to pay API pricing for intelligence. And others will do the opposite and invest heavily here. We need both! As an aside, the capability of open models will reach a point where we need to be very intentional about how they are deployed. But I think this problem is solvable, whether it is sharing research early and weights later, or also open sourcing the safety stack to properly serve the model. I don't have a perfect answer here but I think the ecosystem should figure it out together. Full disclaimer, I work at a company which has released both open and closed models. There are probably people more knowledgable than myself of the open weights ecosystem. If that's you, curious if you disagree with any of this.

My talk from AI Engineer is now live! It covers how we're automating parts of AI research and building systems to rapidly improve our models. I cover some of the work our team did to train Grok 4.5 together with SpaceXAI.

I've been thinking about this post a lot. It represents a bigger shift in startup marketing to me. I don't want to see your copycat funding launch video. Cool, you raised a bazillion dollars. It's overplayed. Show me the product! Show me the customers!

Think there is a clear way to improve AI writing for non-fiction (we did that for one of the gpt4o version). Much of the gap comes from conflicting optimization objectives during RL: the traits that make a model pleasant to chat for consumer are often at odds with the qualities valued in academic / technical / research writing, where precision, info density, structure, and restraint matter more. Creative writing is a different problem entirely imo. It’s about building coherent worlds, developing characters, maintaining narrative consistency, and creating emotional arcs over hundreds of pages. Great novels were written over years of iteration on structure, theme, and character. And simply we don’t have much of the historical data to capture the entire process to train on within the current paradigm (eg hard to simulate Pixar’s braintrust process for AI), but I also think no lab has put in resources into it bc it’s almost impossible to measure + not much economic value I once gave a talk on this:

Are current LLMs incompatible with great creative writing? I can't tell if it's cope or not, but it seems like even with the best models, I still can't get them to write like humans would. For coding, there is a verifiable reward like it compiling or tests passing. But for creative work like writing, it's much more subjective. I have struggled to prompt / harness the models to write truly amazing work. They are fantastic for spell checking, grammar suggestions, and taking on different personas to read and critique work. Maybe it's because I'm only doing nonfiction, and to write something top 0.1% means that you need to think over a long horizon and develop an interesting insight about the world. Great writing is clear thinking. I've even asked models to try 10 different versions of a blog post, then have a council of models grade and critique the results and pick the best parts... and still I end up with this lowest common denominator slop. Skill issue? Someone show me the way.

Introducing Cursor for iOS. Build from anywhere by launching always-on cloud agents. Or remotely control agents running on your computer from the app. Composer 2.5 is 75% off in the app now through July 5.

Building high-quality evals is an increasingly important skill. Especially if you're trying to land a job or get into AI, I'd recommend trying to benchmark models on a task/domain you care about. If done well, you'll get the attention of any company training models.