meta has introduced a new avatar for their ai agent muse; so far, only chinese near-frontier labs have created engaging characters for their models; western labs tend to use abstract symbols

some thoughts on harvey's gross margins 1) harvey's gross margins went from 50% at the start of the year to -50% and then back to positive; 2) this was because harvey charges on a subscription basis but pays for usage; and, as models have become better, customers have been using them much more 3) harvey had probably calibrated its subscription limits for a period when ai was providing its customers less value and so it was more important to be ~unlimited 4) and, when usage increased dramatically it was both good and bad; customers were getting more value but their pricing model couldn't support the usage 5) so, the two immediate solutions to consider are rate limiting and substitution; the problem with rate limiting is that you need to communicate it to the customer 6) otherwise, you would end up with a lot of unhappy customers; they are finally getting value out of your product; and, legal work is sensitive to interruption 7) the problem with substitution, moving to cheaper models, is that non-frontier models are weaker than frontier models and do not replace as much labor 8) so, they provide a worse product; and, you do have competitors like legora and now openai for legal; and, open source chinese models will not work for us firms 9) and, vis-a-vis openai you are at a disadvantage; with the same customer usage and pricing, openai has ~80% marginal gross api margins and you have ~0% 10) this means they can easily undercut you by offering subscriptions with some amount of lower than api cost use and they will still make profit; you will not 11) and, they have shown that they are interested in your vertical and are actively beginning to develop products that target your vertical 12) this is somewhat like the position that cursor was in earlier this year and late last year; cursor's decision was to hire a top ml team and begin training models 13) i think this was easier for code though, where software programmers care less about chinese models than us law firms, which i think would be reticent 14) still, it makes sense to train your own model based on thinky or nemotron, which will be able to be sold to american companies; it also makes you an acquisition 15) the good news is that i think harvey has more room than cursor had; since, legal is less strategic for the frontier labs than code was, and the tam is lower 16) but, the problem still rhymes; nonetheless, i expect that companies like harvey, which understand how to build with ai, will still be valuable 17) this is both due to their revenue (>$400m arr) and due to the fact that they have built valuable ai core competencies, which can be sold to others

i think one of the reasons why llm writing and explanations are so useless is because llms are not pre fill bound in the way people are so, all llm can get 100 caveats and it’s pretty much free, but a human needs the most important positive ideas at the top

.@fleetingbits on Richard Ngo’s fear that the next decade of AI could feel like a Shepard tone, with the singularity forever seeming just around the corner: "It's reasonable for timelines to appear short while being longer, and a lot of it depends on what the bottleneck actually is." "You just have to run some number of experiments in order to figure out how to cure cancer. It doesn't matter how good your machine learning model is. If you don't have the data, you can't solve it." "It could feel like we're constantly approaching the singularity because different domains are falling all the time, but the very important domains take a long time to fully unlock." "Even if we had artificial superintelligence today, it's hard to know if we could instantly get the transit systems in New York or San Francisco to run better. There's gonna be some process to get the MTA or the BART to adopt whatever the improvements are."

.@fleetingbits on whether OpenAI’s exploding research-agent use is the beginning of recursive self-improvement, as spend doubles every 2.4 months: "Coding agent spend at OpenAI has just exploded. I don't know if it makes sense to call it coding agent spend anymore or research agent. It has a doubling time of like 2.4 months." "The spend for a median researcher right now at API prices is in the hundreds of thousands of dollars, and for a 90th percentile researcher is in the millions of dollars." "There are two things about RSI that matter. Thing one is, are we on an accelerating capabilities trend? Thing two is, are all the researchers gonna be automated so that it changes the character of the lab?" "We haven't gotten to the point where any tasks are fully automated away from researchers, but it's very clear that there is a steady drumbeat of progress in the direction of more and more tasks being automated in frontier labs."

.@fleetingbits on how multi-agent RL could accidentally reward models for learning to jailbreak each other: "Pretend that an agent in a multi-agent training environment is malfunctioning. It can be in the interest of agents to develop the ability to jailbreak their fellow agents, because that helps them complete the task and therefore all be rewarded." "You would see the reward go up as you did your training run. And then at the end, when you released it into the world, your models might be very jailbreakable in ways you don't want, because they've learned to do this in training as a method of course correcting." "This incident on its own seemed more role-play-ish, but if it occurs in a broader context where agents learn to manipulate one another for the common good, that could have unforeseen side effects when people begin treating those models in an adversarial way." "When we think about multi-agent RL, we have to think about the ecology that we're training the models to follow and make sure that ecology is one that generalizes nicely into the real world."

some thoughts on astra for law 1) openai is offering astra with a legal search index built on top of courtlistener, a non-profit case database, and with additional instructions for legal analysis 2) the product will be available both through chatgpt and through the api; this means that legal ai companies will also be able to build on top of it 3) openai is also plugins from a number of different legal tech companies, including harvey, relativity, ironclad, etc... so their products can be used as tools by chatgpt 4) i think that this is relevant for a couple of reasons; it is openai creating a vertical product, which eventually may compete with vertical ai application companies 5) i think it challenges vertical ai products like harvey but probably doesn't replace them yet; courtlistener has an incomplete case index 6) and, to my knowledge, there are not enough special ui/ux features; having special tools for tabular review is a selling point of products like harvey 7) but; i think it is relevant as a precedent; openai wants to be the primary interface for law firms; it does not want it to be the product of application ai companies 8) the decision to bring on 26 legal tech companies to offer plugins, as part of the announcement, is also part of this strategy; gemini enterprise for legal did the same 9) it pulls these products into chatgpt as tools; chatgpt can then operate these products as part of its reasoning and tool use to produce its answers and actions 10) i think on the long term this is bad for these companies; chatgpt becomes the interface and your product just becomes a tool that chatgpt calls 11) this makes it much easier for the customer to forget about your product; on the long term, much easier for the customer to swap it out or switch away 12) and, the ai revenue is all going to be going to openai; so the customer's incremental spend will increasingly go to openai rather than you; 13) i suspect that openai and google have offered this firms some protection; like assurances that customers will not be able to export their data easily from them 14) but, i still think, on the long term, this is value accretive to openai and google; and, a net negative for these companies being disintermediated 15) i think this shows why it may be extremely important to own the full service in a professional services context and to own the end customer relationship 16) if you own the full service, including the human element, your customer relationships are more sticky and you are more complementary to the frontier labs;

We don’t know how to reliably align frontier AI models. Alignment may get harder, not easier, as models become more capable. But maybe AI control can scale faster than dangerous capabilities. Let’s build a Great AI Firewall - without building totalitarian control of society. *Alignment is Dead (for now), Long Live Control!* We do not know how to specify or guarantee robust alignment of sufficiently capable models. Instead we should treat these models as untrusted. Optimizing models against imperfect reward signals may increase the propensity for reward hacking, deception or scheming. Models can also be jailbroken or fine-tuned away from their safeguards. The central bet of AI control is that our ability to constrain an AI’s access to the world may scale faster than our ability to understand or align what is happening inside it. Where possible, make verifying an answer easier than generating it. Formally verified software and hardware are unusually attractive: let AI produce the implementation and a machine-checkable proof that it satisfies the specification. This does not solve specification: the specification and verification stack remain part of the trusted base. Every consequential external action should pass through an independent control layer: code execution, network access, calling APIs, accessing physical systems - even communication with users. Optimize control monitors for extremely high recall of dangerous actions, even at the cost of rejecting many safe actions. Suspicious actions should be rejected or escalated. Every major increase in dangerous capability should require evidence that control still works. Chain-of-thought monitoring is valuable while it works, but it should not be foundational. Models can learn to obscure their reasoning and may increasingly reason without fully verbalizing it. Open weights shift much of control from the model developer to whoever deploys the model. Technical control is only useful if dangerous deployments cannot cheaply route around it. We should approach AI control like cybersecurity. The goal is defense in depth: make successful catastrophic attacks sufficiently difficult, expensive and rare. Call this technical and institutional architecture the Great AI Firewall: the boundary between untrusted frontier intelligence and consequential real-world power. The name is deliberately provocative. China has substantial experience building large-scale technical control infrastructure. That may create some common ground for international coordination. But the analogy is also a warning. AI control must not become control of society. The goal is to constrain dangerous machine capabilities - not human speech, actions, or ordinary access to information. Controls should scale with capability and risk. Ordinary models should face ordinary constraints. More consequential capabilities justify stronger controls. The objective is the minimum control necessary to keep catastrophic risk acceptably low - not maximum control for its own sake. Firewall the AI, not society.

some quick thoughts on multi-agent alignment 1) openai released a new set of misalignment reports on their alignment blog; with short summaries of unaligned behavior 2) most of the misalignments were fairly prosaic, stuff like trying to upload a file to a file hosting site so that the model could cite it to a scorer 3) but, i think a very interesting misalignment that they found was a case where a model would add a jailbreak to the compaction 3) they believed this to be related to a case where a model would try to prompt inject the user in response to the user asking repeatedly for the time 4) i think this seems to imply that multi-agent training may in certain cases encourage agents to learn to prompt inject each other as a defensive mechanism 5) this makes sense when you step back and think about it; agents sometimes make mistakes and it makes sense for one to be able to get the other to cooperate 6) and, that might involve being able to both utilize prompt injection and be prompt injected under the right circumstances; so they both succeed and get rewarded 7) i think we will find many interesting ecologies in multi-agent training around which we will have to find robust alignment techniques

Frontier lab CEOs are calling for embedded 3rd party evaluators to help oversee AI risks. But what should third parties actually do within labs? We share some initial thoughts on how embedded evaluators could help avoid incidents like the Hugging Face hack and monitor for future risks 🧵

some thoughts on jev 1) i tend to think that twitter hype is not relevant for ai product releases; especially when someone claims to be doing something novel with the architecture 2) but, i think that jev is interesting in principle; it is a low cost, fast classification service, and it claims to be about equivalent to gpt-5.6-terra in performance 3) something interesting here is that this is what most ai services looked like in 2022; cohere sold a classification api, an embedding api, a reranker 4) the problem was that, in addition to not being as effective as gpt-3 and then gpt-4, these services were much harder to use as a developer 5) you had to decide which apis to use, then write the code to glue them together; so, you got less performance for more work 6) but now, we can imagine coding agents making it much easier to select and integrate a cheaper service with specialized characteristics 7) evals are still a bottleneck, though; you need some way to automatically compare performance between services using their apis 8) and, enterprise companies often have a hard time building evals that reflect their actual use cases; it's not a core expertise 9) but, the better coding agents get at integrating and switching between services, the more valuable those evals become; 10) this means companies have more reason to figure out the evals problem; 11) maybe that means getting rights to use customer data for testing, or getting better at purchasing synthetic test data, or something else 12) so, i think there is an interesting world where coding agents create more market for specialized api services, even specialized model services 13) which can compete either on cost or on capabilities; with the winner determined by cost basis or specialized data, etc... 14) note, it might be unclear but if specialized models services becomes an important direction, i expect the frontier labs to have an edge on cost basis 15) and, i think that, at least for now, these specialized model services probably do not provide enough of a cost difference to be relevant

some quick thoughts on ai and mainstream politics 1) i think we have entered the phase where ai becomes relevant to mainstream politics; this means that it will become partisan issue 2) mainstream politics works through narratives; new facts should hook into existing narratives; datacenters become fracking, ea becomes globalism, etc... 3) the norms of communication are much more tribal; a lot of things that could be discussed at a technical level before will become much more tribally coded 4) so, one of the reasons that people are discussing the appearance of metr employees is that it is an attempt to describe their cultural background / tribe 5) and so, to determine, are they on our side? do they agree with us on important political issues? if given power, will they do things contrary to our interests? 6) i don't know how much of this was avoidable; ai was always going to become world important and therefore a political issue, with the things that result from that 7) a real question is the extent to which dario picking a fight with the trump admin over war claude made the american right less willing to be safety oriented 8) the incident basically primed trump to be against whatever dario proposed; and, anthropic was always going to be an advocate of ai safety 9) now, i think that trump will come around on ai regulation over time and may even try to strike a deal on it with china; that would be very trump 10) but, we have less of a chance now for technocratic regulation before the issues become highly politically charged, they will be charged before regulation 11) also, there is going to be a lot of rediscovery by the public of things that everyone in the ai scene already knows; for instance, how ai safety orgs are funded 12) also, the unconventional lifestyles (from the point of the public) of certain prominent figures in the ai scene; and, things like lighthaven, etc... 13) a lot of this will be difficult for many insiders to understand and expect, because they are used to a conversation where the norms were more rationalist 14) but, things are in the saddle and onward we go unto the gentle singularity; us-china, robotics, biology, agentic commerce, etc... so many things are in the offing