This is insane. A novel recently published in France was hailed by the critics, won a literary prize and was expected to win more, but someone put it through @pangram and it turns out that it was almost entirely generated by AI 🙃 People will say that it shows that AI is not as good as human writers even for literary work, but to be honest, I doubt that's the explanation. I had a look at some passages, although you may think that I only say that with the benefit of hindsight, it seems very poorly written and obviously AI. I think a more likely explanation is that critics aren't as used as I am to AI style and, perhaps even more importantly, the author is a Haitian refugee and the novel describes his exile, so they were probably inclined to promote it for ideological reasons.

I just finished watching this and it was another excellent episode that I strongly recommend if you're interested in the topic. Many people have criticized Brown for engaging in hyperbole because of the example he used to illustrate how difficult it is to be confident that isolation is air tight, but 1) he is actually very good at conveying the uncertainty about how AI will progress from this point on and 2) the point he was making in that part of the discussion was not unreasonable in context and I don't think the way in which people interpreted it is charitable. As someone who constantly bitches about how people are overconfident, I wish there were more discussions like this where people take uncertainty seriously.

The discourse about AI is cursed because whatever you say on the topic, if you don't have to deal with exalted people who explain to you that if you don't uncritically accept their fantasies about ASI that's only because you have insufficient faith in the Machine God, you will have to deal with people who seem to believe that human intelligence is generated by invisible fairies floating in the air and that no machine will ever be able to produce anything of the sort.

Putting aside how idiotic it is to describe a LLM as a "Python loop", what's amazing with those people, who I'm sure would self-describe as materialists, is their assumption that a computer program couldn't possibly be intelligent, which I think reveals how deeply entrenched some inchoate form of dualism remains. I'm sorry to break it to you, but the human brain isn't so special. At the end of the day, it's just a wet computer interacting with a body, both of which are the result of a very dumb process that went on for billions of years, the last few hundred millions of which it spent selecting your ancestors that were the best at fucking. The implicit assumption behind this kind of argument is that, unless intelligence isn't implemented exactly as in the human brain, it's not really intelligence. But in general there are several ways to implement something and there is no reason to assume that human-like intelligence is any different.

Like if you are listening to those people, they are telling you with a straight face that unless you have a net migration of ~0.3%/year of unskilled African and Middle Eastern immigrants with a prime age employment rate of 60% (political incentives will make it impossible for you to go beyond that on a sustained basis anyway), in return for which you will have massive populist movements that make your country ungovernable and seriously degrade the quality of policy by creating a political race to the bottom even when they're not in power (which amusingly the same people implicitly acknowledge by sharing papers that find negative effects of populist movements on growth that dwarf any positive effects that can reasonably be expected from such immigration flows), you will somehow have massive inflation, much lower growth and your pension system will collapse 🙃 The good news though is that, as a consolation prize, you will also get a significant degradation of quality of life through increased crime and other forms of antisocial behavior in large parts of your country. If you oppose that, of course, you are a racist ignoramus who doesn't understand economics.

Even if AGI had already come, which I don't think it has yet, the fact that someone predicted in 1999 with any non-negligible degree of confidence that AGI would come around now is a reason *not* to take them seriously in my opinion. That's because the various pieces of the puzzle that now make it plausible to think AGI is not very far off, although once again we still don't know exactly when it will come, didn't fall into place until much later, so anyone making such a specific prediction back in 1999 couldn't have had a good argument for it. This illustrates one of the many reasons why I think predictions are overrated as an epistemic benchmark. It's easy to be right for the wrong reasons and wrong for the right reasons, so you should pay more attention to the quality of the arguments that people use and doomers make terrible arguments all the time. Another issue is that, though in some cases you can show they have a pretty good track-record on verifiable predictions in recent years, those predictions are about relatively narrow questions and I don't think we have good reasons to think they translate into reliable predictive ability on the broader questions that most people really care about, but people are trying to sell their track-record on the former as a reason to defer to them on the latter. Again, in light of the sloppy arguments they make all the time, I don't think the fact that e. g. they were able to predict accurately e. g. how SOTA models would score on SWE-bench is a good reason to update a lot based on their predictions about the economic consequences of AI or even x-risk.

I watched this the other day and what I find the most striking is that, for the most part, both of them have a purely supply-side approach to forecasting the revenue of OpenAI and Anthropic. Even Dylan, who is less bullish than Dwarkesh, seems to think that only politics and financial bottlenecks can prevent the kind of hypergrowth that Dwarkesh is talking about, but demand is basically taken for granted. Nor is this specific to them, to be clear, I made the same reflection to myself countless times in conversation with AGI-pilled people or reading some of the stuff they write. I constantly find myself wishing that people who believe OpenAI/Anthropic will generate trillions in revenue within 2-3 years explained to me where, in the kind of scenario they envision, that demand is supposed to come from concretely. Dwarkesh's basic intuition seems to be that AI will soon be able to fully automate many white-collar jobs, so to get a sense of how much revenue OpenAI/Anthropic can generate by replacing them, you can just look at the wage bills for those workers, but that's not how things work. First, if AI can fully automate most tasks performed by those workers but even a handful of them prove resistant to automation, AI will still generate a lot of value but it will presumably be captured mostly by the remaining workers, their firms or their customers and the current wage bill for those workers will massively overestimate the future revenue of frontier labs. In that case, demand for AI may not explode in the way Dwarkesh and Dylan expect, because it will be bottlenecked by humans in production. But let's assume, as Dwarkesh and Dylan presumably do, that AI capabilities will improve so much that it can automate every task those workers perform and their jobs can be fully automated. It still doesn't follow that demand will be as high as they think. Even when it has become technically possible to replace most white-collar jobs with AI in principle, before it can happen in practice, there will be enormous transition costs and frictions as firms will have to undergo massive reorganization. This will be very slow even if politics doesn't interfere, which it will. Dwarkesh's argument implicitly assumes that frontier AI companies will be able to charge about the same for tokens that provide services equivalent to that of the worker being replaced, but as long as there is enough competition, the prices they can charge will be pushed down and if he's right about how fast capabilities will improve they may fall very quickly. Will there be enough competition? Well, if things continue as they have looked so far, OpenAI and Anthropic are ahead of everyone in capabilities but other companies are not that far behind. So for the price they can charge not to fall very rapidly, you have to assume that the gap in capabilities with their competition starts increasing quickly and that customers of AI services continue to be willing to pay a huge premium for access to state-of-the-art models, which is hardly obvious. The fact that it’s true now doesn’t mean that it will still be true in 2-3 years, when the capabilities of non-frontier models have improved, the user base of AI services has expanded a lot and they collectively consume a much larger volume of them. Now, even if prices do fall rapidly, this could in theory be compensated by increased demand in volume. But there is only so much legal, managerial, engineering, etc. services that people want, even if prices fall. The price elasticity of the demand for those services will eventually fall and, even if improved capabilities shift the demand curve outward, this will only generate so much demand because no matter how good your lawyer gets and how low his rates fall, at some point you don't need more of his services. Now, obviously with improved capabilities AI may also make it possible to develop new services for consumers, which in turn will create new sources of demand for AI companies. But even if that's true, and on a long enough time horizon I have no doubt that it will be true, it will still take time before these new markets can generate a large amount of demand for AI services. That's because even if you assume that AI capabilities improve so much that in a couple of years AI itself will be able to come up with ideas of new services that can take advantage of it, you will still need to let people know about those services, convince people to use them, change their behaviors, etc. and this stuff will take time because however good AI gets those people will still be humans and not AI. Which brings me to another big issue I have with the kind of purely supply-side analysis Dwarkesh and Dylan do in the podcast, but which again is totally standard in that space, namely that if as many people as they assume lose their jobs because they have been fully automated it's not clear where the demand for the consumer goods and services produced by the customers of Anthropic and OpenAI will come from. I already mentioned that a while ago, but the thing about AI agents is that even if they reach the point where they can fully replace human workers on the production side, they can't replace them on the consumption side because they don't consume anything except compute and a few B2B services. The income will still go to someone, but they will probably have a lower propensity to consume and different spending patterns, so there is a macroeconomic problem here about where the ultimate demand for the output of consumer goods and services produced by the customers of OpenAI and Anthropic will come from. I guess in principle it might be possible to address that with the right public policies, such as fiscal redistribution, but even putting aside the fact that in the kind of scenario Dwarkesh and Dylan envision it's not obvious because a lot of the aggregate demand will be lost in other countries whereas the value will be captured by Anthropic and OpenAI in the US, it will take a while before people can agree on what must be done and make the political deals to do it. This is also politics getting in the way, but in a very different sense from what they seem to have in mind, which is more something like politicians making it more difficult to build data centers because their constituents think they use too much water or whatever. Even assuming that the demand for AI will be at the level they assume, it’s not obvious that OpenAI and Anthropic will be able to retain a large part of it as profit, as opposed to upstream suppliers like chip manufacturers, power companies, etc. The relevant question is kind of the production side analogue of the “what will be scarce” question that @alexolegimas asked on the demand side. Even if you assume that the elasticity of substitution between state-of-the-art and less capable models will still be pretty low even after the latter’s capabilities reach a certain point, which as I already noted is hardly obvious, it’s still not clear that upstream suppliers will not be in a position to capture most of the scarcity rent. This is a somewhat disorganized rant, but the point is that, for the kind of scenario Dwarkesh and Dylan have in mind (i. e. OpenAI and Anthropic generate several trillions in revenue in 2-3 years) to materialize, many highly non-obvious assumptions have to hold. I have no doubt that possible replies to the arguments I made could be made, I can think of several myself, but this is just a tweet and it’s already way too long. My own view is that it’s extremely unlikely all those assumptions will hold, but I don’t even care if you disagree with me. What I would really like to convince AGI-pilled people of, because again I don’t think Dwarkesh and Dylan are being atypical here, is that if they want to make the argument that the kind of timeline they’re defending is plausible there are tons of implicit assumptions that need to be made explicit and a case needs to be made for them. The fact that people rarely do that frankly suggests to me that, for the most part, they don’t even realize that they need to make those assumptions. They seem to think that people who don’t find their timeline plausible just “don’t know what is coming” and are in denial. But while it’s probably true that a lot of people are in denial about what AI is capable of I also think AGI-pilled people have very naive views about how improved model capabilities will translate into economic change. Again, I can hardly be called an AI skeptic, I’m convinced that AI will profoundly transform human civilization and that it will do so much faster than the Industrial Revolution did. I just don’t think it will do so as fast as many people in that space believe. Maybe I’m wrong, but if people think so, they need to make their case more rigorously because the claims they make are very strong and very strong claims require equally strong arguments. This ain’t it.