Register and share your invite link to earn from video plays and referrals.

Brett Winton
@wintonARK
Chief Futurist @ARKInvest. ARK Venture IC. Welcome to the Great Acceleration.
Joined August 2012
595 Following    210.6K Followers
Cost per performance in AI seems to be falling more than ~200x annualized (and higher at higher levels of benchmark performance.) And exceeding 500x annualized cost declines moving along the efficient frontier of the performance curve. Probably more relevant practically for users, at $1 per task whereas today you might only have 50/50 odds that the agent succeeds, by the end of year, you should have an 89% chance of success and by a year from now a 97% chance (at least on deepSWE 1.1 type tasks.) The moving pareto frontier makes it difficult to cleanly report a performance cost improvement due to the shape of the cost performance curve. At the highest asymptote of performance you go from literally not being able to achieve such a low error rate *at any cost* to being able to get that performance for 10s of $s per task (an infinite cost decline). We forecast that top-end rate separately. In the belly of the curve you need only cross a performance cost threshold, but as the curve steepens you can also less expensively buy more performance (though not 100% clear that the curve really is steepening that much.) We also model that performance curve, though given uncertainty effectively zero out continued improvement for the conservative case. And at lower levels of benchmark performance you get a cleaner understanding of the underlying cost-per-performance improvement though at thresholds that are less interesting practically. If anything I suspect that people are wildly under indexing on the rate of improvement of these models. Even last month's AI spend will fall to a fraction of a fraction within the year (if we weren't going to so predictably deploy the new capabilities against new workloads, tasks and challenges.) Spending a lot of time trying to painstakingly carve out specific current workflows into more efficient models, at this point in the improvement curve, seems, if anything, a fool's errand designed to line the pockets of a consultant. Let the tokens flow.
Show more