注册并分享邀请链接,可获得视频播放与邀请奖励。

Will Bryk
@WilliamBryk
加入 May 2021
191 正在关注    23.1K 粉丝
Exa is now one of the largest indexes in the world. We serve 80B pages, track 1.4T urls, and are on track to be Google-scale in early 2027. It's difficult to know the size of indexes, but we estimate Google to be ~1T, Bing to be ~500B, and Yandex ~200B. Brave mentioned in April that they're at 40B. Most of the web is trash that can hurt AI outputs, so you actually have to crawl way more than you serve and train models to filter out the trash. In terms of peak QPS, we estimate Bing at roughly 30k/s and Google at ~500k/s. In our case, because agent traffic often requires large fan-out (deep searches can use dozens of sub-searches) and can be spiky (like when AI labs RL with us), we're already starting to provision for Bing-scale traffic. Within 2 years, agents will search at many millions per second. They'll also want comprehensiveness over all data, far bigger than any index today. So the retrieval infra required will be larger than Google-scale in both dimensions.
显示更多
0
167
3.5K
224
转发到社区