Just spoke to one of the big data labeling businesses.
Few interesting insights:
- They predict the majority of their revenue will come from Fortune 1000 enterprises, not labs in a few years
- They believe every company will want to own their intelligence, but owning intelligence does not necessarily mean using open source models
- A company’s evals will become their main proprietary IP given the improvement in agent performance after properly setting up & running internal eval environments
- Most enterprises haven’t graduated from coding agents and it’s largely due to not having the proper eval infrastructure to make non-Eng agents performant