some thoughts on jev
1) i tend to think that twitter hype is not relevant for ai product releases; especially when someone claims to be doing something novel with the architecture
2) but, i think that jev is interesting in principle; it is a low cost, fast classification service, and it claims to be about equivalent to gpt-5.6-terra in performance
3) something interesting here is that this is what most ai services looked like in 2022; cohere sold a classification api, an embedding api, a reranker
4) the problem was that, in addition to not being as effective as gpt-3 and then gpt-4, these services were much harder to use as a developer
5) you had to decide which apis to use, then write the code to glue them together; so, you got less performance for more work
6) but now, we can imagine coding agents making it much easier to select and integrate a cheaper service with specialized characteristics
7) evals are still a bottleneck, though; you need some way to automatically compare performance between services using their apis
8) and, enterprise companies often have a hard time building evals that reflect their actual use cases; it's not a core expertise
9) but, the better coding agents get at integrating and switching between services, the more valuable those evals become;
10) this means companies have more reason to figure out the evals problem;
11) maybe that means getting rights to use customer data for testing, or getting better at purchasing synthetic test data, or something else
12) so, i think there is an interesting world where coding agents create more market for specialized api services, even specialized model services
13) which can compete either on cost or on capabilities; with the winner determined by cost basis or specialized data, etc...
14) note, it might be unclear but if specialized models services becomes an important direction, i expect the frontier labs to have an edge on cost basis
15) and, i think that, at least for now, these specialized model services probably do not provide enough of a cost difference to be relevant
더 보기