注册并分享邀请链接,可获得视频播放与邀请奖励。

Mathias Lechner
@mlech26l
Cofounder/CTO at Liquid AI and Research Affiliate MIT
加入 December 2017
233 正在关注    2.6K 粉丝
Pipette is our new LLM leaderboard that shows not only eval performance but also how fast the models run and how much memory they consume
Today we release Pipette, a model evaluation suite for on-device intelligence, in partnership with @ArtificialAnlys Most benchmarking platforms are optimized to measure core capabilities and speed profile of foundation models served in the cloud. Pipette gives the field a common, reproducible way to measure the quality, speed, latency, and memory use of AI models on devices such as phones, laptops, PCs, AI boxes, and embedded hardware. > Pipette is open source > In Pipette, models get compated as model + quantization + runtime + device from one interface. > It comes with a warehouse of verified benchmark results, currently with 10k+ results across 35 model classes, 7 quants, llama.cpp runtimes, and 4 devices. > Pipette is a dynamic platform, allowing new contributions from day one, adding new devices, runtimes, model families, and quantization levels. 🧵
显示更多