Live inference data is the best training data for speculators. The hard part is training them online without storing hidden states or impacting inference performance and reliability.
We built exactly that into the Baseten Inference Stack. Result: +20% median acceptance rates, with some domain-specific workloads improving by 100%.
显示更多
