New Speculative Decode Video!
Explore the interactive presentation yourself:
TLDW: A cheap draft model generates multiple tokens speculatively, and the large model verifies them in a single forward pass, same output quality as a big model, less latency.