Horace He(@cHHillee):@giffmana @YouJiacheng Why stop there? Every single decode step you're loading all of your cached tokens! We can just count those in tokens used :)

Horace He

@cHHillee

@thinkymachines Formerly @PyTorch "My learning style is Horace twitter threads" - @typedfemale

Joined February 2010

610 Following 53K Followers

Horace He@cHHillee

2026.08.24 23:23

@giffmana @YouJiacheng Why stop there? Every single decode step you're loading all of your cached tokens! We can just count those in tokens used :)

Forward to community

Most Popular Users

Unipcs (aka 'Bonk Guy') 🎒

286.9K Followers

594.1K Followers

90.4K Followers

241.6M Followers

19.8K Followers

681.6K Followers

242.9K Followers

2.1M Followers

93.4K Followers

1M Followers

195.2K Followers

406.5K Followers

9M Followers

70.4K Followers

1.8M Followers