Rylan Schaeffer
@RylanSchaeffer
Creating something new. Ex-Meta TBD. On-Leave from Stanford w/ @sanmikoyejo. Prev @ Gemini, MIT, Harvard, Uber, UCL, UC Davis
2K Following 6.9K Followers
Really cool work on data-constrained pretraining!
Key idea is to assign repeated data an effectiveness: how many fresh tokens would have produced the same validation loss? That lets them put repeated and fresh data on a common scale.
1/7
Show more
