John Schulman
@johnschulman2
@thinkymachines. Interested in reinforcement learning, alignment, birds, jazz music
Joined May 2021
2.3K Following 83.8K Followers
As a follow-up, it's exceedingly unlikely that training on user data contributes much to frontier model gains in areas like math -- those come from scaling up pretraining and RLVR. User data is more likely used to find failure modes or situations that are hard to recreate with hired annotators.
That said, model companies vary in how aggressively they train on user data (and uploading repos isn't hypothetical). I wish there were stronger norms around disclosing how companies train on user data -- with what methods, and to improve what capabilities.
Show more
This isn't the most notable aspect of today's news, but on the user data issue, there are different kinds of *training on user data* with very different privacy/IP implications. Sadly, AI cos don't like to disclose what they're doing.
- pretrain on user data, with users' tokens as prediction targets: high regurgitation risk, improper
- use user prompts to distill large models into small ones: low regurg. risk, some companies probably do this
- use user traces to construct RL tasks: low regurg. risk, because RL has low memorization abilities, but can extract customer IP, depending on how it's done. Ranges from benign "use explicit user feedback in reward model training" to invasive "upload user's coding environment and commit history to turn into rl envs"
"De-identification" is weak -- you can identify someone with a small number of bits, and long traces have more than enough. And it doesn't affect IP leakage concerns.
Show more
