We check OOD generalization for
@periodiclabs Neon and report optimal cost-performance here, too. This indicates useful generalization for our model.
But our experimental data also opens up an interesting ML research program.
For example, RL tasks to directly predict experimental outcomes given all prior experimental data known up until that date. This is loosely analogous to the often-proposed ML experiment of training AI on all literature before 1905 and then evaluating whether it can derive the theory of relativity.
An AI that already knows the answer through pre-training can cheat on RL tasks like this (i.e., it can skip reasoning and simply output a memorized experimental result). A unique, complete, dated system of record makes this work possible and fruitful.