Was reading this thread on pretraining data for Marin and it mentions a great set of data curation papers. This paper ( was especially interesting. Had not seen it before, but the core argument is that data selection should depend on available compute. E.g., if you have tons of compute, training on lower quality data may be better than repeating high-quality data. Simple but useful! "We demonstrate that making filtering decisions independent of training compute is often suboptimal: the limited high-quality data rapidly loses its utility when repeated, eventually requiring the inclusion of 'unseen' but 'lower-quality' data... Our scaling laws characterize the differing 'utility' of various quality subsets of web data; account for how utility diminishes for a data point at its 'nth' repetition; and formulate the mutual interaction of various data pools when combined, enabling the estimation of model performance on a combination of multiple data pools without ever jointly training on them."