TL;DR: Adding features to a recommender system is easy, but safely removing them is brutally hard. Pinterest built a production framework that automates that whole cleanup process. Title: FORTE: A Production Framework for Feature Evaluation and Deprecation in Industrial Recommender Systems URL: Key points 🧩 Adding a feature just needs a metric win, but removing one must not regress any metric, a built-in asymmetry that stalls cleanup ⏱️ Daily perturbation experiments automatically track feature importance and rank stability over time 🧪 Grouped perturbation evaluates candidate feature sets together, catching interactions before costly retraining 🚀 Pooled execution resolves all experiments for a model in a single distributed inference pass, cutting runtime 54x for 300 experiments 🔗 Automated lineage tracking finds every downstream consumer model, making deprecation safe 📉 On the Pin Ranker model, removing 8 features cut parameters by 29.9%, boosted throughput 19.8-25.5%, and saved hundreds of thousands of dollars a year Feature removal infrastructure deserves as much engineering investment as feature addition, and this looks like a solid blueprint for it. #RecSys# #MLOps#