A useful but little-known OpenAI API feature 🔄 Want to fairly compare OpenAI models against alternatives? Now you can, on the same evaluation framework. OpenAI's "External models" feature lets you evaluate non-OpenAI models on the same Evals infrastructure. One platform, one set of graders, apples-to-apples comparison. 📌 Title: External models 🔗 URL: 🧩 Overview Model selection and migration decisions require fair comparison on the same criteria. External models lets you evaluate Claude, Gemini, open-source models, and others alongside OpenAI models using the same graders and datasets on OpenAI's Evals platform. No more maintaining separate evaluation tools for each vendor. 🛠 How to use it Register external model connection details (API endpoints, credentials) in Evals and add them as evaluation targets. Then run tests using your datasets and graders just like normal Evals. Results show up side by side on the same dashboard for easy comparison. 🏗 Building it into production ・Model selection process: when a new model drops, benchmark it against your current model on the same tasks. ・Migration decisions: quantitatively compare quality on your actual tasks before switching models. ・Cost optimization: periodically scan for cheaper models that deliver equivalent quality on your workloads. ・Multi-model strategy: build an evaluation framework for choosing the best model per task type. 💡 Use cases 🏆 Multi-model benchmark comparisons 🔀 Quality verification before model migration 💰 Cost-to-quality optimization 📋 Task-specific optimal model selection ⚠️ Watch out External model API keys and usage costs are separate. Some models may have different response formats or error handling, requiring output normalization during evaluation. Always compare on the same tasks and datasets for a fair assessment. ✨ Move model selection from gut feeling to data-driven. Start by lining up your current model against a challenger in Evals and see the numbers. #OpenAI# #LLM#