# Decision Points for Embedding AI Agents in Enterprise Systems # General-Purpose Large Model vs Small Fine-Tuned Model — Choosing the Inference Engine 🎯 The Hook For your agent's inference engine, do you go with a GPT-4-class general-purpose large model controlled via prompts, or a task-specific small fine-tuned (FT) model? "Bigger is smarter" is true, but "bigger is always the right answer" is not. The optimal choice shifts based on task diversity, call frequency, cost constraints, and rate of change. Most enterprises find the practical sweet spot in a staged migration: start with a large model, accumulate training data from logs, then replace high-frequency tasks with FT models one by one 🔑 📋 Overview General-purpose large model plus prompting shines when tasks are diverse and cannot be enumerated in advance, when business requirements change rapidly, and when you want to handle new tools or APIs by simply updating prompts. Small FT models are appropriate when the task is narrowly defined with stable input/output patterns, call frequency is high and cost reduction matters, or latency requirements are strict. These are not mutually exclusive — a router plus specialist model hybrid is the standard enterprise approach 📊 🔍 Decision Points Here is the decision flow: If task types number 10 or more with new ones added frequently, choose the general-purpose large model. The flexibility to handle new tasks with just a prompt change is a decisive advantage. If monthly request volume exceeds several hundred thousand, calculate the cost benefit of small FT. Compare on total cost of ownership — not just API call costs, but also training costs, GPU inference infrastructure, and training data management overhead. If the latency requirement is P95 under 500ms, prioritize small FT. Large model inference times may not meet this bar. If you cannot prepare and continuously update several thousand or more training examples, stay with the large model. FT on insufficient training data will underperform the general-purpose model. If business requirements change more frequently than monthly, the re-training cost often makes maintaining the large model the better choice ⚡ 💡 Key Details Three hybrid patterns to understand: Router plus specialist models — Use the large model to classify request intent, then route routine tasks to small FT models. Intent classification cost is low, and specialist task inference cost drops dramatically. A Slack Bot takes company-wide questions, routes ticket classification to a small FT, and handles complex investigations with the large model. FT for drafts, large model for finishing — Generate drafts quickly with a small FT model, then verify and refine with the large model only for quality-critical cases. A practical pattern that balances cost and quality. Staged migration — Start with everything on the large model, accumulate training data from call logs, and replace high-frequency tasks with small FT models one by one. This is the lowest-risk approach and the one most enterprises adopt 🔄 ⚖️ Trade-offs Premature FT investment is the most common failure. Building an FT model before sufficient logs accumulate produces accuracy below the general-purpose model. Wait for at least three months of logs before considering FT. Reaching for FT during the PoC phase is premature 😩 FT model staleness is equally serious. Business rules and product lineups change, but training data is not updated, causing the model to answer with outdated knowledge. Automating the re-training pipeline is a mandatory investment for operating FT models. Cost comparison errors are frequent. Comparing only API call costs while overlooking FT training costs, GPU inference infrastructure costs, and training data management costs leads to flawed decisions. Compare on TCO (total cost of ownership) or the analysis is meaningless. Missing quality evaluation is another blind spot. Without regularly evaluating FT model accuracy, model drift goes undetected. Continuous A/B testing against the general-purpose model is the foundation of quality assurance ⚠️ 🛠️ Use Cases Company-wide help desk agent (Slack Bot): Handles broad queries across IT, admin, HR, and accounting. The large model classifies intent, then routes ticket categorization and prioritization (tens of thousands per month) to a small FT. Complex investigations and novel question types are handled by the large model 📚 Ticket classification agent (Zendesk integration): Automatically classifies category, priority, and assigned team for tens of thousands of support tickets daily. High call frequency and stable I/O patterns make small FT the optimal choice. Training data is updated monthly, and accuracy is monitored via A/B testing against the large model 🏢 Product review analysis agent (Shopify integration): Sentiment analysis and tagging of product reviews are batch-processed by a small FT. Customer insight summaries derived from reviews are generated by the large model. Routine classification goes to FT, creative analysis goes to the large model — a clear division of responsibility 🔧 Practical tip: "Start with the large model and accumulate logs" is the golden rule. Identify high-frequency tasks from three or more months of logs, build training data, validate FT effectiveness via A/B testing against the large model, then migrate. Make the FT transition a data-driven decision 💪 #AIAgents# #EnterpriseArchitecture#