# Decision Points for Embedding AI Agents in Enterprise Systems # Compile-time vs Runtime LLM 🎯 The Hook Are you calling an LLM every time you generate the same daily report format? Whether the agent's behavior is "locked in ahead of time (compile-time)" or "decided dynamically by the LLM at execution (runtime)" depends on task frequency, format stability, cost, and latency. Using runtime LLM for high-frequency, stable-format tasks can easily balloon monthly costs into tens of thousands of dollars 🔑 📋 Overview The compile-time approach excels when task input/output formats are stable and patterns are fixed. Posting Salesforce opportunity stage changes to Slack in a standard format is a classic example — fully templateable processing. When execution frequency reaches hundreds to thousands per day, the cost savings from eliminating LLM calls are massive. Implementation uses code generation, template engines, or rule engines, guaranteeing reproducibility and testability with same-input-same-output behavior. The runtime LLM approach fits tasks where content varies every time and natural language understanding and generation are inherently required. Customer inquiry responses on Zendesk, automatic Jira ticket classification — cases where input diversity makes rule-based branching explode 📊 🔍 Decision Points Evaluate in this order: Is the input/output format stable? → Stable means compile-time, high variability means runtime What is the execution frequency? → Hundreds per day or more warrants compile-time cost optimization Is natural language understanding/generation inherently needed? → If yes, runtime. If it is just structured data transformation, compile-time Is reproducibility important? → Same input must produce same output? Compile-time How fast must you adapt to new patterns? → Need to handle new patterns without code changes? Runtime In environments where deploy cycles take days, runtime LLM flexibility is especially valuable. New inquiry patterns can be addressed with prompt adjustments alone, without waiting for a code deployment ⚡ 💡 Key Details Most production systems build pipelines that combine compile-time and runtime: LLM classifies, compiled flow executes: The runtime LLM classifies user intent, and the classification result triggers a precompiled workflow (template or rule engine). This balances classification flexibility with execution efficiency. Compiled pipeline + LLM fallback: Fixed patterns are processed at high speed by precompiled logic, and only inputs that do not match any pattern fall back to the LLM. This is the most cost-efficient configuration. Gradual compilation: Start with runtime LLM, extract frequent patterns from logs, and progressively migrate them to precompiled flows. Combining this with a feedback loop for continuous compilation is the ideal trajectory 🔄 ⚖️ Trade-offs Relying entirely on runtime LLM means paying LLM inference costs for report generation tasks that a template engine could handle at 1/100th the cost. Over thousands of daily invocations, this adds up fast 😩 Fixing everything at compile-time leads to branching condition explosion and an unmaintainable rule engine. When the rule count exceeds 100, it is a signal to consider migrating to LLM-based handling. Compiling too early is also a failure pattern. If you compile before patterns have stabilized, you end up with frequent code changes that negate the benefits. Run on runtime LLM first, confirm pattern stability, then compile. Another trap: hybrid routing logic that exists only in one developer's head. Codify the criteria for which requests go to compiled flows and which go to the LLM ⚠️ 🛠️ Use Cases Slack notification formatting: Salesforce opportunity stage changes posted to Slack are fully templateable (compile-time). Hundreds per day, no reason to involve an LLM. However, generating win-probability analysis comments for those same opportunities does require a runtime LLM 📚 Customer support (Zendesk): FAQ-style standard responses use templates with variable substitution (compile-time). Novel, complex inquiries use runtime LLM for answer generation. The standard hybrid architecture: LLM classifies intent → routes to standard/non-standard → each feeds into its processing pipeline 🛒 Report generation pipeline: Daily KPI reports use a template engine (compile-time — once per day but the format is completely fixed). Quarterly insight analysis reports use runtime LLM (natural language commentary and recommendations are required). Both compile-time and runtime coexist within the same pipeline 🔧 Practical tip: Start with runtime LLM and extract frequent patterns from your logs. Typically, the top 20% of patterns account for 80% of traffic. Compiling just that 20% can dramatically reduce LLM costs 💪 #AIAgents# #EnterpriseArchitecture#