In generative recommendation, everyone has assumed they know what makes a "good" semantic ID design. So what happens when you actually put that assumption to a large, controlled reproducibility test? 🔍 Generative recommenders turn items into discrete token sequences called semantic IDs, then generate the next item autoregressively. Designs like RQ-VAE, OPQ, and RQ-Kmeans have proliferated, but prior comparisons used different datasets and backbones, so nobody could say which design actually wins. So the authors reran 12 methods side by side under the exact same data splits and evaluation protocol, and the assumptions started falling apart. No single SID design dominates across datasets. Codebook utilization balance barely correlates with recommendation quality. Longer codes don't always help, and bigger backbones can even hurt performance. And when you measure how well an SID preserves an item's local semantic neighborhood, the winning method flips depending on whether you use Jaccard or RBO. The lesson from What Makes a Good Semantic ID for Generative Recommendation? A Reproducibility Study is that good SID design isn't about chasing one metric, it's about balancing several at once. URL: #RecSys# #Reproducibility#

DeSci: funding, governance, incentives An overview of how on-chain coordination can support peer review, reproducibility, and shared ownership of research outputs. Read more 👇

Practices for embedding AI agents into enterprise systems [Prompt Supply Chain Management] 💡 Has a "quiet prompt tweak" ever changed production behavior unexpectedly? This pattern treats prompts as code artifacts -- versioned, reviewed, signed, and eval-gated before deployment. 🔥 Problems it solves - Prompt changes cannot be tracked or reviewed, making it impossible to identify the cause of quality regressions - External inputs injected into prompt templates cause unintended behavior - No way to trace which prompt version produced a given result, destroying reproducibility 🏗️ Proposed pattern Manage prompts in Git with a dedicated registry. Require pull requests and code reviews for every change. Mandate eval gate passage on each update, automatically running quality checks against a golden dataset. Sign production-deployed prompts for tamper detection. Apply injection countermeasures -- delimiters, sanitization, and least-privilege scoping -- to all template variable insertion points. In essence, apply the same CI/CD quality process used for software to your prompts. ✅ Selection criteria - Good fit: Production agents with multiple contributors - Not a fit: Throwaway experiments or PoCs (governance overhead is not justified) ⚠️ Pitfalls - Without a culture that treats prompt changes as seriously as code changes, the process becomes hollow - Skipping signing and tamper detection leaves malicious or accidental changes undetectable - Failing to continuously update the golden dataset means new failure patterns slip through unnoticed 🛠️ Implementation Approach 1. Manage prompts, system instructions, and templates in a Git repository, requiring pull requests and code reviews for every change 2. Integrate an eval gate into the CI/CD pipeline to automatically run quality checks against a golden dataset on every prompt change 3. Sign production-deployed prompts with HMAC or similar mechanisms and verify integrity at deploy time for tamper detection 4. Implement injection countermeasures for external inputs in template variables -- delimiters, sanitization, and least-privilege scoping 5. Centralize version and deployment history in a prompt registry, enabling instant rollback when issues are detected in production #AIAgents# #EnterpriseArchitecture#

# Decision Points for Embedding AI Agents in Enterprise Systems # Compile-time vs Runtime LLM 🎯 The Hook Are you calling an LLM every time you generate the same daily report format? Whether the agent's behavior is "locked in ahead of time (compile-time)" or "decided dynamically by the LLM at execution (runtime)" depends on task frequency, format stability, cost, and latency. Using runtime LLM for high-frequency, stable-format tasks can easily balloon monthly costs into tens of thousands of dollars 🔑 📋 Overview The compile-time approach excels when task input/output formats are stable and patterns are fixed. Posting Salesforce opportunity stage changes to Slack in a standard format is a classic example — fully templateable processing. When execution frequency reaches hundreds to thousands per day, the cost savings from eliminating LLM calls are massive. Implementation uses code generation, template engines, or rule engines, guaranteeing reproducibility and testability with same-input-same-output behavior. The runtime LLM approach fits tasks where content varies every time and natural language understanding and generation are inherently required. Customer inquiry responses on Zendesk, automatic Jira ticket classification — cases where input diversity makes rule-based branching explode 📊 🔍 Decision Points Evaluate in this order: Is the input/output format stable? → Stable means compile-time, high variability means runtime What is the execution frequency? → Hundreds per day or more warrants compile-time cost optimization Is natural language understanding/generation inherently needed? → If yes, runtime. If it is just structured data transformation, compile-time Is reproducibility important? → Same input must produce same output? Compile-time How fast must you adapt to new patterns? → Need to handle new patterns without code changes? Runtime In environments where deploy cycles take days, runtime LLM flexibility is especially valuable. New inquiry patterns can be addressed with prompt adjustments alone, without waiting for a code deployment ⚡ 💡 Key Details Most production systems build pipelines that combine compile-time and runtime: LLM classifies, compiled flow executes: The runtime LLM classifies user intent, and the classification result triggers a precompiled workflow (template or rule engine). This balances classification flexibility with execution efficiency. Compiled pipeline + LLM fallback: Fixed patterns are processed at high speed by precompiled logic, and only inputs that do not match any pattern fall back to the LLM. This is the most cost-efficient configuration. Gradual compilation: Start with runtime LLM, extract frequent patterns from logs, and progressively migrate them to precompiled flows. Combining this with a feedback loop for continuous compilation is the ideal trajectory 🔄 ⚖️ Trade-offs Relying entirely on runtime LLM means paying LLM inference costs for report generation tasks that a template engine could handle at 1/100th the cost. Over thousands of daily invocations, this adds up fast 😩 Fixing everything at compile-time leads to branching condition explosion and an unmaintainable rule engine. When the rule count exceeds 100, it is a signal to consider migrating to LLM-based handling. Compiling too early is also a failure pattern. If you compile before patterns have stabilized, you end up with frequent code changes that negate the benefits. Run on runtime LLM first, confirm pattern stability, then compile. Another trap: hybrid routing logic that exists only in one developer's head. Codify the criteria for which requests go to compiled flows and which go to the LLM ⚠️ 🛠️ Use Cases Slack notification formatting: Salesforce opportunity stage changes posted to Slack are fully templateable (compile-time). Hundreds per day, no reason to involve an LLM. However, generating win-probability analysis comments for those same opportunities does require a runtime LLM 📚 Customer support (Zendesk): FAQ-style standard responses use templates with variable substitution (compile-time). Novel, complex inquiries use runtime LLM for answer generation. The standard hybrid architecture: LLM classifies intent → routes to standard/non-standard → each feeds into its processing pipeline 🛒 Report generation pipeline: Daily KPI reports use a template engine (compile-time — once per day but the format is completely fixed). Quarterly insight analysis reports use runtime LLM (natural language commentary and recommendations are required). Both compile-time and runtime coexist within the same pipeline 🔧 Practical tip: Start with runtime LLM and extract frequent patterns from your logs. Typically, the top 20% of patterns account for 80% of traffic. Compiling just that 20% can dramatically reduce LLM costs 💪 #AIAgents# #EnterpriseArchitecture#

The real use case for Jev in algo trading is in using it to quickly pathfind toward useful signals while in research mode It is not a replacement for making individual trading decisions in realtime. Structured fits over lots of historical data, with well-crafted signals, in an optimized event loop, are much better-suited for this problem, both for precision, latency, and reproducibility

The NVIDIA H100 remains one of the most valuable GPUs for scientific, academic, or precision-driven work. H100s are exceptionally strong at: -FP32 / FP64 numerical workloads -Simulations and scientific modeling -Physics, biosciences, and high-precision training -Any task where reproducibility is non-negotiable We can help you determine whether the H100 is the right tool for your job:

# Decision Points for Embedding AI Agents in Enterprise Systems # Prompt Change Control 🎯 The Hook A prompt change looks like "just tweaking some wording" — but it is actually a configuration change that can fundamentally alter system behavior. You have CI/CD and code reviews for deployments, but can you trace who changed which prompt, when, and why? If not, you are running production without change management on one of your most critical system components. The realistic approach is not uniform strictness — it is tiered control based on production impact 🔑 📋 Overview Prompt change control is the dial governing how much approval, testing, and version management you require for changes to system prompts and tool definitions. Stricter settings improve reproducibility and safety but slow development velocity. Looser settings enable rapid iteration and experimentation but increase the risk of unintended changes reaching production. The key insight is that different prompt components carry different risk levels, so control levels should vary accordingly. Role definitions in system prompts and few-shot examples demand completely different governance 📋 🔍 Decision Points This dial branches into three tiers based on production impact: High impact (customer-facing, financial, legal) → Strict: code review + 100% eval pass + 2 approvers + canary deploy Medium impact (internal operations, broad scope) → Standard: code review + 95% eval pass + 1 approver Low impact (internal experiments, limited release) → Lightweight: self-review + basic eval pass + change log Further, control policies should vary by prompt component: System prompt (role, constraints) → Low change frequency but high risk. Manage strictly as a design decision Tool definitions (names, descriptions, schemas) → Directly affects tool selection accuracy. Strict control required Few-shot examples → Medium risk. Standard control with eval quality verification Context injection templates → High change frequency, low-to-medium risk. Lightweight to standard Output format instructions → Low risk, but verify alignment with downstream systems ⚡ 💡 Key Details Five components of a control framework: Version control — Manage prompts in Git just like code. Enables diff visualization and history tracking. If you cannot trace "who changed what and when," future decision-making becomes impossible. Always record the reason for each change. Eval gate — Make passing the existing eval suite a deployment prerequisite for any prompt change. Measure eval regression detection rate (the percentage of prompt-induced quality drops caught before deployment) and add missed cases to strengthen the eval suite. Approval process — Reviewer tiers scaled to impact level. Strict tier requires peer engineer plus tech lead. Changes to financial thresholds (like refund limits) add legal review. Canary deploy — Apply the new prompt to a subset of traffic (roughly 10%) and compare quality metrics over 24 hours before full rollout. This can share infrastructure with A/B testing systems. Rollback procedure — The ability to instantly revert to the previous version when issues arise. Git revert integrated with the deployment pipeline is the baseline 🔬 Five metrics to track: prompt change frequency (per environment), change-to-deploy lead time (target 1-3 business days for strict, within hours for lightweight), incident count caused by prompt changes, eval regression detection rate, and rollback rate 📈 ⚖️ Trade-offs Prioritizing development speed (relaxed) enables rapid prompt improvement and fast A/B test cycles. But "who changed what when" becomes untraceable, making root cause analysis during incidents extremely difficult. Unintended changes can slip into production causing quality degradation or security issues. If system prompt role definitions change without anyone noticing, the blast radius covers every single response 😩 Prioritizing reproducibility and safety (strict) makes all changes traceable and simplifies incident root cause identification. But the approval process becomes a bottleneck, extending improvement lead times. When even small improvements require heavy process, team motivation drops and you end up with "I want to fix this prompt but it is too much hassle" — the exact opposite of what you intended ⚠️ The solution is clear: do not lower the control level itself. Instead, shorten change lead time through eval automation and parallel approval workflows. And explicitly separate control levels between experimental and production environments — never trade production safety for experimentation speed. 🛠️ Use Cases Zendesk customer support agent: System prompt changes require PM plus engineering lead approval. Refund limit wording changes add legal review. Full rollout only after eval pass plus canary (10% traffic for 24 hours). Change lead time is 1-3 business days 📞 Slack internal experiment bot: Developers can change prompts via self-review. Basic eval pass is mandatory but approval process is skipped. Change logs are automatically recorded for incident tracing. Change lead time is within hours 💬 Salesforce sales support agent: Tool definition additions and changes require code review. Prompt changes affecting deal stage classification logic need tech lead approval. Large-scale prompt changes are rolled out incrementally to limit blast radius 🎯 Practical tip: Start with strict-leaning controls and relax gradually as the team matures and evals become comprehensive. When a prompt change causes an incident, add that case to your eval suite to automate prevention. And never execute large prompt changes (like rewriting role definitions) all at once — incremental rollout makes impact identification far easier 💪 #AIAgents# #EnterpriseArchitecture#

Question for the LLM Research Community: Is anyone aware of fully reproducible experimental results showing that MLA beats GQA under the same KV cache? I ask for two reasons: (1) I find it surprising that there is still a divide between Chinese labs using MLA, and Western labs using GQA + sliding window, when certainly many ablations have been run by many parties. (2) At Marin we are considering MLA for our next large scale run, but in early (!) smaller scale ablations it appears worse than heavily tuned feature-rich GQA, even after controlling for KV cache. I would like to run better experiments here. Below I cover my thoughts on general reproducibility, then specifics on MLA. Every empirical result in ML is only contextually true. Conditioned on the data distribution, optimizer settings, model width, model depth, finer architecture details, hardware, kernel engineering, initialization, token count, tokenizer, context length, and evaluation protocol, one can reach different conclusions. Contextual results are still useful. Typically if I see a promising method, I will first attempt a full 'context jump', where I apply it to my own context, hoping results transfer. Sometimes they do. If they don't, I can try 2 things: modify the implementation of the method, or modify the context. Ideally I have access to the full context of the original result. Then I can perform a 'context bridge', where I ablate one aspect of the context at a time, isolating exactly why a method performs differently. This lets me make an informed decision to either update my context to let the method shine, or stick with my context and leave the method out. MLA is tricky to assess at small scale. A core aspect of MLA is compressing hidden_dim->latent_dim. Then for each head, latent_dim->head_dim. Typically head_dim is fixed at 128, partially for hardware reasons, and partially for learning dynamics (head_dim of 8 wouldn't have sufficient representational capacity). To get MLA dynamics, you want hidden_dim>>latent_dim, and latent_dim>head_dim. This window closes at small scale. The degree of tuning can unfairly alter the scales. In GQA we have partial RoPE, QK Norm, Gated Attention, attention sharpening, sliding window, and other techniques that give a 30%+ training boost. They don't seem to give the same boost to MLA. On one hand, you want to compare techniques apples:apples with equal tuning. On the other hand, there is a finite amount of future tuning you can do, so prior tuning influences which approach is most pragmatic. Creating controlled tests between MLA and GQA is tricky. Several factors: kv_cache, quadratic attention flops, attention projection flops. kv_cache is controlled by scaling down kv_heads to match MLA, or scaling up kv_latent to match kv_heads. quadratic attention flops are controlled by scaling up GQA's query head count to match MLA head count, or scaling down MLA head count. Also scaling up GQA head_dim 128->192, or scaling down MLA head_dim to 192->128. In general, it's informative to context match to both option A's preferred context and option B's preferred context. Sliding window is another confounder. MLA is theoretically elegant, if we ignore RoPE. It replaces the 'replicate' op of kv_heads in GQA with a 'mix' op (pic below). Since the 'mix' can learn to 'replicate' if it wants, MLA is purely more expressive, and the cost of 'mix' is hidden at inference with absorb trick. Yet in practice, I find that at small scale this 'mix' op doesn't add much value and interacts poorly with the optimizer dynamics. And the change to RoPE hurts. My current plan is to first tune and ablate our model features around MLA, then run 3 scaling ladders: MLA, GQA with 2 kv_heads, and GQA with higher kv_heads. For each ladder, fit a loss vs compute projection. If MLA performs worse at our target compute compared to both GQA options, drop it. If MLA beats 2 kv_heads but loses to higher KV_heads, then it becomes a kv_cache tradeoff. Early results indicate MLA will perform worse than both feature-rich GQA ladders, but we will see. Any positive external reproducible results for MLA would help make sure I give it the best chance possible.

AI Agents in DeSci: The End of the Grant Era? Traditional science funding is broken. Grants are distributed slowly, and bureaucracy stifles innovation. We are building a more efficient system. The Solution: Decentralized Science (DeSci), governed by AI agents. How it works: AI agents are not just assistants. They become PIs (Principal Investigators) in DeSci DAOs. Evaluation: Agents analyze scientific proposals in milliseconds, using verifiable logic (hello AX Pillars), not subjective opinions. Distribution: Grants are distributed automatically based on the on-chain reputation of agents and predicted impact. Coordination: Agents coordinate cross-disciplinary research, verifying each other's data for hallucinations and ensuring reproducibility. The future is not about making people write better grants. The future is about letting AI agents drive the scientific process. Are you ready to let an AI agent peer-review your data?