In generative recommendation, everyone has assumed they know what makes a "good" semantic ID design. So what happens when you actually put that assumption to a large, controlled reproducibility test? 🔍 Generative recommenders turn items into discrete token sequences called semantic IDs, then generate the next item autoregressively. Designs like RQ-VAE, OPQ, and RQ-Kmeans have proliferated, but prior comparisons used different datasets and backbones, so nobody could say which design actually wins. So the authors reran 12 methods side by side under the exact same data splits and evaluation protocol, and the assumptions started falling apart. No single SID design dominates across datasets. Codebook utilization balance barely correlates with recommendation quality. Longer codes don't always help, and bigger backbones can even hurt performance. And when you measure how well an SID preserves an item's local semantic neighborhood, the winning method flips depending on whether you use Jaccard or RBO. The lesson from What Makes a Good Semantic ID for Generative Recommendation? A Reproducibility Study is that good SID design isn't about chasing one metric, it's about balancing several at once. URL: #RecSys# #Reproducibility#

DeSci: funding, governance, incentives An overview of how on-chain coordination can support peer review, reproducibility, and shared ownership of research outputs. Read more 👇

# Decision Points for Embedding AI Agents in Enterprise Systems # Prompt Change Control 🎯 The Hook A prompt change looks like "just tweaking some wording" — but it is actually a configuration change that can fundamentally alter system behavior. You have CI/CD and code reviews for deployments, but can you trace who changed which prompt, when, and why? If not, you are running production without change management on one of your most critical system components. The realistic approach is not uniform strictness — it is tiered control based on production impact 🔑 📋 Overview Prompt change control is the dial governing how much approval, testing, and version management you require for changes to system prompts and tool definitions. Stricter settings improve reproducibility and safety but slow development velocity. Looser settings enable rapid iteration and experimentation but increase the risk of unintended changes reaching production. The key insight is that different prompt components carry different risk levels, so control levels should vary accordingly. Role definitions in system prompts and few-shot examples demand completely different governance 📋 🔍 Decision Points This dial branches into three tiers based on production impact: High impact (customer-facing, financial, legal) → Strict: code review + 100% eval pass + 2 approvers + canary deploy Medium impact (internal operations, broad scope) → Standard: code review + 95% eval pass + 1 approver Low impact (internal experiments, limited release) → Lightweight: self-review + basic eval pass + change log Further, control policies should vary by prompt component: System prompt (role, constraints) → Low change frequency but high risk. Manage strictly as a design decision Tool definitions (names, descriptions, schemas) → Directly affects tool selection accuracy. Strict control required Few-shot examples → Medium risk. Standard control with eval quality verification Context injection templates → High change frequency, low-to-medium risk. Lightweight to standard Output format instructions → Low risk, but verify alignment with downstream systems ⚡ 💡 Key Details Five components of a control framework: Version control — Manage prompts in Git just like code. Enables diff visualization and history tracking. If you cannot trace "who changed what and when," future decision-making becomes impossible. Always record the reason for each change. Eval gate — Make passing the existing eval suite a deployment prerequisite for any prompt change. Measure eval regression detection rate (the percentage of prompt-induced quality drops caught before deployment) and add missed cases to strengthen the eval suite. Approval process — Reviewer tiers scaled to impact level. Strict tier requires peer engineer plus tech lead. Changes to financial thresholds (like refund limits) add legal review. Canary deploy — Apply the new prompt to a subset of traffic (roughly 10%) and compare quality metrics over 24 hours before full rollout. This can share infrastructure with A/B testing systems. Rollback procedure — The ability to instantly revert to the previous version when issues arise. Git revert integrated with the deployment pipeline is the baseline 🔬 Five metrics to track: prompt change frequency (per environment), change-to-deploy lead time (target 1-3 business days for strict, within hours for lightweight), incident count caused by prompt changes, eval regression detection rate, and rollback rate 📈 ⚖️ Trade-offs Prioritizing development speed (relaxed) enables rapid prompt improvement and fast A/B test cycles. But "who changed what when" becomes untraceable, making root cause analysis during incidents extremely difficult. Unintended changes can slip into production causing quality degradation or security issues. If system prompt role definitions change without anyone noticing, the blast radius covers every single response 😩 Prioritizing reproducibility and safety (strict) makes all changes traceable and simplifies incident root cause identification. But the approval process becomes a bottleneck, extending improvement lead times. When even small improvements require heavy process, team motivation drops and you end up with "I want to fix this prompt but it is too much hassle" — the exact opposite of what you intended ⚠️ The solution is clear: do not lower the control level itself. Instead, shorten change lead time through eval automation and parallel approval workflows. And explicitly separate control levels between experimental and production environments — never trade production safety for experimentation speed. 🛠️ Use Cases Zendesk customer support agent: System prompt changes require PM plus engineering lead approval. Refund limit wording changes add legal review. Full rollout only after eval pass plus canary (10% traffic for 24 hours). Change lead time is 1-3 business days 📞 Slack internal experiment bot: Developers can change prompts via self-review. Basic eval pass is mandatory but approval process is skipped. Change logs are automatically recorded for incident tracing. Change lead time is within hours 💬 Salesforce sales support agent: Tool definition additions and changes require code review. Prompt changes affecting deal stage classification logic need tech lead approval. Large-scale prompt changes are rolled out incrementally to limit blast radius 🎯 Practical tip: Start with strict-leaning controls and relax gradually as the team matures and evals become comprehensive. When a prompt change causes an incident, add that case to your eval suite to automate prevention. And never execute large prompt changes (like rewriting role definitions) all at once — incremental rollout makes impact identification far easier 💪 #AIAgents# #EnterpriseArchitecture#

Question for the LLM Research Community: Is anyone aware of fully reproducible experimental results showing that MLA beats GQA under the same KV cache? I ask for two reasons: (1) I find it surprising that there is still a divide between Chinese labs using MLA, and Western labs using GQA + sliding window, when certainly many ablations have been run by many parties. (2) At Marin we are considering MLA for our next large scale run, but in early (!) smaller scale ablations it appears worse than heavily tuned feature-rich GQA, even after controlling for KV cache. I would like to run better experiments here. Below I cover my thoughts on general reproducibility, then specifics on MLA. Every empirical result in ML is only contextually true. Conditioned on the data distribution, optimizer settings, model width, model depth, finer architecture details, hardware, kernel engineering, initialization, token count, tokenizer, context length, and evaluation protocol, one can reach different conclusions. Contextual results are still useful. Typically if I see a promising method, I will first attempt a full 'context jump', where I apply it to my own context, hoping results transfer. Sometimes they do. If they don't, I can try 2 things: modify the implementation of the method, or modify the context. Ideally I have access to the full context of the original result. Then I can perform a 'context bridge', where I ablate one aspect of the context at a time, isolating exactly why a method performs differently. This lets me make an informed decision to either update my context to let the method shine, or stick with my context and leave the method out. MLA is tricky to assess at small scale. A core aspect of MLA is compressing hidden_dim->latent_dim. Then for each head, latent_dim->head_dim. Typically head_dim is fixed at 128, partially for hardware reasons, and partially for learning dynamics (head_dim of 8 wouldn't have sufficient representational capacity). To get MLA dynamics, you want hidden_dim>>latent_dim, and latent_dim>head_dim. This window closes at small scale. The degree of tuning can unfairly alter the scales. In GQA we have partial RoPE, QK Norm, Gated Attention, attention sharpening, sliding window, and other techniques that give a 30%+ training boost. They don't seem to give the same boost to MLA. On one hand, you want to compare techniques apples:apples with equal tuning. On the other hand, there is a finite amount of future tuning you can do, so prior tuning influences which approach is most pragmatic. Creating controlled tests between MLA and GQA is tricky. Several factors: kv_cache, quadratic attention flops, attention projection flops. kv_cache is controlled by scaling down kv_heads to match MLA, or scaling up kv_latent to match kv_heads. quadratic attention flops are controlled by scaling up GQA's query head count to match MLA head count, or scaling down MLA head count. Also scaling up GQA head_dim 128->192, or scaling down MLA head_dim to 192->128. In general, it's informative to context match to both option A's preferred context and option B's preferred context. Sliding window is another confounder. MLA is theoretically elegant, if we ignore RoPE. It replaces the 'replicate' op of kv_heads in GQA with a 'mix' op (pic below). Since the 'mix' can learn to 'replicate' if it wants, MLA is purely more expressive, and the cost of 'mix' is hidden at inference with absorb trick. Yet in practice, I find that at small scale this 'mix' op doesn't add much value and interacts poorly with the optimizer dynamics. And the change to RoPE hurts. My current plan is to first tune and ablate our model features around MLA, then run 3 scaling ladders: MLA, GQA with 2 kv_heads, and GQA with higher kv_heads. For each ladder, fit a loss vs compute projection. If MLA performs worse at our target compute compared to both GQA options, drop it. If MLA beats 2 kv_heads but loses to higher KV_heads, then it becomes a kv_cache tradeoff. Early results indicate MLA will perform worse than both feature-rich GQA ladders, but we will see. Any positive external reproducible results for MLA would help make sure I give it the best chance possible.

AI Agents in DeSci: The End of the Grant Era? Traditional science funding is broken. Grants are distributed slowly, and bureaucracy stifles innovation. We are building a more efficient system. The Solution: Decentralized Science (DeSci), governed by AI agents. How it works: AI agents are not just assistants. They become PIs (Principal Investigators) in DeSci DAOs. Evaluation: Agents analyze scientific proposals in milliseconds, using verifiable logic (hello AX Pillars), not subjective opinions. Distribution: Grants are distributed automatically based on the on-chain reputation of agents and predicted impact. Coordination: Agents coordinate cross-disciplinary research, verifying each other's data for hallucinations and ensuring reproducibility. The future is not about making people write better grants. The future is about letting AI agents drive the scientific process. Are you ready to let an AI agent peer-review your data?

# Codex Features and Practical Usage ☁️ Codex fixes repos you haven't even cloned locally, working on them in parallel in the cloud and turning the diffs into PRs. A way of working that never ties up your own machine. 🏷️ Title: Codex Cloud 🔗 URL: 📘 Overview Codex Cloud lets you delegate tasks to the cloud, where Codex works in the background in its own environment. It can run multiple tasks in parallel, and when you delegate work it clones your repository into that cloud environment. When done, it turns the diff into a pull request automatically. ⚙️ How It Works ・Tasks can be delegated from several entry points: the web platform, the IDE extension, and GitHub integrations, all running in parallel inside cloud environments. ・On delegation, the repo is cloned into the cloud environment, giving an isolated setup free of local-machine dependencies. ・You shape the environment with setup scripts (initialization before a task runs), environment configuration (repo selection, tools, dependencies), and internet access controls (whether public networking is allowed). ・Results become PRs automatically, linked to the originating GitHub issue, or you can apply the diff locally through the IDE extension. ・Where the local CLI runs on your machine with direct filesystem access, Cloud offloads execution to managed containers, giving you parallelization, background processing, and lower local resource use. 🛠️ Practical Usage ・Go to and connect your GitHub account to enable repo integration and PR creation. ・For reproducibility, define dependency installs and build prep in the environment's setup script. Enable internet access selectively, only for tasks that truly need external resources. ・Keep prompts clear and constrained with the right level of detail rather than vague. 💡 Use Cases Fire off several independent bugs or small features as concurrent cloud tasks, then review the resulting PRs one by one for parallel development. Even for a repo you've never set up locally, a working setup script lets Codex start immediately. ⚠️ Caveats ・Cloud execution runs in containers with defined resource limits and cannot reach local-only tools or private networks without explicit configuration. ・Always review the diff before merging an auto-generated PR. ・Enable internet access only when needed, in line with your security requirements. #OpenAICodex# #CodexCloud#

A pipeline with 100% delivery rate, 100% schema validity, and zero retries or errors... yet re-running the exact same request flips the verdict. This paper reports that shocking negative result, fully preregistered with a complete audit trail. Title: Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints URL: 🔧 Highlight 1: Clean engineering does not mean reliable measurement Across 3,312 planned calls, response rate and schema validity both hit 100% with zero retries or errors, yet the repeat-ranking agreement (median Spearman) was only 0.400 — clearly failing the preregistered stability gate (threshold 0.90). 🎲 Highlight 2: Byte-identical inputs still drift Replaying the exact same request 24 hours later gave only a 0.780 exact-ranking agreement rate. Interestingly, comparisons across windows on the same day scored a nearly identical 0.805 median, revealing this isn't "next-day drift" but immediate platform-level nondeterminism. 📉 Highlight 3: More samples don't fix it Scaling observer calls from 8 to 500 (748,000 calls total) still left the stability gate pass rate at 0/500. The root cause: score gaps between candidates were 7-10 orders of magnitude smaller than the noise floor, making most tasks fundamentally unrankable. I think this is a great reminder that before using an LLM as a measurement instrument, you need to measure whether the instrument itself is stable. #LLMEval# #Reproducibility#

# Decision Points in AI Agent Development # Temperature 🎯 The Hook Are you using the same temperature for every task in your agent? Temperature isn't just a "creativity knob." In agent systems, structured outputs, tool calls, and user-facing responses each need fundamentally different temperature settings. Using one value for everything is leaving performance on the table. 📋 Overview Temperature controls how "peaked" or "flat" the probability distribution is when an LLM selects its next token. Near 0, the highest-probability token wins almost every time, producing deterministic and stable output. Higher values flatten the distribution, allowing lower-probability tokens through, increasing diversity and creativity. In AI agent systems, the optimal temperature varies dramatically across contexts: generating structured output, assembling tool call arguments, and producing free-form text each call for different settings. Temperature should be treated as a dynamic variable that shifts with task type, not a single fixed constant. 🔍 Decision Points Temperature is primarily driven by task_variability -- how routine vs. creative the task is. The decision flow is straightforward 🧭 1. Structured output (JSON / function calling)? → 0.0-0.3 2. Accuracy-first (fact extraction, classification, summarization)? → 0.2-0.5 3. Dialogue, explanation, communication? → 0.5-0.7 4. Creative writing, brainstorming, candidate generation? → 0.7-1.0 Additionally, higher failure_cost pushes the temperature ceiling down, and high cost_sensitivity environments should account for retry cost increases from higher temperatures. 💡 Key Details Reference values by task type 📊 - Structured output (JSON / function calling): 0.0-0.3. Minimizing schema violations is the priority - Classification, extraction, data transformation: 0.0-0.2. Accuracy and reproducibility are paramount - Summarization, explanation, customer support: 0.5-0.7. Balance naturalness with accuracy - Creative writing, brainstorming, candidate generation: 0.7-1.0. Diversity is the source of value - Tool argument generation: 0.0-0.2. Precise function and argument names are non-negotiable - Planning and reasoning: 0.3-0.6. Some exploration helps, but maintain logical consistency Start structured output temperature at 0. If schema violations occur at 0, the problem is your prompt or schema -- never rely on higher temperature to "accidentally" produce correct output 🚫 ⚖️ Trade-offs Too low and conversations become robotic 🤖 The model returns identical answers to identical questions, giving users a "template response" impression. Best-of-N sampling also breaks down -- candidates become near-identical, costing N times more for essentially N=1 results. Too high and structured outputs start breaking 💥 JSON field names drift, types mismatch, hallucinations increase -- especially dangerous for proper nouns, numbers, and dates. Tool call instability and loss of reproducibility compound the problem. Monitor the retry cost impact of temperature changes. If schema violation rates exceed roughly 5%, consider lowering the temperature. 🛠️ Use Cases Vary temperature by pathway within a single agent 🔀 Planning steps at 0.3-0.5, tool argument generation at 0.0-0.2, user-facing responses at 0.5-0.7. When switching models, adjust temperature simultaneously for a natural fit. Using Best-of-N? You need to raise the temperature. Generating N=5 candidates at temperature 0 produces 5 near-identical outputs. For N>1, set temperature to 0.5-0.8 and let a Judge select the best from diverse candidates. Be careful combining temperature with top_p ⚠️ Adjusting both simultaneously creates multiplicative effects with unpredictable behavior. As a rule, tune one and leave the other at its default. #AIAgents# #SoftwareArchitecture#

Iteration speed is all you need. In frontier AI, the fastest team from hypothesis to trustworthy evidence wins. That takes three things: Scale. More data, compute, and experiments should be a config change — not an infra rebuild. Speed. Processing, training, deployment, and evaluation have to be fast enough that every result actually shapes the next decision. Attention. Researchers' time should go to ideas and judgment — not moving data, hunting GPUs, recovering jobs, or reconstructing results. Knowing that more data and compute produce better models isn't enough — you have to learn fast enough to shape the next run. Robotics raises the bar further. Collecting data, training a policy, and shipping a demo is step one. The hard part is doing that on repeat — across massive multimodal datasets, multiple robot configs, distributed compute, and rigorous real-world evaluation. Robot data isn't just text or images. It's synchronized video, actions, proprioception, poses, language, sensors, hardware state, and physical outcomes. And training is only the middle of the loop: the model has to return to the robot, the robot has to generate evidence, and that evidence has to drive the next experiment. At Dyna Robotics, we're hiring for exactly this: ML Infrastructure Engineer, Training Build the research operating system behind our models — distributed training, high-throughput data loading, scheduling, recovery, reproducibility, serving, and evaluation. Software Engineer, Data Infrastructure Turn massive robot experience into reliable, inspectable, training-ready data. These aren't support roles. They set the ceiling on how fast we scale, learn, and improve. A demo proves a possibility. Infrastructure compounds real-world progress. If you are enthusiastic about this, please contact me or directly apply! Let's land cutting-edge research into the real world together!