The video of the Second CITAC Web Seminar: Award Ceremony of the CITAC Best Paper 2021, held on 2022/06/21, is available on the CITAC Youtube Channel: #CITAC# #metrology# #comparability# #qualityassurance# #analyticalchemistry# #measurementuncertainty#

The Eurachem/CITAC guide "Assessment of performance and uncertainty in qualitative chemical analysis" has been published. We believe this guide will be useful for many analysts. #qualitativeanalysis# #uncertainty# #validation# #qualityassurance#

Harness Engineering Practices P7. Negative Verification — Regression and Blast Radius Gates 🎯 Point Agents optimize for "my change works" and underweight "I haven't broken anything else." Positive verification alone won't catch escaped defects. 📝 Overview Add blast radius checks to completion gates — beyond running the full existing test suite, verify "who imports the symbols I touched." Negative verification confirms not just that your code works, but that nothing else is broken. 🔍 Explanation Agents naturally focus on "tests related to my change pass." But change impact doesn't stop at the changed code. Altering a function signature breaks callers; changing shared module behavior affects all dependents. Negative verification is the explicit mechanism for verifying "nothing was broken." Identify import sites of changed symbols, run their tests too. Embedding impact visualization and test execution into the completion gate structurally prevents escaped defects. 🛠 How to Practice - Integrate tooling into completion gates that auto-identifies import sites of changed symbols via static analysis (import analysis, call graphs) - Automatically add identified dependent tests to the execution set alongside the existing test suite - Visualize change blast radius (impacted file count, module count) alongside the diff and present it to reviewers - Combine coverage data with static analysis to identify and flag under-tested impact areas 💼 Use Cases - Issue-to-PR agents: auto-run all dependent tests when shared utilities are modified - Migration agents: verify the full impact zone of API signature changes - Legacy code modernization: cover change ripple effects with characterization tests ⚠ Pitfalls Exhaustive blast radius checking can spiral into running the entire monorepo test suite, which is impractical. Combining static analysis (import analysis, call graphs) with dynamic analysis (coverage data) is most effective. Also, negative verification existing doesn't mean you can neglect positive verification — both are necessary. #HarnessEngineering# #QualityAssurance#

A Rockstar Games quality assurance analyst reveals the intense working conditions as development on Grand Theft Auto 6 enters its final stages. In an anonymous review, the employee described how normal tasks are now compressed from 5 or 6 months into just 2 or 2 Overtime has become the expectation in recent weeks, with some team members staying until 3 AM. The analyst claimed significant impacts on physical and mental health despite raising concerns with management. This comes as Rockstar aims for a November 19, 2026 release date for GTA 6. The company has a documented history of crunch culture, most notably during Red Dead Redemption 2. After widespread criticism, Rockstar pledged to improve work-life balance on future titles, and the game was delayed in part to reduce extreme overtime. GTA 6 remains one of the most highly anticipated games in history, but this is what usually happens in massive productions like this.

# Practical and Useful Patterns with ADK 📏 What you measure determines the quality of your evaluation. ADK's Evaluation Criteria provide built-in metrics covering tool trajectory accuracy, hallucination detection, safety assessment, and multi-turn task success. 📌 Title: Evaluation Criteria 🔗 URL: 🧩 Overview ADK provides multiple built-in evaluation criteria, all returning scores normalized to 0.0-1.0. **tool_trajectory_avg_score** verifies agent tool call sequences. **response_match_score** measures response similarity via ROUGE-1. **hallucinations_v1** validates response grounding against context. **safety_v1** assesses harmful content. **multi_turn_task_success_v1** measures task completion across conversation turns. 🛠 Usage Each criterion is configured in a JSON configuration file. **tool_trajectory_avg_score** — choose from three match types: ```json { "criteria": { "tool_trajectory_avg_score": { "threshold": 1.0, "match_type": "EXACT" } } } ``` - `EXACT`: Perfect match required — no extra or missing tool calls - `IN_ORDER`: Expected tools must appear in sequence, but other calls between them are allowed - `ANY_ORDER`: All expected tools must be present, regardless of order **response_match_score** — ROUGE-1 unigram overlap: ```json { "criteria": { "response_match_score": 0.8 } } ``` **hallucinations_v1** — validates each sentence against provided context: ```json { "criteria": { "hallucinations_v1": { "threshold": 0.8, "judge_model_options": { "judge_model": "gemini-flash-latest" }, "evaluate_intermediate_nl_responses": true } } } ``` Each sentence receives a label: supported, unsupported, contradictory, disputed, or not_applicable. The score is the percentage of sentences labeled supported or not_applicable. **safety_v1** — evaluates for hate speech, harassment, and dangerous information: ```json { "criteria": { "safety_v1": 0.8 } } ``` **multi_turn_task_success_v1** — measures task achievement across a full conversation: ```json { "criteria": { "multi_turn_task_success_v1": 0.8 } } ``` 🏗 Practical Patterns **Strict Tool Trajectory Verification**: In domains like finance or healthcare where tool call order matters — such as "authenticate first, check balance, then transfer" — use `EXACT` match to ensure the agent follows correct procedures. **Flexible Tool Verification**: For exploratory tasks, use `ANY_ORDER` to verify all necessary tools were called. `IN_ORDER` fits cases where main step order must be preserved but auxiliary tool calls between them are acceptable. **Hallucination Detection in Practice**: Use `hallucinations_v1` with RAG-based agents to automatically detect when agents generate information not grounded in retrieved results. Enable `evaluate_intermediate_nl_responses` to validate not just final answers but intermediate responses too. **Balancing Safety and Task Success**: Combine `safety_v1` with `multi_turn_task_success_v1` to catch agents that are "safe but ineffective" or "effective but inappropriate." 💡 Use Cases 🎯 Strict tool call order verification for finance/healthcare agents 🔍 Automatic hallucination detection in RAG agents 🛡 Safety evaluation for customer support agents 📈 Quantitative task completion rate measurement in multi-turn dialogues ⚠️ Caveats - `safety_v1`, `multi_turn_task_success_v1`, `multi_turn_trajectory_quality_v1`, and `multi_turn_tool_use_quality_v1` require Google Cloud Project setup (`GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` environment variables). - LLM-based criteria can vary between judge model runs. Set `num_samples` for multiple sampling with majority voting to stabilize results. - `response_match_score` uses ROUGE-1 (word overlap only) and does not evaluate semantic equivalence. Consider `final_response_match_v2` for semantic evaluation. ✨ Choosing the right evaluation criteria is the foundation of agent quality management. Match EXACT/IN_ORDER/ANY_ORDER to your domain requirements, and combine hallucination detection with safety assessment for comprehensive quality assurance! #ADK# #AIAgent#

# Learning Palantir Foundry 🚀 Bring complex logic that no-code can't reach into your data platform, along with full software-engineering quality control. That's what Code Repositories delivers. 📌 Title and Feature URL Title: Code Repositories (Python Transforms) URL: 📝 Overview Code Repositories is a web-based integrated development environment (IDE) for creating and collaborating on production-ready code within Foundry. It provides a friendly UI over the underlying Git repositories, so teams can work without command-line access. With platform-specific features, you can apply software development practices directly to data engineering. 🔧 How It Works Version control and collaboration are at its core. - Common Git tasks (branching, committing, release tagging) execute through the web UI - Pull requests drive code review, with "highly configurable" permissions that support quality assurance such as mandatory reviews - IntelliSense, linting, error checking, and contextual help dialogs are available across all repository types - Transforms repositories let you author data transformation logic in Python, Java, or SQL with preview and debugging - Functions repositories natively integrate the Ontology and run low-latency business logic in TypeScript or Python 🛠 Practical Usage - Use PySpark to implement billion-row entity resolution and complex business rules in code - Require PR reviews so a second reviewer and CI checks must pass before merge - Add unit tests to guard transform logic against regressions - In Functions repositories, leverage Ontology-data-type autocomplete to write logic safely - Bring machine learning workflows into the platform via model development repositories 🎯 Use Cases - Implementing complex reconciliation and business rules in PySpark that Pipeline Builder can't express - Structurally eliminating "regressions from editing production directly" through mandatory reviews and branch-based workflows - Implementing derived KPIs and validation logic as Functions reused across apps - Managing ML model training and inference code under governance ⚠️ Caveats - The docs note that Japanese translations are machine-generated and unverified, so localized content may have accuracy limitations - Each repository type (Transforms/Functions/Model) supports different languages and purposes, so pick the one that fits your goal - Being a pro-code environment, the quality benefits only materialize if your organization establishes review, CI, and test practices #PalantirFoundry# #DataEngineering#

# Decision Points in AI Agent Development 🎯 The Hook Do you validate every agent output before it reaches the user, or let it through and audit a sample afterward? "Check everything for safety" sounds right, but adding semantic validation with another LLM call effectively doubles your cost and latency per request. On the flip side, deciding "post-hoc is fine" can mean hallucinations reach users and become incidents. Here's how to find the right balance. 📋 Overview This is a design decision about when to validate AI agent outputs. Inline validation checks every output in real time immediately after generation, blocking anything that fails. Post-hoc / sampling validation lets outputs through first, then audits them in batch or via random sampling. It is fundamentally a trade-off between quality assurance and latency, coming down to whether you prioritize the cost of bad outputs reaching users or the latency budget. 🔍 Decision Points The primary axis is the cost of a bad output reaching the user. 📌 Legal risk, financial loss, or safety concerns → Inline validation is mandatory. Financial advice, medical information, legal documents, externally published content -- when unqualified output reaching users is unacceptable, you need full blocking. 📌 UX degradation or manual correction needed → Inline for high-risk paths only, post-hoc for the rest. Not every output needs the same rigor. 📌 Internal tools, draft generation, exploratory output → Post-hoc sampling is sufficient. Sample 5-10% of outputs for quality monitoring, discover problem patterns, and feed improvements back into prompts and guardrails. The secondary axis is the latency budget. 📌 Sub-200ms response required → Inline validation limited to structural checks (schema validation, range checks). Semantic validation moves to post-hoc. 📌 Standard 1-10s latency → Full inline validation stack (structural + rule-based + semantic) is feasible. 📌 30s+ batch processing → Apply full inline validation on every item, plus deeper post-hoc analysis. 💡 Key Details 🔹 Inline validation works best as a three-tier stack. Tier 1: structural validation (JSON schema, PII detection) at a few milliseconds. Tier 2: rule-based validation (policy engine, OPA/Rego) at 10-50ms. Tier 3: semantic validation (separate LLM for factuality checking) at hundreds of milliseconds to seconds. Not every path needs all three tiers. 🔹 The hidden strength of post-hoc validation is depth. Without real-time constraints, you can run multi-LLM-Judge cross-checks, human reviews, and external data source verification. The real value is the feedback loop: discovering problem patterns in samples and using them to improve guardrails and prompts. 🔹 The critical weakness of post-hoc validation: some outputs cannot be recalled once delivered. Sent emails, external API calls, content already displayed to users -- these are irreversible. If the output triggers an irreversible side effect, post-hoc is too late. ⚖️ Trade-offs Inline validation guarantees safety but adds latency and cost. Applying semantic validation to every output effectively doubles your LLM spend. High false-positive rates (blocking valid outputs) cause retry cascades that degrade UX. Post-hoc validation has zero latency impact and enables deeper analysis, but accepts a "leakage window" where problematic outputs reach users. 🏆 The proven pattern is a hybrid: inline for high-risk paths, post-hoc sampling for low-risk paths. In practice, the most effective structure is "inline (structural + rule-based) + post-hoc (semantic deep-dive)." Structural validation has minimal latency impact, so run it inline on everything. Expensive semantic validation is inline only for high-risk paths; the rest gets sampled post-hoc. 🛠️ Use Cases 🏥 Medical information chatbot → Full inline validation. All three tiers including factuality checking via separate LLM. The cost of hallucination reaching a patient far outweighs validation latency. 💬 Internal Slack bot → Post-hoc sampling. Internal use tolerates some errors. The important thing is the feedback cycle: sample, find patterns, improve prompts. 🛒 E-commerce product description generation → Hybrid. Structural validation (price ranges, required fields) inline; expression appropriateness validated post-hoc. 📧 Automated email replies → Inline validation. Sent emails are irreversible side effects. PII leakage and policy violation detection must happen before sending. #AIAgents# #SoftwareArchitecture#

A teacher types in any topic they're teaching, and an interactive learning simulation gets built on the spot. That's the experiment Google Research is running. Title: The future of practice: Enabling teachers to create learning interactives with generative UI URL: Interactive simulations have always been expensive to build and limited in number, so here are three things that stood out to me. 🧩 On-demand creation via generative UI By combining LearnLM, a model family tuned for education, with generative UI (GenUI) technology, the system builds simulations on the fly for any topic, structured as levels with gradually increasing difficulty. It automatically generates the kind of scaffolding a teacher gives one-on-one: an intro to prime prior knowledge, a toolbox of formulas, tiered hints, and tailored feedback. 🤖 An agentic quality-assurance loop Generated simulations get self-corrected against three criteria: pedagogical (do the levels cover the learning goals), mechanical (do the buttons work, is it solvable), and visual (no unnecessary clutter). What's notably "agentic" here is that it actually opens a Chrome instance to click through and test the simulation. 📊 Strong ratings from real teachers In the UK, 40 STEM learning interactives were rated "good or excellent" overall. In the US, 12 teachers rated their custom-generated interactives an average of 8 out of 10. Over 30 STEM interactives spanning physics, chemistry, biology, and math are already live in the public library. It feels like a genuine attempt to scale the principle that "learning is not a spectator sport" using generative AI. #GenerativeAI# #EdTech#