Run inference over millions of records — free of SQL, and without your data ever leaving Snowflake. Here's distributed batch inference at scale ⚙️ Title: Batch Inference at Scale URL: ⚙️ Overview A capability that runs distributed inference workloads on Snowpark Container Services (SPCS) with Ray as the execution framework. Inference runs as a dedicated distributed workload, supporting both traditional models and LLMs, consolidating complex operations into a single API call. ❓ Challenges Solved Many customers, especially those migrating from non-SQL systems, need batch inference decoupled from SQL. ・This is especially true for files and unstructured data at large scale ・Rearchitecting workflows around SQL-first patterns is a heavy burden 💡 Methodology & How It Works ・The input DataFrame is materialized and written to a stage as Parquet files ・A job is provisioned on SPCS; the primary node initializes as the Ray head and replicas join as workers ・Each worker reads staged data, performs inference independently, and writes results to an output stage ・Unified API: a single run_batch() call handles both structured and unstructured data ・Multimodal support (images, audio, video); workers load weights once and reuse across batches; JobSpec controls workers and GPU allocation 🌍 Use Cases ・Nightly summarization of millions of support tickets ・Product catalog enrichment via image-to-text generation ・Information extraction from scanned PDFs, audio transcription and labeling, video classification and description BatchInferenceTask integrates with Snowflake Tasks for DAG automation, and all processing stays inside Snowflake — running large-scale inference while preserving data governance. #Snowflake# #BatchInference#

# Decision Points for Embedding AI Agents in Enterprise Systems # Online vs Offline Intelligence — When to Run Inference 🎯 The Hook Are you trying to run all your agent's inference in real time? Online (real-time at request time) and offline (pre-computed in batch) inference have fundamentally different profiles across latency, cost, and accuracy. Most production systems use both, with a hybrid architecture where offline pre-computes results and online applies the finishing touches 🔑 📋 Overview Online inference is essential for interactive use cases where user input cannot be predicted in advance — a Zendesk live chat agent answering in real time, or a Salesforce agent advising sales reps on next actions from an opportunity screen. Offline inference excels when processing large volumes of data on a regular schedule — analyzing all employee skill data overnight, or batch-optimizing all product descriptions for SEO monthly. In enterprise environments, mixing both according to workload characteristics is the standard approach 📊 🔍 Decision Points Two axes drive this decision: immediacy requirements and processing volume/cost. If the response must arrive within seconds, online is non-negotiable. Users abandon when latency exceeds roughly 30 seconds. If the processing target is hundreds of items or more as a batch, consider offline. Processing them one by one in real time makes costs prohibitive. If per-item inference cost is high (roughly $0.05 or more per item), offline batch processing is typically more cost-efficient. If the results can tolerate a staleness of several hours, offline becomes a strong candidate. If you need characteristics of both, adopt a hybrid: pre-compute offline, then apply differential corrections online ⚡ 💡 Key Details Three hybrid patterns to understand: Offline pre-compute, online finish — Generate product recommendation candidates in a nightly batch, then adjust the final ranking online based on the user's real-time behavior. Heavy inference is done offline; online handles only lightweight adjustments, balancing latency and cost. Online results aggregated offline — Analyze real-time customer interaction logs in offline batches to auto-update FAQs and improve response templates. Online experience feeds back into offline quality improvement — a virtuous cycle. Semantic cache layer — Accumulate responses for frequent queries in a semantic cache and serve similar queries from cache to reduce online costs. The key is matching on semantic similarity, not exact string matching 🔄 ⚖️ Trade-offs Resist the temptation to make everything online. Running large-scale data analysis synchronously pushes latency past 30 seconds, causing user abandonment and hitting API timeouts. Worse, failing to estimate monthly costs of routing all requests through an LLM leads to post-launch budget surprises 😩 On the flip side, offline freshness management matters. When batch execution frequency does not match data update frequency, you serve stale results. Showing inventory status with a one-day delay has real business impact. Without a mechanism to invalidate offline computation results when master data changes, data consistency breaks down. Offline batch error handling is another blind spot. If a batch fails midway and only partial results are reflected, data consistency is compromised ⚠️ 🛠️ Use Cases Customer support agent (Zendesk integration): Live chat answers are generated online in real time. Meanwhile, a nightly offline batch extracts frequent patterns from past interaction logs and updates the FAQ. Online response quality is continuously improved by offline analysis — a feedback loop 📚 Skill analysis agent (Workday integration): Analyze all employee skill data in a nightly offline batch to generate skill gap reports. When a manager asks online "Who are the best internal candidates for this position?", the agent responds instantly based on pre-computed skill matching results 🏢 Product recommendation agent (Shopify integration): Pre-generate product similarity calculations and per-segment recommendation candidates offline. Adjust the final recommendation ranking online in response to the user's real-time browsing behavior 🔧 Practical tip: Start by defining the target response time (P95 latency) for online interactions. Any processing that exceeds that target should be carved out to offline. Always estimate costs on a monthly basis and optimize the online-to-offline ratio accordingly 💪 #AIAgents# #EnterpriseArchitecture#

introducing preemptible compute for together gpu clusters same nvidia gpu infrastructure, 50% of the on-demand price built for evals, fine-tuning, batch inference + short experiments, with up to 5 minutes to checkpoint before reclamation now in public preview

The Compute Desk Nvidia B300 GPU-hour index is at its all-time high. As margins on training grow thinner, neoclouds are collectively shifting focus to inference. High-priced B300s are generating ROI by lowering the marginal cost per token generated through batch inference.

$SKHY Is Expanding Beyond HBM SK Hynix is developing Processing-in-Memory (PIM), which processes data inside or near memory to reduce data movement, latency, and power consumption The company claims PIM can deliver 300× the memory capacity of SRAM in the same chip area and is exploring hybrid bonding to stack compute and memory dies without sacrificing density Meanwhile, High Bandwidth Flash (HBF) combines 3D NAND density with HBM-style interconnects to deliver higher capacity at lower cost, targeting long-context and large-batch inference SK Hynix is also developing SALT-KV, software that distributes KV cache across HBM, DRAM, and SSDs to reduce dependence on expensive memory