Register and share your invite link to earn from video plays and referrals.

Search results for Datastream
Datastream community
One keyword maps to one global community path.
Create community
People
Not Found
Tweets including Datastream
As we close out an incredible Snowflake Summit '26, here’s what’s new across the AI Data Cloud. Watch the June Snow Report and catch up on everything announced this month! Now generally available: • Snowflake CoWork – a personal work agent and control plane that understands your enterprise data and context, helping knowledge workers research, reason, and take action across business tools. • Snowflake CoCo – Snowflake’s AI coding agent for the enterprise, enabling teams to build, connect, and automate data workflows and AI applications from desktop, mobile, or Slack. • Snowflake Datastream – a fully managed streaming service for Apache Kafka that brings real-time data and AI together on a single governed platform. • Horizon Catalog – an agentic catalog that connects your entire data estate, making it easier to discover, govern, and trust data across clouds and engines. • Horizon Context – a governed semantic layer that ensures people, tools, and agents operate from the same trusted business definitions. • New AI Security Innovations – delivering granular controls for AI agents and enterprise-grade security across your AI data estate. Missed a keynote or technical session? They’re now available on demand, including Sridhar Ramaswamy’s mainstage conversation with Anthropic CEO Daniela Amodei. Upcoming events: • June 18 — Go from prompt to production and build a Streamlit app with CoCo • June 23 — Build batch data extraction pipelines at scale with Cortex AI Functions • June 24 — See Snowflake Postgres in action: one platform, zero pipelines • July 1 — Modernize SQL Server ETL with an AI-assisted migration using CoCo • July 9 — Build a fully working AI agent in just 90 minutes More detail:
Show more
# Elasticsearch Features and Practical Usage 🌊 The era of manually rotating ` is over. With data streams your app just keeps writing to the same name, and all the backing-index management is handled for you. 🏷️ Title: Data stream (logical stream for append-only time-series data) 🔗 URL: 📘 Overview A data stream is an abstraction layer that lets you address a set of indices, optimized for append-only time-series data, under a single name. It suits continuously flowing data like logs, events, and metrics — you read and write to one resource name without worrying about the hidden "backing indices" behind it. ⚙️ How It Works ・A data stream is made of multiple auto-generated, hidden backing indices. The naming is `.ds--<>-`, where the generation is a six-digit, zero-padded integer starting at `000001` (e.g. `.ds-logs-myapp-prod-2026.06.10-000001`). ・Only the most recent "write index" accepts new documents. You cannot write directly to older backing indices. ・Searches automatically route to all backing indices, so queries span the entire dataset. ・Every document needs an `@timestamp` field mapped as `date` or `date_nanos`. If the template omits it, a default `date` mapping is applied automatically. ・A matching index template containing a `data_stream` definition is mandatory; it holds the mappings, settings, and lifecycle policy. One template can be shared across multiple data streams. ・When an age or size threshold is hit, a "rollover" creates a new backing index and switches the write index. This is automated via ILM or data stream lifecycle. 🛠️ Practical Usage First, define an index template that includes `data_stream`. `PUT _index_template/logs-myapp-template` `{ "index_patterns": ["logs-myapp-*"], "data_stream": {}, "template": { "mappings": { "properties": { "@timestamp": { "type": "date" } } } } }` Then your app simply keeps writing to the same name. `POST logs-myapp-prod/_doc` `{ "@timestamp": "2026-06-10T09:00:00Z", "level": "INFO", "message": "started" }` 💡 Use Cases A common setup is writing application logs to a `logs-myapp-prod` data stream and letting backing-index rollover be managed automatically. The app no longer rotates date-based indices (` itself; it just POSTs to the same endpoint every time. ⚠️ Caveats ・Backing index names are an internal implementation detail. They can change during restore or shrink, so never build logic (like dates) off the names. ・It is a poor fit for frequently overwriting the same ID (last-write-wins); for that, use a regular index or an alias. ・Use dedicated APIs like `update by query` and `delete by query` for modifications, and remember you cannot write directly to anything but the write index. ・You cannot delete an index template that is in use by a data stream. #Elasticsearch# #DataStreams#
Show more
Data stream lifecycle finally reaches the frozen tier. frozen_after sits next to data_retention, indices get snapshotted to object storage and stay queryable.
Worth a listen if you care where Agentic DeFi is going. @kukulabanze drops some alpha on @stabledash. "What we've really been building with our data science team since 2021 is the defining DeFi data stream"
Show more
# Elasticsearch Features and Practical Usage 📦 No need to lock down a table schema first — the moment you throw JSON at it, your data is usable. An Elasticsearch index is a document-oriented data store built for search and scale from day one. 🏷️ Title: Index / Document-oriented data store 🔗 URL: 📘 Overview An index is the fundamental unit of storage in Elasticsearch and the level at which you interact with your data. Data is stored one record at a time as JSON "documents," and each document is a set of field key-value pairs plus system metadata such as `_index`, `_id`, and `_version`. ⚙️ How It Works ・Your actual data lives in the `_source` field, while `_index` (owning index), `_id` (unique ID), and `_version` are system-managed metadata. ・A "mapping" defines each field's type and how it is indexed and queried. You pick types like `text` (for full-text search), `keyword` (exact match and aggregations), `integer`, and `date`. ・Even without an explicit mapping, "dynamic mapping" infers types from the incoming JSON, so you can add new fields later and still index them with no schema migration. ・Internally, an index is split into "shards" distributed across nodes. Data inside a shard is written as immutable "segments," and replica shards provide redundancy and scale. ・Settings like `index.number_of_shards` (fixed at creation), `index.number_of_replicas` (adjustable later), and `index.refresh_interval` (default 1s) control index behavior. 🛠️ Practical Usage Indexing a single document is straightforward. `POST products/_doc/p-1001` `{ "name": "Wireless earbuds", "price": 8900, "stock": 120, "category": "audio" }` For large loads, the `_bulk` API batches many operations in one request. `POST products/_bulk` `{ "index": { "_id": "p-1001" } }` `{ "name": "Wireless earbuds", "price": 8900 }` `{ "index": { "_id": "p-1002" } }` `{ "name": "USB-C cable", "price": 1200 }` Adding a brand-new field later (e.g. `sustainability_score`) just works thanks to dynamic mapping. 💡 Use Cases A classic pattern is modeling an e-commerce product catalog as a `products` index, one product = one JSON document (name, price, stock, category, description). A daily batch loads hundreds of thousands of records via `_bulk`, and you can introduce new attributes without waiting on an RDB schema change. ⚠️ Caveats ・A field's type cannot be changed once set. To change a type you must reindex into a new index. ・Use a regular index for frequently updated documents; use a data stream for append-only time-series data. ・Shard count and size directly affect query speed and cluster stability. Avoid huge numbers of tiny shards and aim for sensible shard sizing. #Elasticsearch# #DataModeling#
Show more
At Flink Forward Asia Shenzhen 2026, NVIDIA’s Chuan Chen shared how NVIDIA and Alibaba Cloud accelerate multimodal data stream processing for Apache Flink: “NVIDIA and Alibaba Cloud's team technically collaborate to enable the CUDA library-accelerated multimodal data stream processing of Apache Flink.” This open-source collaboration enables end-to-end, high-performance multimodal streaming architectures for AI commentary, live image-text feeds, and interactive Q&A. #NVIDIA# #AlibabaCloud# #ApacheFlink# #DataAI# #AI# #Multimodal# #RealTimeStreaming#
Show more