🎉 Vane 0.1.0 Is Officially Released Vane Data is a high-performance multimodal data engine built for AI workloads. Forked from DuckDB, it provides native multimodal processing and a unified execution model for both local and distributed environments. 🔧 Core Features Distributed DuckDB execution engine Extends DuckDB with distributed physical plans, distributed Plan Fragments, FTE (Fault-Tolerant Execution) scheduling, and distributed operator execution, with Arrow Flight providing cross-worker Exchange/Shuffle data transport. Python UDFs Relation UDFs support row-wise map, Arrow Table-based map_batches, and one-to-many flat_map. Expression UDFs provide @vane.func, @vane.cls, and their corresponding .batch forms. Scalar, batch, and class UDFs can all be registered as SQL functions through vane.attach_function(). AI Functions Provides typed Prompt and Embed APIs across the Python Expression API, Relation API, and SQL. Prompt integrates with OpenAI, Anthropic, Google, and the native vLLM backend, while Embed supports OpenAI, Google, and SentenceTransformers. Structured outputs and image Prompt inputs are available where supported by the provider. Native vLLM batch execution Implements a native Physical VLLM operator and Actor Pool, with bounded task submission enforced through in-flight limits. Prompts are bucketed by shared prefixes and routed to actors using prefix-aware routing to improve opportunities for reusing the vLLM Prefix Cache. The native vLLM Prompt path currently supports text input only. Adaptive multimodal batching and backpressure The UDF and vLLM execution paths dynamically split or combine batches according to row count, data size, and in-flight limits. Resource admission control and object-stream backpressure limit the number of queued tasks and their memory consumption. Fault-Tolerant Execution Supports task retries, Worker failure detection and replacement, Split reassignment, Attempt Fencing, cancellation, and resource cleanup. Ray Runner and Local Runner The same SQL and Relation plan model can run through either the distributed Ray Runner or the local In-Process FTE Runner. Ray Runner is the default execution path and supports both single-machine and distributed execution. Local Runner targets lightweight, lower-overhead local execution without Ray. Local Runner is currently experimental. Multimodal benchmarks Provides comparable Vane, Ray Data, and Daft pipelines covering audio transcription, document embedding, image classification, and video object detection. The current benchmarks use local files on a single-GPU machine. They represent a single-node environment and are not a direct reproduction of the original distributed Ray Data benchmark. 👏 Thank You to Our Contributors @kaka11chen @WangErxi @caomaocao @hubgeter @liwuhen @pollychen-lab @figurant @zy-kkk @suxiaogang223 @jingdaws @freemandealer @liujiwen-up @StanleyXu512 🗺️ Roadmap 1. Distributed Extension for Ray Runner — Implement a distributed extension compatible with the Ray runner, building upon the existing DuckDB extension architecture. 2. Native Multimodal Type Support — Add first-class native type support for multimodal data. 3. C++ Embedded Functions for Multimodal Types — Implement C++ built-in/embedded functions operating on multimodal types. 4. Distributed Lance Read/Write — Enable distributed read and write capabilities for the Lance format. 5. Distributed Iceberg Read/Write — Enable distributed read and write capabilities for the Iceberg table format. 6. Turbopuffer Sink Implementation — Implement a data sink for Turbopuffer. 7. Distributed CSV/JSON Read/Write — Enable distributed read and write capabilities for CSV and JSON formats. 8. Dynamic Batch Size — Implement dynamic batch size adjustment. 9. Merge DuckDB 1.5.0 → 1.5.5 PRs — Cherry-pick and merge relevant PRs from DuckDB versions 1.5.0 through 1.5.5. 10. UDF Parameter Type Support — Add support for NumPy dict, cuDF, Pandas, and Tensor parameter types in UDFs. 📎 Learn more: Vane 0.1.0 Release · AstroVela/vane 🔗 Explore Vane 🌐 Website: ⭐ GitHub:

Meet Vane: a high-performance, multimodal-native engine for AI workloads. Build pipelines with Python or SQL, run AI operations natively, and scale from local execution to Ray clusters. Vane Data is open source and actively evolving.

Honored to become an Apache Paimon committer. Thanks to the Paimon PMC and community for the trust and support. I’ll keep contributing, reviewing, and learning with the community. #ApachePaimon# #ApacheSoftwareFoundation# #OpenSource#