StreamNative Ursa: LakeHouse-Native Architecture Based on Object Storage

Ursa is a proprietary messaging engine from StreamNative, positioned as an alternative to Pulsar (my interpretation), with the core goal of Kafka protocol compatibility and optimization for cloud deployments. The target scenario is clear: log streaming where higher latency (200–500ms) is acceptable but cost is a concern, and where data must be ETLed to a data lake. By writing directly to LakeHouse formats and eliminating cross-zone replication, Ursa claims a 10x cost reduction (per their PR).

In September 2025, StreamNative's Ursa paper won the VLDB (Very Large Data Bases) Best Industrial Paper Award. Here I share my understanding of Ursa.

Ursa asks: Can infrastructure cost be reduced by 10x while maintaining full Kafka API compatibility? This is a topic the messaging community has been discussing. In optimizations for Apache Kafka by Redpanda, AutoMQ, and others, cost is also the main focus.

From a cost-optimization perspective, the directions are: use object storage instead of local storage, reduce cross-availability-zone traffic costs, and achieve elasticity through storage-compute separation. Below is a look at Ursa's design.

Core Problems to Solve

Ursa aims to solve two problems in cloud deployments: high cost and the split between streaming and analytics.

Cost comes from several sources. Cross-AZ data transfer is the largest. Kafka's Leader-Follower model requires replicating data from the Leader to Followers in other availability zones. For systems processing TB per day, this can cost tens of thousands of dollars per month. Disk-based replication is also expensive. With 3 replicas, each copy lives on SSD; SSD costs 5–10x more than object storage. Storage-compute coupling causes over-provisioning: to add storage, you must add brokers, even when CPU is idle. Moving data from Kafka to the data lake needs a separate Kafka Connect cluster; data is duplicated and governance gets complex.

The split between streaming and analytics is the other main issue. Real-time streams and analytics usually run on separate infrastructure. Data first lands in Kafka for real-time apps, then gets moved to the data lake for analytics. This causes latency (real-time data must wait for ETL), complexity (maintaining two systems), and consistency (two copies can diverge). Ideally, stream data would live directly in the LakeHouse without extra movement.

Core Features

To meet these goals, Ursa's approach can be summarized in three terms: Leaderless, LakeHouse-Native, Kafka-compatible.

Leaderless removes the concept of Partition Leader. Any Broker can handle requests for any Partition; a centralized metadata service (Oxia) ensures offset ordering. This eliminates cross-AZ replica replication and cuts network cost significantly.

LakeHouse-Native writes stream data directly as Parquet and registers it in Iceberg/Delta Lake tables. The same data can be consumed by Kafka consumers and queried by analytics engines—stream-table duality. This removes ETL and duplication.

Kafka compatible supports the full Kafka API. Users don't need to change application code; existing Kafka clients, tools, and monitoring systems work as-is, enabling seamless migration.

Technical Implementation

Architecture: Three-Tier Separation

Ursa uses a clear three-tier design. The metadata service uses Oxia (similar to Kraft's built-in metadata store and to RobustMQ's Meta Service) for offset allocation, partition metadata, and transaction state, handling millions of ops per second. The stream storage layer separates storage and compute: new data is first written to the WAL, then converted to Parquet by a Compaction Service. The WAL supports two modes: latency-optimized with BookKeeper (single-digit ms), and cost-optimized with object storage (hundreds of ms). Stateless Broker nodes handle client requests without maintaining partition state and can scale independently. Zone-aware routing ensures Producers and Consumers connect to local Brokers to reduce cross-zone traffic.

Ordering: Oxia's Centralized Offset Allocation

The main challenge of a Leaderless design is ordering. Ursa's solution is to move ordering from the data layer to the metadata layer. Multiple Brokers can accept writes concurrently, but offset allocation is serialized through Oxia transactions. After writing to the WAL, the Broker requests an offset from Oxia; Oxia reads the current max offset in a transaction, computes a new range, and atomically updates the index. Raft ensures global ordering. This design decentralizes data flow (any Broker can write) while centralizing metadata (Oxia enforces order), keeping Kafka's ordering semantics while gaining Leaderless flexibility.

Stream-Table Duality: One Dataset, Two Views

Stream-Table Duality is Ursa's key innovation. The same Parquet data serves both as a Kafka Topic and as an Iceberg/Delta Lake table. The mechanism is dual metadata: when writing, data is converted to Parquet and stored in object storage, and two sets of metadata are built. Kafka metadata records Topic, Partition, offset ranges, and file locations; Iceberg/Delta metadata records Table schema, Snapshots, Partitions, and file locations. Both point to the same Parquet files. On read, Kafka consumers use Kafka metadata for offset-ordered access; analytics engines use Iceberg/Delta metadata for columnar queries. One physical copy, two logical access patterns—no duplication and no ETL.

Compaction Service: Background Row-to-Column Conversion

The Compaction Service converts row-wise WAL objects into columnar Parquet files via a three-phase pipeline. Phase 1: Leader checks offset ranges, splits compaction tasks by throughput, and checkpoints to Oxia every 30 seconds. Phase 2: Workers read from the WAL, deserialize with the schema, and write Parquet. Phase 3: Leader collects metadata, batch-updates the offset index, and atomically commits to the Iceberg/Delta catalog. The whole process preserves exactly-once semantics; the offset index stays centralized so data is not duplicated, out of order, or lost.

Zone-Aware Routing: Lowering Cross-Zone Cost

Zone-aware routing is how Ursa reduces cross-AZ network cost. Each Broker reads its AZ from Kubernetes or EC2 metadata and registers with Oxia. When a client includes zone information in client.id, the Broker uses consistent hashing to compute the "responsible Broker" for that zone and returns it in the Metadata response. All subsequent requests go to that local Broker. Producers and Consumers only talk to Brokers in their own AZ, eliminating client-to-Broker cross-zone traffic. The same Partition in the same zone always routes to the same Broker, which can keep a local write buffer so the latest data is served from memory without hitting S3.

Design Tradeoffs

Latency vs. Cost

Ursa's main tradeoff is latency for cost. Traditional Kafka uses local SSD with single-digit ms write latency; Ursa uses object storage with tens to hundreds of ms. This increase is notable, but acceptable in many scenarios. Ursa introduces a "New CAP Theorem" notion: cost, availability, and performance tolerance cannot all be maximized. Sub-50ms latency plus multi-AZ HA needs expensive cross-zone replication; but if apps can tolerate 200–500ms, object storage offers large savings. Most data-intensive ingestion workloads (log collection, user behavior tracking, CDC, data pipelines) don't truly need single-digit ms; a few hundred ms has little impact on business value, while a 10x cost reduction is attractive.

StreamNative keeps a dual-engine strategy. Classic Pulsar Engine uses BookKeeper (sub-10ms). Ursa Engine uses object storage (hundreds of ms). Users pick by scenario: critical low-latency workloads use Classic Pulsar; cost-sensitive ingestion uses Ursa. Ursa Storage Extension lets you enable Ursa's LakeHouse storage on Classic Pulsar clusters: hot data stays in BookKeeper (low latency), cold data automatically moves to object storage (low cost). This gradual migration lowers adoption friction.

Centralized Metadata Choice

Ursa's Leaderless design is not fully decentralized. The Broker tier is leaderless; the metadata tier (Oxia) is centralized. Fully decentralized systems (e.g., Cassandra) remove all single points but cannot guarantee global ordering. Kafka's core semantics are strict ordering, which Ursa keeps. Ursa takes a pragmatic path: decentralized data, centralized metadata. Oxia uses Raft, supports sharding, and handles millions of ops per second; it rarely becomes the bottleneck.

Pragmatic Schema Evolution

Ursa adopts a streaming-friendly schema strategy. Traditional databases validate schema strictly at write time; incompatible data is rejected, which can cause data loss in streaming systems. Ursa skips schema checks at write; each message carries a schema-ID and is handled during Compaction. Incompatible data is routed to a dead-letter topic instead of dropped; offsets remain gap-free. On the LakeHouse side, all historical schema versions are union-merged; new fields are nullable for backward compatibility.

My Take

Note: Ursa is StreamNative's proprietary product, only available on StreamNative Cloud. It is not open source; users pay for access and cannot self-host or modify the code.

Technically, Ursa addresses cost and elasticity pain points for message streaming platforms in cloud deployments. The goals are cost, ETL, data lake integration, and elastic scaling. This is cloud-native architecture designed for cloud deployment, distinct from traditional on-prem designs.

LakeHouse-Native is the main innovation. Writing stream data directly as Parquet and registering it in Iceberg/Delta Lake isn't just format conversion—it redefines the relationship between streaming and analytics. In traditional setups, streams and analytics are separate, connected by ETL. Ursa puts stream data natively in the LakeHouse, removing that split. Data is queryable by SQL right after write, without waiting for scheduled ETL. Using open standards (Iceberg, Delta) avoids lock-in and allows any compatible engine to access the data.

The combination of Leaderless and Centralized Metadata is clever. It keeps Kafka's core semantics (order, exactly-once) and adds benefits (no Leader election, low network cost, fast recovery). Centralizing offset allocation through Oxia preserves ordering while decentralizing the Broker tier. This is particularly valuable in the cloud, where cross-AZ replication is a major cost driver.

The 10x cost reduction claim has backing: production on StreamNative Cloud (5 GB/s workload, 5% cost) supports it. But as a relatively young system, long-term stability and market success still need time to prove. Whether the innovation translates to business success depends on many factors.

The ecosystem strategy is worth noting. Full Kafka API compatibility enables migration without rewriting apps or tools, avoiding the ecosystem issues Pulsar once faced. Using open standards (Iceberg, Delta Lake, Parquet) reduces vendor lock-in; these choices show awareness of market realities.

Ursa's scope is clear: cost-sensitive, latency-tolerant, data-intensive workloads at hundreds of ms latency, not ultra-low-latency scenarios. This pragmatic boundary is sensible. StreamNative's dual-engine approach lets users choose by scenario: Classic Pulsar for low latency, Ursa for cost. That flexibility matches the diversity of real-world needs.

Ursa offers useful reference points for the messaging industry. Cost optimization is real market demand; as cloud usage grows, infrastructure cost becomes a major concern—optimizing cost can be more attractive than optimizing pure performance. Architectural innovation is harder to copy than engineering tricks; reimplementing in a new language or tuning I/O is easier to replicate than structural innovation. Ecosystem compatibility is crucial; building a new protocol’s ecosystem is very hard; reusing existing ecosystems is more realistic. Open standards matter; using open formats reduces lock-in and helps systems fit into broader data ecosystems.

Summary

Ursa is a message streaming platform with architectural innovation. Through Leaderless design (addresses cross-AZ cost), LakeHouse-Native storage (addresses local disk cost), and Stream-Table Duality (addresses ETL from streams to LakeHouse), it tackles the cost and streaming-analytics split of traditional message queues in cloud deployments. Significant savings come from eliminating cross-zone network, using object storage, and removing ETL. Kafka API compatibility and open standards help Ursa avoid the ecosystem struggles of new protocols. Clear tradeoffs and positioning let users make informed choices.

The VLDB 2025 Best Industrial Paper Award recognizes Ursa's technical merit. The system has been running on StreamNative Cloud for two years, but long-term technical and market performance will need ongoing observation. As cloud-native architecture for cloud deployment, Ursa represents an important direction: order-of-magnitude cost reduction through architectural innovation, while staying compatible with existing ecosystems. For anyone building infrastructure software, it offers plenty to think about.