RobustMQ: Thoughts on Message Queue Storage Layer Design
If you follow the message queue space, you'll notice that many new open source projects have emerged in recent years. Looking closely at these projects reveals an interesting phenomenon: their core narrative is almost always "Kafka replacement."
Why is everyone focused on Kafka? Because Kafka has become sufficiently standardized as the de facto standard for stream processing. At the same time, Kafka's architecture has well-known issues: difficult elastic scaling, high storage costs, and significant operational complexity. From a business perspective, there is genuine demand to address these pain points.
But if you dig deeper into the technical differences between these projects, you'll find that the most fundamental distinction lies in the storage layer. The storage layer design determines performance in latency, reliability, throughput, cost, and more. One could say that storage layer design determines the success or failure of a message queue project.
RobustMQ is positioned as an All In One unified message platform. From the start, we hoped to match the use cases of all message middleware. This means our storage engine must be able to handle diverse scenario requirements. I've been thinking: how should the storage engine be designed?
Why Compute-Storage Separation
Almost every new-generation message queue talks about compute-storage separation. The core reasons are elasticity and cost.
Let's start with elasticity. As a storage component, message queues face an awkward problem: if compute and storage are coupled, ensuring message ordering often requires data migration when scaling out. Data migration is time-consuming, preventing rapid elastic scaling. In the cloud-native era, elasticity is essential.
Then there's cost. Cost splits into two directions:
First is resource mismatch. In operations, we often see two situations: either storage space is tight but CPU is idle, or CPU is insufficient but there's plenty of storage. The former wastes CPU when scaling for storage, the latter wastes storage when scaling for compute. Compute and storage resources must be independently scalable.
Second is cost differences across storage media. In the cloud-native era, local disks or cloud drives are expensive, while object storage is currently the cheapest option. Moving cold data to object storage can significantly reduce costs. Of course, this trades off some latency, but for many scenarios this trade-off is worthwhile.
Compute-storage separation isn't about architectural novelty—it's driven by real business needs.
Five Major Message Queue Scenarios
RobustMQ aims to support all major scenarios in the message queue space. Based on our research and understanding, these fall into five categories:
Scenario One: Low-latency, high-throughput. This is Kafka's domain—the core requirement for stream processing and big data. It demands data persistence, multi-replica guarantees, high throughput, and low latency. This is the industry standard and must be supported well.
Scenario Two: High-throughput, low-cost. When business scale reaches a certain level, storage cost becomes significant. Object storage is introduced, trading latency for cost. Suitable for large data volumes and cost-sensitive scenarios.
Scenario Three: Millions of Topics/Partitions. Some businesses need many partitions for data isolation and ordering. For example, multi-tenant scenarios with one topic per tenant; IoT scenarios with one partition per device. Storage must support a massive number of partitions without breaking down.
Scenario Four: Ultra-low latency, high QPS. Financial trading and similar scenarios require end-to-end latency under 1 millisecond with extremely high QPS. These scenarios often accept data loss in pursuit of extreme speed. Memory storage, zero-copy, avoiding disk I/O entirely.
Scenario Five: Edge message queue. Edge computing environments are resource-constrained, requiring no external dependencies, lightweight design, and high cohesion. Only memory or local files can be used; the storage model must be simple enough.
These five scenarios basically cover the major application directions of message queues. A single storage engine can hardly satisfy all scenario demands.
Trade-offs Between Two Storage Models
Message queue storage is always Append Only (write-only), to ensure data ordering. But in implementation, there are mainly two models.
Model One: One file per Partition. This is Kafka's approach. The advantages are obvious: sequential writes, sequential reads, extremely high performance. Combined with zero-copy, throughput can be maximized. The downside is equally obvious: too many small files hurt disk performance and put pressure on metadata. This model fits low-latency, high-throughput scenarios, but don't create too many partitions.
Model Two: Multiple Partitions share files. This is RocketMQ's choice. The advantage is support for millions of partitions with few small files and low metadata pressure. The cost is the inability to read sequentially, reducing read performance. This model fits scenarios requiring massive Topics/Partitions for data isolation.
The core trade-off between the two models is read/write performance versus partition count. Simpler functionality means higher performance; more complex functionality means lower performance. This is an engineering inevitability.
Plug-in Storage Design
Since a single storage model can't meet all needs, what's the solution?
RobustMQ's answer: build a pluggable storage solution from scratch, letting users choose flexibly by scenario. This includes three dimensions:
First, support both storage models. Both partition-independent-file and partition-shared-file models are supported; users choose based on scenario. Need extreme performance? Use the independent file model. Need massive partitions? Use the shared file model.
Second, support multiple storage engines. Memory, RocksDB (local KV), Journal Engine (log engine), MySQL (relational database), object storage (S3/OSS), etc. Each engine has different latency, cost, and reliability characteristics, covering different scenarios.
Third, flexible replica strategy configuration. Not all scenarios need multiple replicas. Users can choose single or multi-replica based on business characteristics, flexibly trading off performance and reliability.
With this architecture, RobustMQ can adapt to scenarios from edge to cloud, from low latency to low cost. When creating a Topic, users can specify which storage engine, which storage model, and how many replicas—the system adapts automatically.
Rethinking Replicas
A common question: with single-replica memory or file storage, if the node fails, doesn't data get lost? Is there any practical value?
Traditional wisdom says message queues must have replicas, otherwise data loss is unacceptable. But think carefully—this conclusion isn't absolute; it depends on business characteristics.
In many scenarios, message queues are mainly for real-time data distribution, and the business tolerates data loss. For example, the latest stock price—later data overwrites earlier data; losing historical prices doesn't affect the business. Latest sensor temperature readings—only the latest value matters; losing history is fine. Real-time monitoring data—built for ultra-low latency and high QPS, accepting some loss.
NATS's core differentiator is in-memory distribution, ultra-low latency, and extremely high QPS, without doing any replication. Apache IGGY focuses on ultra-low latency and high throughput with file storage plus zero-copy plus MMAP, and doesn't require replicas.
Another interesting scenario is MQTT. By default MQTT doesn't persist data; when a message arrives with no subscribers, it's discarded. MQTT imports data into downstream storage via Connectors, and Connectors themselves act as subscribers.
There's also the cloud-native scenario. Underlying cloud drives already provide multi-replica storage, ensuring data isn't lost. If the application layer uses single replica, the impact is only temporary unavailability of that node's partitions, but data remains on the cloud drive. For some businesses, this is fully acceptable.
So whether replicas are needed depends on the scenario. A single replica strategy can hardly satisfy all needs. RobustMQ's pluggable storage lets users choose based on business characteristics.
Language Performance Differences
We've run benchmark comparisons between Redpanda and Kafka. Under the same hardware, Redpanda's latency clearly outperforms Kafka. What does this mean? With architecture and optimization in place, programming language performance differences have a significant impact.
Rust and C++, as compiled languages, have inherent advantages over Java in performance-sensitive scenarios. No GC pauses, controllable memory layout, room for more aggressive optimization. This isn't language bias—it's determined by technical characteristics.
This is also one important reason RobustMQ chose Rust. We're building an ultra-low latency, high-performance message queue; language choice has been critical from the start.
Next Steps
This article shares our thoughts on the storage layer, but honestly, concrete implementation details aren't fully worked out yet. This is an ongoing process of exploration and optimization.
Our approach: first adapt storage for the MQTT scenario, validate the design in practice, then gradually improve the overall storage model. MQTT will be our first deep validation, Kafka the second—each scenario will bring new challenges and insights.
Long term, we have another idea: can we build on the message queue foundation to support simple retrieval and aggregation scenarios, providing lightweight log analysis? The industry has mature solutions like ELK, but could there be a lighter, simpler alternative? This is just a preliminary thought; how to do it needs deeper consideration.
Storage engine design is one of RobustMQ's core technical challenges. We'll keep exploring, keep optimizing, and validate our ideas in practice.
From the kernel up, one step at a time. That's our choice, and our commitment.
RobustMQ is building a pluggable storage engine to support multiple scenarios from edge to cloud. Follow our GitHub for technical discussion.