RobustMQ Design Philosophy: Why We Choose Composition Over Innovation
The Question About Innovation
A friend recently asked: What's RobustMQ's breakthrough innovation?
That made me pause and think. To be honest, I don't know that RobustMQ has any breakthrough innovation. For the storage layer we use Raft, ISR, RocksDB, segmented storage, tiered migration—all established technologies. The architecture draws from Kafka's log model, Pulsar's compute-storage separation, KRaft's built-in metadata service. Nothing is RobustMQ's invention.
At first that bothered me. If an open-source project has no technical innovation, does it have no value? Should we invent something new, aim for an academic breakthrough? After thinking it through, my view shifted.
The Nature of Infrastructure
In infrastructure, especially for core components like message queues, users don't want "innovation"—they want stability, reliability, performance, and low cost.
I asked myself: As a user, would I choose a message queue because of "technical innovation"? No. I'd care: Is it stable? Is performance good enough? Can it lose data? Is it expensive? Is operations complex? If those are fine, I don't care whether the tech is new or old. If they're not, no amount of innovation would make me use it.
Look at successful projects. Kafka didn't invent new tech; it adapted the log model from database WAL. Docker didn't invent containers—Linux had cgroups and namespaces; Docker made them easy to use. Git didn't invent distributed version control; BitKeeper came first; Git was open and usable.
Their success came from solving real problems, lowering the bar, doing better than what came before—not from technical breakthroughs.
Composition Requires Deep Thought Too
Not chasing innovation doesn't mean cobbling existing tech together. How to combine and what to combine needs careful thought.
Kafka's ISR is mature; Pulsar's compute-storage separation is good. Can we combine them? Pulsar has separation but uses Broker + BookKeeper + ZooKeeper—complex operations. Can we get separation in a single process? Kafka KRaft removed ZooKeeper but stays compute-storage coupled. Can we remove external dependencies and still separate compute and storage?
These aren't "innovation" questions—they're "how do we better compose existing tech?" Our answer is single process, multiple roles: one Broker binary that, via role config, can be compute-focused, storage-focused, or both. Internal metadata is via Raft; no external coordination.
That design isn't new, but we spent a lot of time on it. Is it optimal? Unknown. Within what we can conceive, it's a reasonable choice. There may be better options we haven't thought of.
Solving Today's Visible Problems
In design we listed obvious problems with current message queues and looked for solutions.
Kafka scaling requires data migration—slow and impacts performance. We address it with segmented storage + compute-storage separation: no historical migration, new Segments flow to new nodes. We didn't invent this; RocketMQ uses segments, Pulsar uses separation; we combined them for this specific problem.
Pulsar's multi-component setup is operationally complex. We address it with single-process multi-role. That's not new tech either; many systems have role config; we apply it here. Storage is costly; we use tiered storage and move cold data to S3. Industry standard.
These are existing, visible problems with existing solutions. We're composing those solutions so one system addresses all of them.
The Limit of What I Can Conceive
Back to the original question: Does RobustMQ have breakthrough innovation? No. Does it need it?
My honest answer: we'd like it, but we haven't found a better angle. I'd love to have something original that breaks through in some direction. We've thought a lot and haven't found it. Message queues have evolved for years; the main approaches have been explored. Raft, Paxos, ISR, Quorum—consistency protocols exist. Local storage, object storage, tiered storage—all tried.
I think our current design is the limit of what we can conceive. Without taking technical risk or going radical, we're reasonably composing existing tech to solve known problems and fit more scenarios. There may be better ideas we haven't thought of; new possibilities may appear as tech evolves. For now, this is what we can do.
Importance of Engineering Implementation
Once we decided not to chase innovation, the question became: How do we actually implement these mature designs well?
Raft done well vs poorly—huge difference. Same with ISR: correctness under failures (network partition, node crash, disk failure) needs careful handling. Zero-copy—real zero-copy vs conceptual—completely different outcomes.
Redpanda rewrote Kafka in C++; architecturally little new, but strong implementation made it faster than Kafka. That made it clear: for message queues, implementation quality may matter more than architectural innovation.
RobustMQ chose Rust—no GC pauses, memory safety from the compiler. Those are language-level advantages. Turning them into system-level advantages depends on code quality. The devil of distributed systems is in the details: correctness of each module, thoroughness of each optimization, handling of edge cases. Those accumulate into final quality.
Getting the direction right is only the first step. Doing every detail well, optimizing performance thoroughly, making stability production-grade—that's the real challenge. I don't know if we'll get there, but it's worth aiming for.
Some Uncertainty
These are current thoughts; they may be immature or wrong.
Is not chasing innovation right? Will we lose competitiveness without it? Is composing existing tech the best strategy? Are we missing better possibilities? Will single-process multi-role really balance simplicity and flexibility, or will we hit its limits in implementation?
We don't have answers yet; we need to validate in practice. We'll start with MQTT and see if the design holds up, then adjust based on what we learn.
Summary
RobustMQ follows composition and optimization, not breakthrough and innovation. Not because we don't want innovation, but because we haven't found a better angle and believe our current composition is a reasonable choice within what we can conceive.
The next focus is turning design into reality and delivering through solid engineering. Will we succeed? We don't know. But at least we've thought through the direction.
About RobustMQ
RobustMQ is a high-performance message queue written in Rust that solves real problems through sensible composition of mature technologies. We're still exploring and validating; many ideas may change with practice. Follow us on GitHub and join the technical discussion.