In Message Queues, Revolutionary Innovation Is Hard to Come By

While developing RobustMQ, I often think: where is there room for innovation in message queues? After looking at many new products and directions in the industry, I’ve reached a conclusion: the message queue field is unlikely to see truly revolutionary innovation. Not just message queues—the same applies across foundational software.

This isn’t pessimism; it’s reality. Foundational software has entered maturity. Innovation shows up more in engineering combination and implementation optimization than in theoretical breakthroughs.

The Message Queue Landscape Is Already Complete

If you look at the message queue space, every direction has already been explored.

In-memory systems: NATS, targeting extreme performance—a single node handles tens of millions of messages per second. High-performance persistence: Redpanda, Kafka rewritten in C++ with claimed multi-fold performance gains. Storage-compute separation: Pulsar, compute and storage scale independently. Object storage for cost: AutoMQ and Ursa, cutting cost by 90% using cloud storage. MQTT and IoT: EMQX. Traditional brokers: RabbitMQ and RocketMQ. These cover the main dimensions: performance, cost, use case, architecture. It’s hard to find a new direction nobody has tried.

Even StreamNative Ursa, which won the VLDB 2025 Best Industrial Paper Award, is essentially a combination of existing ideas. Leaderless design comes from Dynamo and Cassandra; object storage as persistence has been done by WarpStream and AutoMQ; writing stream data as Parquet is standard in data lakes; stream-table duality has been discussed by Kafka Streams and Flink for years. Ursa’s value isn’t new theory—it’s combining these in a message queue context to solve real problems.

Theoretical Innovation Is Largely Done

This phenomenon isn’t limited to message queues; it applies across foundational software.

In databases, the relational model, ACID transactions, and SQL have been around for 40+ years. Recent "innovation"? Distributed databases, cloud-native, HTAP, vector databases. But it’s still optimization and combination on the traditional model. Truly disruptive ideas like NoSQL were over a decade ago, and many found SQL still works better.

In OLAP, columnar storage, MPP, and vectorized execution are not new. ClickHouse, Snowflake, Databricks mainly combine and implement these more elegantly and more cloud-natively. There’s no fundamentally new query or storage model.

Same with API gateways. Reverse proxy, routing, rate limiting, auth—these functions haven’t changed for decades. Kong, Traefik, Envoy offer better implementations, more flexible config, more cloud-native deployments, but nobody has invented a new gateway paradigm.

Why? Because the main directions have been explored. Academia built solid foundations over decades: Paxos, Raft, two-phase commit for distributed systems; LSM-Tree, B+ Tree for storage; CAP, PACELC for consistency. These theories are largely complete; breakthroughs are unlikely.

And the boundaries of mature fields are clear. What tech for what scenario is well understood. Message queues handle data flow; databases handle storage and query; the boundaries are clear. Cross-boundary experiments mostly fail. User needs are stable; the core needs of infrastructure haven’t changed: reliable, fast, easy, cheap.

Even cloud-native and AI-native, which are trendy lately, are essentially adapting existing theory to new infrastructure—architectural redesign rather than theoretical change. Cloud-native emphasizes storage-compute separation, elastic scaling, object storage, but the core theory of message queues (log model, partitions, replication) hasn’t changed. AI-native emphasizes getting stream data into lakes quickly and real-time feature engineering, but the underpinnings are still the same distributed storage and stream processing ideas. What changes is architecture and combination, not the original theory.

Where Innovation Is Today

If theoretical innovation is scarce, where does innovation in foundational software live today? I see it as engineering innovation, not theoretical—mainly in three areas: combinatorial innovation, implementation innovation, and scenario innovation.

Combinatorial innovation is combining existing tech better. Ursa combines stream storage with the data lake; one dataset can be stream consumed and batch analyzed. Nothing new per se, but it solves data duplication and ETL complexity. The value is simpler user stacks and lower ops cost.

Implementation innovation is solving the same problems with better tech. Redpanda rewrote Kafka in C++, using modern C++ and closer-to-hardware optimization for higher performance. Iggy uses Rust + io_uring + thread-per-core for single-node extremes. The problem is the same as Kafka; the implementation is better. Language, runtime, I/O model choices are implementation-layer innovation.

Scenario innovation is optimizing existing tech for new scenarios. AI apps need stream data in the lake for training; the classic Kafka + Kafka Connect + Spark pipeline is heavy. Ursa writes Parquet directly and simplifies this. Edge computing needs lightweight messaging; NATS’s simple deployment fits. These make old tech serve new scenarios, not invent new tech.

Engineering Is the Real Competency

I believe foundational software is unlikely to have revolutionary theoretical innovation anymore. That’s not bad—it means the field is mature and the focus of innovation has shifted.

That’s also why foundational software isn’t as exciting as AI. AI is still in a theory-explosion phase—new models, new capabilities, new use cases every few months. But foundational software theory is mature; what remains is engineering: optimizing performance, handling edge cases, improving stability, improving docs, building ecosystems. This work matters but isn’t glamorous; it’s hard to make headlines.

Academic theory is solid—distributed consensus, storage structures, consistency models. Breakthroughs there are unlikely. But turning theory into reliable production systems, handling edge cases, staying stable under load—that still takes heavy engineering.

For example, the core model of message queues is append-only log. That model is good enough. Making it faster, more stable, cheaper—that’s an engineering problem and what users care about. Message queue use cases are largely known; brand-new requirements don’t appear often. But serving multiple scenarios in one system and lowering the cost of switching scenarios—that’s the value of scenario integration.

Thoughts on RobustMQ

When developing RobustMQ, I went through this mentally. At first I asked "can I do something innovative?" After studying different approaches, I saw that the main directions are already tried. That creates some anxiety: if there’s no innovation, where’s RobustMQ’s value?

I eventually realized RobustMQ’s value doesn’t have to be "new theory." It can be "implement message queues better, cover more scenarios, and make them nicer to use."

Specifically: combination and engineering. Combination means addressing scenario fragmentation. A company may need IoT device access (MQTT), backend data pipelines (Kafka), and object storage integration (for cost). Today that means multiple systems and complex ops. A single system that covers all of it is combinatorial value.

Engineering means doing it better with better tech. Rust for memory safety and performance; a well-designed storage layer for flexible scaling; clean, maintainable code. That’s engineering, not theory, but it determines system quality.

My two principles follow from this: the technical architecture ceiling must be high enough, and the market scenario ceiling must be high enough. Neither is raised by "theoretical innovation" but by sound technical choices and clear market positioning.

Summary

Revolutionary innovation is hard in message queues; that’s the reality. Databases, OLAP, gateways—similar situation. Theoretical innovation has largely been done; competition is now in implementation and scenario combination.

That doesn’t mean no opportunity.Quite the opposite—because theory is mature, there’s more room for engineering. Whoever implements existing tech better, combines it more sensibly, and adapts it more appropriately will be competitive.

For RobustMQ, the goal isn’t theoretical breakthroughs but engineering excellence. More efficient implementation in Rust, an architecture that supports long-term evolution, multi-protocol support to unify scenario fragmentation. That’s combinatorial and engineering innovation—maybe not flashy, but the most pragmatic path.

The value of foundational software is reliability and practicality, not concepts and breakthroughs. That’s the reality of mature fields and the direction long-term projects should stick to. Engineering done well is itself innovation.