RobustMQ

English

简体中文

English

简体中文

Appearance

RobustMQ: Redefining the Future of Cloud-Native Message Queues with Rust ​

RobustMQ Logo

In this data-driven era, message queues have become the "nervous system" of modern application architectures. From microservice communication to real-time data stream processing, from IoT devices to AI systems, message queues are everywhere. But as business complexity grows exponentially, traditional message queues are facing unprecedented challenges. It's time to rethink this field with a completely new perspective.

🔥 The "New Era Dilemma" of Message Queues ​

In daily architecture design and system operations, have you encountered these problems?

The Pain of Protocol Fragmentation ​

🤔 Scenario 1: IoT projects need MQTT
🤔 Scenario 2: Big data processing needs Kafka  
🤔 Scenario 3: Microservice communication needs RabbitMQ
🤔 Scenario 4: Financial trading needs RocketMQ

The result? A company might have to maintain 4 different messaging systems, each with its own:

  • Deployment methods
  • Monitoring systems
  • Operations procedures
  • Learning curves

This not only increases technical complexity but also becomes an "invisible killer" of team efficiency.

Limitations of Compute-Storage Integrated Architecture ​

Traditional MQs adopt compute-storage integrated architecture design, exposing serious adaptation problems in cloud-native environments:

  • Difficult elastic scaling: Taking Kafka as an example, scaling requires Partition Rebalance, a process that can last for hours, affecting business performance and even causing message backlogs
  • Unable to support Serverless: Storage and compute are tightly coupled, each Broker node must maintain local storage, unable to achieve true on-demand computing and second-level cold starts
  • Low resource utilization: Compute-intensive and storage-intensive workloads cannot be scheduled independently, often resulting in idle CPU with full disks, or idle disks with insufficient CPU
  • High operational complexity: Node failures require simultaneous handling of compute and storage recovery, with data migration and load balancing affecting each other, leading to long recovery times

The Dilemma of Single Storage Engine ​

Traditional message queues typically support only one storage engine, unable to flexibly adapt to the differentiated needs of various business scenarios:

  • High-performance scenarios: Real-time trading, IoT data collection require extremely low-latency read/write, but with relatively small data volumes, suitable for memory or SSD storage
  • Big data scenarios: Log collection, data analysis have lower performance requirements but massive data volumes, requiring low-cost object storage
  • Hybrid scenarios: The same business system has both hot data requiring high-performance access and cold data requiring long-term low-cost storage
  • Cost optimization needs: Unable to automatically select the most suitable storage medium based on data access patterns, leading to persistently high overall costs

Latency Performance Bottlenecks ​

Traditional message queues commonly suffer from performance instability issues in high-concurrency scenarios:

  • Severe latency jitter: Under high load, message processing latency is extremely unstable, suddenly jumping from milliseconds to seconds
  • Unpredictable processing time: The same message takes vastly different processing times at different points, failing to meet real-time requirements
  • Obvious performance degradation: As connections and message volume grow, system performance shows cliff-like decline
  • GC pause impact: Java systems' garbage collection causes periodic service interruptions, affecting user experience

New Challenges in the AI Era ​

With the explosive development of AI technology, message queues face unprecedented new challenges:

  • Exponential data growth: AI training data and multimodal data (text/image/video/audio) scale from TB to PB levels, making traditional MQ storage architectures inadequate

  • Complex and diverse AI workflows: From data collection to model training to inference services, each stage has completely different requirements for message queue latency, throughput, and persistence

  • AI infrastructure cost optimization demands: Expensive GPU computing power and high training data storage costs require elastic scheduling and intelligent tiered storage for cost reduction

  • Multi-tenant AI platform needs: Different AI teams require resource isolation, fine-grained permission control, and cost accounting management

💡 RobustMQ: A Solution Born for the Future ​

Against this background, RobustMQ was born.

It's not simply "yet another message queue," but a comprehensive rethinking and redesign for the AI era and cloud-native needs.

RobustMQ's core design goals: AI-Ready, Cloud-Native, Protocol-Unified, Storage-Flexible.

🦀 Rust: The Perfect Combination of Performance and Safety ​

Choosing Rust as the development language is not about chasing technological trends, but a carefully considered technical choice.

Why is Rust the ideal choice for message queues?

  • Memory safety: Eliminates security vulnerabilities like dangling pointers and buffer overflows
  • Zero-cost abstractions: High-level language features without runtime performance loss
  • No GC pauses: Friendly to latency-sensitive scenarios
  • Concurrency primitives: Native async/await support for massive concurrency
  • Mature ecosystem: High-quality libraries like Tokio, Serde, RocksDB

🌐 Multi-Protocol Unification: One Cluster, Supporting All Scenarios ​

One of RobustMQ's core innovations is multi-protocol unified architecture:

┌─────────────────────────────────────────────┐
│                RobustMQ Cluster             │
├─────────────┬─────────────┬─────────────────┤
│ MQTT        │ Kafka       │ AMQP            │
│ Port: 1883  │ Port: 9092  │ Port: 5672      │
│ ├─ IoT      │ ├─ Big Data │ ├─ Enterprise   │
│ ├─ Mobile   │ ├─ Stream   │ ├─ Microservice │
│ └─ Real-time│ └─ Logging  │ └─ Transactions │
└─────────────┴─────────────┴─────────────────┘

What does this mean?

  • 80% reduction in operational costs: From 4 systems to 1 system
  • Dramatic reduction in learning costs: One API, one monitoring system, one deployment process
  • Improved resource utilization: Unified resource pool, avoiding resource silos

☁️ Designed for Serverless and Cost Optimization ​

RobustMQ targets Serverless and cost optimization as two core goals from the initial architectural design.

Core advantages:

  • Serverless-ready: Stateless compute layer, supporting second-level cold starts and on-demand scaling
  • Pluggable storage: Supports multiple storage engines, from memory to object storage, flexibly adapting to different scenario requirements
  • Elastic scaling: No data migration required, scaling reduced from hours to seconds

🎯 Technical Architecture: Modern Compute-Storage Separation Design ​

RobustMQ adopts compute-storage separation architecture, consisting of three core components:

  • Broker Server: Stateless protocol processing layer, supporting multi-protocol and million-level connections
  • Meta Service: Raft-based scheduling layer, responsible for cluster management and service discovery
  • Journal Server: Pluggable storage layer, supporting multiple storage engines and intelligent tiering

🌟 Core Features: Redefining Message Queues ​

Core Feature Highlights ​

  1. 🔌 Multi-Protocol Unification: One cluster simultaneously supports MQTT, Kafka, AMQP and other protocols, eliminating technology stack fragmentation
  2. 🚀 Ultimate Performance: Rust-based zero-cost abstractions, single-machine million connections, microsecond-level latency
  3. 💾 Pluggable Storage: Intelligent tiered storage, hot data memory access, cold data object storage, 90% cost reduction
  4. ☁️ Serverless: Ultra-fast stateless elastic scaling, supporting second-level cold starts and on-demand computing
  5. 🔐 Enterprise-Grade Security: Multiple authentication methods, fine-grained permission control, end-to-end encryption
  6. 📊 Full-Chain Observability: Built-in monitoring and alerting, performance analysis, distributed tracing

🛠️ Development Experience: Making Complexity Simple ​

Simple and Easy Development Experience ​

Multiple deployment methods: Source compilation, pre-compiled binaries, Docker, Kubernetes, meeting different environment needs

Visual management interface: Web console provides complete functionality including cluster monitoring, user management, configuration management

Powerful CLI tools: Supports comprehensive operations including user management, permission control, real-time monitoring, message testing

📈 Development Status: Community Power in Action ​

Community Development Status ​

GitHub metrics: 1000+ Stars, 100+ Forks, 50+ Contributors, 2100+ Commits, active global developer community

Technical maturity: MQTT protocol fully supports production use, Kafka protocol under development, AMQP and RocketMQ protocols in planning

Deployment support: Supports standalone, cluster, Docker, Kubernetes and other deployment modes

🗺️ 2025 Roadmap: Towards Production Grade ​

2025 Development Roadmap:

  • Q4 Goal: Achieve MQTT production readiness, release version 0.2.0 as the first release

2026 Planning:

  • Core Task: Enhance Kafka capabilities, improve protocol compatibility and performance

Long-term Goals:

  • Become an Apache top-level project, alongside Kafka and Pulsar

🎯 Our Vision: Open Source First ​

Open source is not just our development model, but our core values.

🌍 Open Source Drives Innovation ​

  • Fully open source code: All core code is completely open source under Apache 2.0 license, with no commercial restrictions
  • Community collaboration: Building together with global developers, making technological innovation benefit everyone
  • Transparent development: Project decisions, technical roadmaps, and code reviews are fully open and transparent

🚀 Technical Excellence ​

  • Ultimate performance: High-performance message queue built with Rust, pursuing optimization of every microsecond
  • Innovative architecture: Compute-storage separation, multi-protocol unification, solving traditional MQ pain points through innovation
  • Engineering aesthetics: Elegant code design, making technology an art form

🏆 The Apache Path ​

Our goal is to become an Apache top-level project, which represents:

  • Technical benchmarks alongside projects like Kafka and Pulsar
  • Recognition and trust from the global open source community
  • International influence of Chinese open source projects

Open source first, technology supreme - this is RobustMQ's original intention and our commitment to the technical community.

🌟 Why Choose RobustMQ? ​

Why choose RobustMQ?

RobustMQ possesses advantages that traditional MQs cannot match: Rust-driven ultimate performance, multi-protocol unified platform, compute-storage separated Serverless architecture, pluggable intelligent storage, enterprise-grade security governance, and full-chain observability.

🚀 Get Started Now: 5-Minute RobustMQ Setup ​

Quick start steps:

  1. Install: One-click installation script or download pre-compiled packages
  2. Start: robust-server start to launch the cluster
  3. Test: Create users, publish and subscribe messages
  4. Manage: Access the web console to experience visual management

Detailed tutorials: Visit robustmq.com for complete guides.

🤝 Join the RobustMQ Community ​

RobustMQ's success depends on the power of the community. We sincerely invite you to become part of this exciting project!

🎯 How to Participate? ​

We welcome all forms of participation: code contributions, documentation improvements, testing feedback, community promotion. Whether you're a developer, architect, or technology enthusiast, you can find ways to participate in RobustMQ.

📞 Contact Us ​

WeChat Group

🌈 Conclusion: Creating the Future of Message Queues Together ​

Message queues are the "circulatory system" of modern application architectures, and in the AI era, they are the "neural networks" connecting data, algorithms, and applications.

Through innovations like Rust rewriting, compute-storage separation, multi-protocol unification, and intelligent storage, RobustMQ is committed to becoming the new benchmark for message queues in the AI era.

🚀 RobustMQ —— Next-generation cloud-native message queue, the future is here!

If you're interested in RobustMQ, welcome to follow our official account for the latest updates, or directly participate in project development. Let's change the world with code together!

Built with ❤️ by the RobustMQ team and contributors worldwide.