Project Origin

Write the first line of code in October 2023. There were two goals:

1. Learn Rust. Rust is a language with a high barrier to entry. It needs to be learned and mastered in the real world. I couldn't find a suitable project at that time, so I decided to write my own.
2. Explore Rust's use of message queues. Any open source component in the message queue space has issues. I want to start a new wheel and try to see if I can solve these problems without the burden of history.

About License

Apache License 2.0, our goal is to enter the Apache Foundation. Super cool.

About the Vision

Our vision is: To create the next-generation cloud-native integrated messaging queue, tailor a unified messaging infrastructure for AI systems, and provide a unified and efficient communication foundation for AI applications.

Is this vision too pretentious?

In my opinion, if you don't do it, how can you know if it can be achieved? As someone in the technical field, one should always have some ideals. Look up at the sky, walk down with your head down. Having walked on the technical path for so long, one should leave some traces. I hope to do something cool and interesting with you all.

Is this a commercial project?

No, this is a self-indulgent community project based on technology. The ultimate goal is to create an excellent message queue component, then enter the Apache Foundation. And then it'll be all about showing off. Ha ha ha ha.

Will we be competitive in the long run

There must be some good, absolutely confident.

Combined with the current development of Rust, cloud native, AI and other fields, continuous integration and learning, on the basis of no historical baggage. We are absolutely confident that we will be able to build a competitive message queue product for the AI age.

The key words are: multi-protocol, Serverless, plug-in storage. That is:

Based on Rust language, create a message queue product that supports multi-protocol, supports complete Serverless capabilities in the architecture, and has plug-in storage capabilities.

This definition will be refined over time, and the combination of MQ and AI will be explored next. And there are some explorations in the financial level Message Queue.

Now the message queue component is doing something

Some of the things that mainstream message queuing components are currently doing: -Separation of storage and computation: solving the problem of elastic expansion and shrinkage of the system.

• Multi-protocol (fusion of message and stream) : to solve the problem that multiple scenarios need to adapt to multiple protocols, and the cost of user switching protocol client transformation is high.
• Remove metadata dependency: Current mainstream message queue architectures rely on metadata components such as Zookeeper, but due to the complexity and stability of metadata components themselves, it also leads to stability and availability problems of message queues.
• Hierarchical storage: You want to reduce the cost of storage by moving some cold data to a lower-cost storage like S3, HDFS, etc.
• Adapt to different storage engines: Depending on the use case, you may need to adapt to different storage engines, such as S3, HDFS, in-house storage engines, etc.

In doing this, we found two types of problems:

• Native architecture defects lead to high complexity of the transformation, requiring a lot of human and material resources, and low revenue.
• It is not compatible with the community after the transformation, it cannot follow the evolution of the community, and it cannot be maintained in the long run.
• Each company will have its own version of the transformation, unable to form unity and synergy, repeated investment.

What do we wish we were

• is a pure message queuing component that doesn't bother to do anything around it until it's done with the core message queuing functionality.
• Is a platform-wide message queue that can be adapted to all popular message queuing protocols to support a variety of scenarios.
• Is a fully Serverless message queue, fully resilient, capable of unperceived horizontal scaling and horizontal scaling.
• It is a pluggable message queue that can support different storage requirements in different scenarios. It can be configured to switch between different storage engines, such as S3, HDFS, Bookeeper, etc.
• Is a high performance, stable, reliable message queue, able to support a large number of connections, QPS, throughput.

And existing message queues

Not yet dare to say the difference, salute the community. But as we continue to try, we will be able to make a message queue product that is better than the current open source components.

Expectation

I hope we stand on the shoulders of giants, look up at the sky, walk with heads down, work at ease, and build the best MQ Infra component.