RobustMQ Storage Layer Design

Continuing from the previous article, to satisfy various scenarios, in our vision RobustMQ's pluggable storage layer mainly consists of the following three types:

1. RocksDB and Memory in standalone mode: Mainly for running test cases in standalone mode and single-node deployment scenarios that require persistent storage.
2. Segment Engine: RobustMQ's built-in storage layer. It is RobustMQ's core storage layer and is built into RobustMQ. It is used to meet low-latency storage scenarios.
   1. Local storage: Local file-based persistent distributed storage
   2. External storage: Such as S3 (cloud object storage), HDFS, MinIO, or even MySQL, Redis, etc.

Storage Adapter

As mentioned in previous articles, RobustMQ's storage layer is uniformly distributed and stored through the Storage Adapter. See the article: Storage Layer Storage Adapter Architecture

Shard Abstraction

Standalone Mode

Because RocksDB's performance is sufficient, RocksDB storage engine is recommended for standalone persistent deployment scenarios. For local development and running test cases, Memory is recommended.

Segment Engine

Multiple Shard Types

1. Segmented / non-segmented
2. Memory storage / file storage

Data is stored in files in segments. Multiple replicas for consistency can be configured optionally.

Tiered Storage

Segmented data is stored to remote storage.

External Storage

Direct storage to external storage engines.