AI Strategy of Message Queue Giants: What StreamNative, Confluent, and Redpanda Are Doing
While researching AI infrastructure recently, I noticed something interesting: almost all major message queue vendors are emphasizing their AI capabilities. StreamNative, Confluent, and Redpanda are all highlighting AI-related features on their landing pages and focusing on AI application cases at tech conferences.
As someone in the industry, I'm curious: how should message queues and AI be combined? What can they actually do? What have these vendors done in the AI direction? Is there real value or is it mostly marketing? This post reviews the AI strategies of these three companies to see what we can learn.
StreamNative: Betting on Agentic AI
StreamNative is the commercial company behind Apache Pulsar, founded by Pulsar's core developers. In the AI space, StreamNative has been the most aggressive, with a clear direction: betting on Agentic AI (AI Agents).
In September of this year, StreamNative released Orca Agent Engine, an event-driven runtime designed specifically for production AI Agents. Orca transforms AI Agents from stateless request-response functions into always-on, event-driven services with persistent streaming memory and built-in observability. It uses Pulsar or Kafka as a shared event bus, enables Agent collaboration and tool discovery via Model Context Protocol (MCP), and creates what they call an "Agent mesh."
Along with Orca, StreamNative also released MCP Server. This is a standardized interface that lets AI Agents (like Claude, ChatGPT) manage Kafka and Pulsar clusters in natural language. You can tell an Agent "increase the retention of topic 'user-signups' to 3 days," and the Agent will automatically execute the corresponding admin command. MCP is an open protocol from Anthropic, often called "USB-C for AI." StreamNative was among the first message queue vendors to support it.
Beyond the Agent direction, StreamNative has the Ursa engine. This is a Lakehouse-native streaming engine that writes stream data directly to open table formats like Apache Iceberg and Delta Lake, eliminating separate ETL pipelines. Stream data can be queried immediately via SQL and analytics tools without extra batch processing. StreamNative claims this architecture can cut costs by 90–95%.
StreamNative's strategy is clear: not just "supporting AI" but "redesigning for AI." Orca is a dedicated Agent runtime, MCP Server is a dedicated Agent interface, and Ursa is a dedicated real-time analytics engine. These are new products, not patches on existing ones.
Confluent: Emphasizing RAG and Real-Time Context
Confluent is the commercial company behind Kafka, founded by Kafka's creators. In the AI space, Confluent focuses on RAG (Retrieval-Augmented Generation) and real-time context.
Confluent's core product is Flink AI Model Inference, which is integrated into Apache Flink and lets you call LLM APIs directly in stream processing. For example, customer support queries flow through Flink, which associates user history and order data in real time, then calls the OpenAI API via Flink SQL UDF to generate responses. The whole flow happens in stream processing, without a separate system.
Confluent also emphasizes vector search integration. You can query external vector stores (MongoDB, Pinecone, Elasticsearch) directly from Flink for RAG scenarios. Stream data is used to generate vector embeddings in real time, store them in a vector database, and perform semantic search at query time.
Confluent also provides 120+ connectors to various AI services (OpenAI, AWS Bedrock, Google Vertex AI) and vector databases. Many case studies on their site describe how to build real-time RAG applications with Confluent, keep vector databases up to date, and provide LLMs with real-time context.
Confluent's approach is "make the message queue serve AI applications better." Instead of building new products, they position Kafka as the data backbone of AI infrastructure through integration (Flink + LLM + vector DB) and connectors.
Redpanda: Agentic Data Plane
Redpanda is a Kafka competitor written in C++ with better performance. In AI, Redpanda recently launched Agentic Data Plane.
Agentic Data Plane is positioned as the "data access layer for AI Agents." Its core capabilities include connectivity (300+ connectors to various data sources), governance (access control, audit, policy management), and real-time query (they recently acquired Oxla, a SQL engine company). Redpanda stresses that the main challenge for enterprise AI Agents is not the model, but how to safely access data spread across different systems.
Redpanda also supports real-time RAG, claiming quick integration with OpenAI, AWS Bedrock, Google Vertex AI, and continuous updates to vector databases. It also has Data Transforms, which uses WebAssembly for real-time data transformation and ML feature engineering.
Redpanda’s strategy sits between StreamNative and Confluent. It introduced new product (Agentic Data Plane) but still emphasizes integration and data governance. Agentic Data Plane is essentially connectivity + governance + query; it’s not tightly coupled to the message queue’s core function and feels more like adding data platform capabilities on top of a message queue.
Common Ground: Everyone Is Focusing on Integration
Despite different products, all three share something: the core is data integration and streaming.
StreamNative’s MCP Server integrates AI Agents with message queues; Orca uses the message queue as the event bus for Agents. Confluent’s Flink AI Model Inference integrates stream processing with LLM APIs; vector search integrates Flink with vector databases. Redpanda’s Agentic Data Plane integrates various data sources with AI Agents.
They’re not building core AI tech (model training, vector search, Agent inference) but are positioning message queues as the data infrastructure for AI applications. They help AI apps get real-time data, keep vector databases updated, collect user feedback, and coordinate services.
This matches an observation: the value of message queues in AI lies mainly in the application layer, not the training layer. They’re used for real-time data streaming, user interaction, and system coordination, not for model training itself. OpenAI’s 30+ Kafka clusters are used for the same purposes.
Observations and Reflections
A few takeaways from these AI strategies:
First, everyone is riding the AI wave, but there’s not that much to do. Message queues excel at data streaming; AI excels at models and algorithms. The overlap is mainly "real-time data." No matter the packaging, the substance is "providing real-time data streams for AI applications."
Second, integration and packaging outweigh innovation. Confluent’s vector search is Flink plus vector DB; Redpanda’s Agentic Data Plane is connectors plus governance tools—both combine existing tech. The only truly original work is StreamNative’s Orca, which is still in private preview and unproven.
Third, StreamNative is the most aggressive. They shipped a dedicated Agent runtime (Orca), MCP Server, a Lakehouse-native engine (Ursa), and even acquired a company (Oxla). Confluent and Redpanda are more conservative, focusing on integrations with existing products. Whether StreamNative is truly betting on AI or over-investing remains to be seen.
Fourth, market demand is unclear. Do AI Agents really need a dedicated "event-driven runtime," or is a general-purpose message queue enough? Will enterprises actually pay for "Agentic Data Plane"? These products look impressive, but real demand still needs validation.
Fifth, message queues remain infrastructure. Regardless of AI narratives, they still do data streaming, system integration, and event-driven behavior. Models, inference, and vector search live in other systems; message queues are supporting cast, not leads.
Summary
StreamNative, Confluent, and Redpanda are all embracing AI, but in essence they’re doing data integration and streaming. StreamNative is the most aggressive with dedicated Agent products. Confluent is the most conservative, mainly integrating existing tech. Redpanda is in the middle—new product, but less radical.
Objectively, message queues have limited room to maneuver in AI. What’s clear is: message queues matter more in the AI application layer than in the model training layer.
In the training layer, AI data patterns and message queue design don’t align. Training needs batch reads, repeated passes (multiple epochs), and random sampling, which fit object storage (S3, OSS) or distributed file systems (HDFS) better. The streaming, sequential, one-time consumption model of message queues offers little advantage there.
In the application layer, message queues add clear value. OpenAI runs 30+ Kafka clusters for real-time user request handling, feedback collection for data flywheels, cross-region data sync, and streaming pipelines (with Flink). All of that is application-layer work—user interaction, data flow, system coordination—not model training.
So when message queue vendors emphasize RAG, real-time context, and Agent collaboration, they’re really about providing data streaming infrastructure for the application layer: helping AI apps get real-time data, keep vector databases current, collect feedback, and coordinate services. These AI capabilities are more about market positioning and use-case framing than breakthrough technology. That’s not bad—AI applications do need reliable data streaming, which is where message queues shine.
About RobustMQ
RobustMQ is a message queue written in Rust, focused on core capabilities rather than chasing trends. We believe strong foundations matter more than flashy features. Welcome to check out our GitHub.