Skip to content

MQTT 专用指标 ​

RobustMQ 通过 Prometheus /metrics 端点暴露 MQTT 协议、连接、消息、延迟队列和 Connector 指标。下表使用 Prometheus 实际暴露的指标名。Rust 中注册 Counter 时不写 _total,prometheus-client 会在暴露时自动追加该后缀。

协议包指标 ​

接收数据包 ​

指标名称类型标签描述
mqtt_packets_received_totalCounternetworkMQTT 接收包总数
mqtt_packets_connect_received_totalCounternetworkCONNECT 包接收数
mqtt_packets_publish_received_totalCounternetworkPUBLISH 包接收数
mqtt_packets_connack_received_totalCounternetworkCONNACK 包接收数
mqtt_packets_puback_received_totalCounternetworkPUBACK 包接收数
mqtt_packets_pubrec_received_totalCounternetworkPUBREC 包接收数
mqtt_packets_pubrel_received_totalCounternetworkPUBREL 包接收数
mqtt_packets_pubcomp_received_totalCounternetworkPUBCOMP 包接收数
mqtt_packets_subscribe_received_totalCounternetworkSUBSCRIBE 包接收数
mqtt_packets_unsubscribe_received_totalCounternetworkUNSUBSCRIBE 包接收数
mqtt_packets_pingreq_received_totalCounternetworkPINGREQ 包接收数
mqtt_packets_disconnect_received_totalCounternetworkDISCONNECT 包接收数
mqtt_packets_auth_received_totalCounternetworkAUTH 包接收数
mqtt_packets_received_error_totalCounternetwork接收或解码时被拒绝的数据包数
mqtt_packets_connack_auth_error_totalCounternetwork授权失败的 CONNACK 包数
mqtt_packets_connack_error_totalCounternetwork返回错误结果的 CONNACK 包数
mqtt_bytes_received_totalCounternetworkMQTT 接收字节总数

发送数据包 ​

指标名称类型标签描述
mqtt_packets_sent_totalCounternetwork, qosMQTT 发送包总数
mqtt_packets_connack_sent_totalCounternetwork, qosCONNACK 包发送数
mqtt_packets_publish_sent_totalCounternetwork, qosPUBLISH 包发送数
mqtt_packets_puback_sent_totalCounternetwork, qosPUBACK 包发送数
mqtt_packets_pubrec_sent_totalCounternetwork, qosPUBREC 包发送数
mqtt_packets_pubrel_sent_totalCounternetwork, qosPUBREL 包发送数
mqtt_packets_pubcomp_sent_totalCounternetwork, qosPUBCOMP 包发送数
mqtt_packets_suback_sent_totalCounternetwork, qosSUBACK 包发送数
mqtt_packets_unsuback_sent_totalCounternetwork, qosUNSUBACK 包发送数
mqtt_packets_pingresp_sent_totalCounternetwork, qosPINGRESP 包发送数
mqtt_packets_disconnect_sent_totalCounternetwork, qosDISCONNECT 包发送数
mqtt_packets_auth_sent_totalCounternetwork, qosAUTH 包发送数;AUTH 不是 PUBLISH,因此 qos 为 -1
mqtt_bytes_sent_totalCounternetwork, qosMQTT 发送字节总数
mqtt_retain_packets_received_totalCounterqos保留消息接收数
mqtt_retain_packets_sent_totalCounterqos保留消息发送数

跨网络汇总 ​

以下无标签 Counter 汇总所有网络类型的协议包和字节数。

指标名称类型描述
mqtt_total_bytes_received_totalCounter所有网络类型接收的 MQTT 字节数
mqtt_total_bytes_sent_totalCounter所有网络类型发送的 MQTT 字节数
mqtt_total_packets_received_totalCounter所有网络类型接收的 MQTT 包数
mqtt_total_packets_sent_totalCounter所有网络类型发送的 MQTT 包数
mqtt_total_packets_connect_totalCounter接收的 CONNECT 包数
mqtt_total_packets_connack_totalCounter发送的 CONNACK 包数
mqtt_total_packets_publish_received_totalCounter接收的 PUBLISH 包数
mqtt_total_packets_publish_sent_totalCounter发送的 PUBLISH 包数
mqtt_total_packets_puback_received_totalCounter接收的 PUBACK 包数
mqtt_total_packets_puback_sent_totalCounter发送的 PUBACK 包数
mqtt_total_packets_pubrec_received_totalCounter接收的 PUBREC 包数
mqtt_total_packets_pubrec_sent_totalCounter发送的 PUBREC 包数
mqtt_total_packets_pubrel_received_totalCounter接收的 PUBREL 包数
mqtt_total_packets_pubrel_sent_totalCounter发送的 PUBREL 包数
mqtt_total_packets_pubcomp_received_totalCounter接收的 PUBCOMP 包数
mqtt_total_packets_pubcomp_sent_totalCounter发送的 PUBCOMP 包数
mqtt_total_packets_subscribe_totalCounter接收的 SUBSCRIBE 包数
mqtt_total_packets_suback_totalCounter发送的 SUBACK 包数
mqtt_total_packets_unsubscribe_totalCounter接收的 UNSUBSCRIBE 包数
mqtt_total_packets_unsuback_totalCounter发送的 UNSUBACK 包数
mqtt_total_packets_pingreq_totalCounter接收的 PINGREQ 包数
mqtt_total_packets_pingresp_totalCounter发送的 PINGRESP 包数
mqtt_total_packets_disconnect_received_totalCounter接收的 DISCONNECT 包数
mqtt_total_packets_disconnect_sent_totalCounter发送的 DISCONNECT 包数
mqtt_total_packets_auth_totalCounter接收的 AUTH 包数
mqtt_total_packets_auth_sent_totalCounter发送的 AUTH 包数

协议包耗时 ​

指标名称类型标签描述
mqtt_packet_process_duration_msHistogramnetwork, packetMQTT 协议包处理耗时(毫秒)
mqtt_packet_send_duration_msHistogramnetwork, packetMQTT 协议包发送耗时(毫秒)

network 的取值为 Tcp、Tls、Websocket、Websockets、Quic;无法获取连接类型时为 Unknown。PUBLISH 包的 qos 为 0、1 或 2,非 PUBLISH 包为 -1。packet 使用小写值,例如 connect、publish 和 subscribe。

连接与认证指标 ​

指标名称类型标签描述
client_connections_totalCountertenant, client_id客户端连接尝试次数,不论结果如何
mqtt_connection_success_totalCounter—MQTT 连接成功次数
mqtt_connection_failed_totalCounter—MQTT 连接失败次数
mqtt_disconnect_success_totalCounter—MQTT 断开成功次数
mqtt_connection_expired_totalCounter—MQTT 连接过期次数
mqtt_auth_success_totalCounter—MQTT 认证成功次数
mqtt_auth_failed_totalCounter—MQTT 认证失败次数
mqtt_acl_success_totalCounter—MQTT ACL 检查通过次数
mqtt_acl_failed_totalCounter—MQTT ACL 检查失败次数
mqtt_blacklist_blocked_totalCounter—MQTT 黑名单拦截次数
mqtt_subscribe_success_totalCounter—订阅成功次数
mqtt_subscribe_failed_totalCounter—订阅失败次数
mqtt_unsubscribe_success_totalCounter—取消订阅成功次数

会话与消息指标 ​

指标名称类型标签描述
mqtt_session_created_totalCounter—MQTT 会话创建数
mqtt_session_deleted_totalCounter—MQTT 会话删除数
session_messages_in_totalCountertenant, client_id按会话统计的接收消息数
session_messages_out_totalCountertenant, client_id按会话统计的发送消息数
connection_messages_in_totalCounterconnection_id按连接统计的接收消息数
connection_messages_out_totalCounterconnection_id按连接统计的发送消息数
mqtt_messages_received_totalCounter—从客户端接收的消息数
mqtt_messages_sent_totalCounter—向客户端发送的消息数
mqtt_message_bytes_received_totalCounter—从客户端接收的消息字节数
mqtt_message_bytes_sent_totalCounter—向客户端发送的消息字节数
mqtt_messages_delayed_totalCounter—延迟发布消息数
mqtt_messages_dropped_no_subscribers_totalCounter—因无订阅者丢弃的消息数

资源统计 ​

指标名称类型标签描述
mqtt_connections_countGauge—当前 MQTT 连接数
mqtt_sessions_countGauge—当前 MQTT 会话数
mqtt_topics_countGauge—当前 MQTT 主题数
mqtt_subscribers_countGauge—当前 MQTT 订阅者数
mqtt_subscriptions_exclusive_countGauge—当前独占订阅数
mqtt_subscriptions_shared_countGauge—当前共享订阅数
mqtt_subscriptions_shared_group_countGauge—当前共享订阅组数
mqtt_retained_countGauge—当前保留消息数
mqtt_delay_queue_total_capacityGaugeshard_no延迟队列总容量
mqtt_delay_queue_used_capacityGaugeshard_no延迟队列已用容量
mqtt_delay_queue_remaining_capacityGaugeshard_no延迟队列剩余容量

Topic 与订阅指标 ​

指标名称类型标签描述
topic_messages_written_totalCountertenant, topic写入 Topic 的消息数
topic_bytes_written_totalCountertenant, topic写入 Topic 的字节数
topic_messages_sent_totalCountertenant, topic从 Topic 发送的消息数
topic_bytes_sent_totalCountertenant, topic从 Topic 发送的字节数
subscribe_messages_sent_totalCountertenant, client_id, path, status向订阅路径发送的消息数
subscribe_topic_messages_sent_totalCountertenant, client_id, path, topic_name, status向订阅路径和主题发送的消息数
subscribe_bytes_sent_totalCountertenant, client_id, path, status向订阅路径发送的字节数
subscribe_topic_bytes_sent_totalCountertenant, client_id, path, topic_name, status向订阅路径和主题发送的字节数

订阅 status 的取值为 success 和 failed。

延迟消息与 Connector 指标 ​

指标名称类型标签描述
delay_msg_enqueue_totalCounter—延迟消息入队数
delay_msg_deliver_totalCounter—延迟消息投递数
delay_msg_deliver_fail_totalCounter—延迟消息投递失败数
delay_msg_recover_totalCounter—启动时恢复的延迟消息数
delay_msg_recover_expired_totalCounter—恢复时发现的过期消息数
delay_message_retry_totalCounterstatus延迟消息重试次数,status 为 success 或 failed
delay_message_retry_count_totalCounterattempt按数字重试次数统计
delay_message_reenqueued_totalCounter—投递失败后重新入队的消息数
delay_message_dead_letter_totalCounter—发送到死信队列的消息数
delay_msg_enqueue_duration_msHistogram—延迟消息入队耗时
delay_msg_deliver_duration_msHistogram—延迟消息投递耗时
delay_task_created_totalCounter—创建的延迟任务数
delay_task_executed_totalCountertask_type成功执行的延迟任务数
delay_task_execute_failed_totalCountertask_type延迟任务执行失败数
delay_task_schedule_latency_secondsHistogramtask_type实际执行时间与目标时间的差值
mqtt_connector_messages_sent_success_totalCountertenant, connector_type, connector_nameConnector 成功发送的消息数
mqtt_connector_messages_sent_failure_totalCountertenant, connector_type, connector_nameConnector 发送失败的消息数
mqtt_connector_send_duration_msHistogramtenant, connector_type, connector_nameConnector 发送耗时
mqtt_connector_retry_totalCountertenant, connector_type, connector_name, strategyConnector 重试次数
mqtt_connector_messages_discarded_totalCountertenant, connector_type, connector_name, strategy按终态策略丢弃的消息数
mqtt_connector_dlq_messages_totalCountertenant, connector_type, connector_name, result按结果统计的死信写入尝试数
mqtt_connector_offset_commit_failure_totalCountertenant, connector_type, connector_nameConnector offset 提交失败数
mqtt_connector_source_read_failure_totalCountertenant, connector_type, connector_nameConnector 源读取失败数
mqtt_connector_upGaugetenant, connector_type, connector_nameConnector 状态:1 运行,0 停止
mqtt_connector_messages_sent_success_agg_totalCounter—所有 Connector 成功发送总数
mqtt_connector_messages_sent_failure_agg_totalCounter—所有 Connector 发送失败总数
mqtt_connector_send_duration_ms_aggHistogram—所有 Connector 的聚合发送耗时

使用示例 ​

rust
use common_metrics::mqtt::{auth, event, time};
use metadata_struct::connection::NetworkConnectionType;

event::record_mqtt_connection_success();
auth::record_mqtt_auth_failed();
time::record_packet_process_duration(
    &NetworkConnectionType::Tcp,
    "publish",
    15.5,
);

PromQL 示例:

promql
rate(mqtt_packets_publish_received_total[5m])
rate(mqtt_packets_publish_sent_total[5m])
rate(mqtt_auth_failed_total[5m]) /
  (rate(mqtt_auth_success_total[5m]) + rate(mqtt_auth_failed_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(mqtt_packet_process_duration_ms_bucket[5m])))
mqtt_delay_queue_used_capacity / mqtt_delay_queue_total_capacity

Grafana 仪表板配置 ​

仓库提供了可直接导入的仪表板文件 grafana/robustmq-broker.json。配置和导入步骤请参阅 Grafana 配置指南。下面的示例是面板查询片段,不是完整的仪表板 JSON 文件。

连接概览面板 ​

json
{
  "title": "MQTT 连接概览",
  "targets": [
    {
      "expr": "mqtt_connections_count",
      "legendFormat": "当前连接数"
    },
    {
      "expr": "rate(mqtt_connection_success_total[5m])",
      "legendFormat": "连接成功率/秒"
    },
    {
      "expr": "rate(mqtt_connection_failed_total[5m])",
      "legendFormat": "连接失败率/秒"
    }
  ]
}

消息吞吐量面板 ​

协议包和字节 Counter 都带有 network 标签。sum by (network) 会合并发送方向的 QoS 序列,同时在图例中保留网络类型。

json
{
  "title": "MQTT 消息吞吐量",
  "targets": [
    {
      "expr": "sum by (network) (rate(mqtt_packets_publish_received_total[5m]))",
      "legendFormat": "接收 PUBLISH/秒 ({{network}})"
    },
    {
      "expr": "sum by (network) (rate(mqtt_packets_publish_sent_total[5m]))",
      "legendFormat": "发送 PUBLISH/秒 ({{network}})"
    },
    {
      "expr": "sum by (network) (rate(mqtt_bytes_received_total[5m]))",
      "legendFormat": "接收字节/秒 ({{network}})"
    },
    {
      "expr": "sum by (network) (rate(mqtt_bytes_sent_total[5m]))",
      "legendFormat": "发送字节/秒 ({{network}})"
    }
  ]
}

性能监控面板 ​

计算分位数前必须按 le 聚合 Histogram bucket。

json
{
  "title": "MQTT 性能监控",
  "targets": [
    {
      "expr": "histogram_quantile(0.95, sum by (le) (rate(mqtt_packet_process_duration_ms_bucket[5m])))",
      "legendFormat": "P95 处理耗时(毫秒)"
    },
    {
      "expr": "histogram_quantile(0.99, sum by (le) (rate(mqtt_packet_process_duration_ms_bucket[5m])))",
      "legendFormat": "P99 处理耗时(毫秒)"
    }
  ]
}

MQTT 告警规则 ​

以下 Prometheus 规则示例使用当前实现实际暴露的指标名。仓库还提供了更完整的告警规则文件 grafana/robustmq-alerts.yml;配置方法请参阅 Grafana 配置指南。这些片段用于说明 MQTT 专用告警条件,并不替代完整规则文件。

连接告警 ​

yaml
- alert: RobustMQHighConnectionFailureRate
  expr: (sum(rate(mqtt_connection_failed_total[5m])) / clamp_min(sum(rate(mqtt_connection_success_total[5m])) + sum(rate(mqtt_connection_failed_total[5m])), 1e-9)) > 0.1
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "MQTT 连接失败率过高"
    description: "MQTT 连接失败占比为 {{ $value | humanizePercentage }}。"

- alert: RobustMQHighConnectionCount
  expr: mqtt_connections_count > 5000
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "MQTT 连接数过高"
    description: "当前 MQTT 连接数:{{ $value }}"

性能与错误告警 ​

Histogram bucket 按 le 聚合,并保留可用的 network 和 packet 标签,以便告警指出受影响的传输类型和数据包类型。

yaml
- alert: RobustMQHighMqttPacketProcessingLatency
  expr: histogram_quantile(0.95, sum by (le, network, packet) (rate(mqtt_packet_process_duration_ms_bucket[5m]))) > 100
  for: 3m
  labels:
    severity: warning
  annotations:
    summary: "MQTT 数据包处理延迟过高"
    description: "{{ $labels.network }} 上 {{ $labels.packet }} 的 P95 延迟为 {{ $value }}ms。"

- alert: RobustMQPacketErrors
  expr: sum by (network) (rate(mqtt_packets_received_error_total[5m])) > 1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "MQTT 数据包错误率过高"
    description: "{{ $labels.network }} 每秒发生 {{ $value }} 次数据包错误。"

认证与容量告警 ​

yaml
- alert: RobustMQHighMqttAuthenticationFailureRatio
  expr: (sum(rate(mqtt_auth_failed_total[5m])) / clamp_min(sum(rate(mqtt_auth_success_total[5m])) + sum(rate(mqtt_auth_failed_total[5m])), 1e-9)) > 0.2
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "MQTT 认证失败占比过高"
    description: "认证失败占比为 {{ $value | humanizePercentage }}。"

- alert: RobustMQDelayQueueNearFull
  expr: (mqtt_delay_queue_total_capacity > 0) and ((mqtt_delay_queue_used_capacity / mqtt_delay_queue_total_capacity) > 0.9)
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "MQTT 延迟队列接近满载"
    description: "分片 {{ $labels.shard_no }} 的容量使用率为 {{ $value | humanizePercentage }}。"

- alert: RobustMQTooManyRetainedMessages
  expr: mqtt_retained_count > 100000
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "MQTT 保留消息过多"
    description: "当前保留消息数:{{ $value }}"
🎉 既然都登录了 GitHub,不如顺手给我们点个 Star 吧!⭐ 你的支持是我们最大的动力 🚀