System types · Built-in All levels

Apache Kafka

The built-in Apache Kafka system type — detection prefixes, the kafkametrics receiver setup, starter checks for consumer lag, empty groups, in-sync replicas and broker visibility, and tuning notes.

SLSluicio team 5 min read Updated Jul 2026

Apache Kafka is the distributed event-streaming platform behind most high-volume integration landscapes. Its health story has two layers: consumers (lag building up, groups losing all their members) and the cluster itself (partitions losing in-sync replicas, brokers dropping out). This system type has starter checks on both layers — including the one Kafka failure that deserves an immediate page: a partition with zero in-sync replicas.

Key: kafka · applied automatically to services emitting kafka.-prefixed metrics.

Getting the telemetry

The Collector’s kafkametrics receiver (contrib distribution) connects to the brokers and scrapes brokers, topics and consumer groups:

receivers:
  kafkametrics:
    brokers: ["kafka-1:9092", "kafka-2:9092"]
    protocol_version: "3.7.0"
    scrapers: [brokers, topics, consumers]
    collection_interval: 60s

The full pipeline — the resource processor that names the service, wiring into a metrics/kafka pipeline, verification — is worked through in Monitoring with the OpenTelemetry Collector, which uses Kafka as its running example.

What Sluicio detects

Metrics whose names start with kafka. — kafka.consumer_group.lag_sum, kafka.consumer_group.members, kafka.partition.replicas_in_sync, kafka.brokers, and the rest of the receiver’s export.

Starter checks

Check	Condition	Severity	Why it matters
Consumer lag	`kafka.consumer_group.lag_sum` > 1,000, split by consumer group	Warning	The group is falling behind its topics — the streaming equivalent of a queue backlog.
Consumer group empty	`kafka.consumer_group.members` < 1	Warning	Every consumer in the group is gone — nothing is processing, and lag is about to climb.
Partition not in sync	`kafka.partition.replicas_in_sync` = 0 for a partition	Critical	With zero in-sync replicas the partition is unavailable, and any unclean recovery risks data loss.
Brokers visible	`kafka.brokers` below the expected broker count	Warning	A broker the Collector can no longer see is down or partitioned — tune the threshold to your cluster size.

Tuning notes

Brokers visible needs your number first. The check can’t know your cluster size — set the threshold to it (alert when kafka.brokers < 3 on a 3-broker cluster) as the first thing you tune after applying the type.
Lag threshold is per-group. 1,000 offsets is minutes of backlog for a slow group and milliseconds for a fast one. Since the check splits by group, tune each group against its own throughput; a duration (“lag above X for 10 minutes”) separates rebalance blips from real drift.
Consumer group empty is normal for groups that only run on a schedule. Scope or disable the check for batch-style groups.
The receiver reports lag only for groups it can describe — if a group never shows up, check the Collector’s ACLs on DescribeGroups.