OpenTelemetry Collector

The OpenTelemetry Collector is the pipe everything else flows through — receiving OTLP from your apps, scraping your infrastructure, exporting it all to Sluicio. That makes it the one component whose failure hides every other failure: a struggling Collector doesn’t look like an outage, it looks like suspiciously good news. This is a service type (it describes a workload you run, not a broker to scrape), and its starter checks watch the failure chain of a telemetry pipeline: data refused on the way in, queued and failing on the way out, dropped in the middle, and the process itself running out of memory.

Key: otel-collector · applied automatically to services emitting otelcol-prefixed metrics.

Getting the telemetry

The Collector reports on itself: its internal telemetry appears as otelcol_* metrics on a local Prometheus endpoint, which you loop back through the Collector’s own pipeline:

otel-collector-config.yaml (self-monitoring, minimal)

receivers:
  prometheus/self:
    config:
      scrape_configs:
        - job_name: otel-collector
          scrape_interval: 60s
          static_configs:
            - targets: ["127.0.0.1:8888"]

service:
  telemetry:
    metrics:
      level: normal
      readers:
        - pull:
            exporter:
              prometheus:
                host: 127.0.0.1
                port: 8888

The complete version — the resource processor that names the service, the pipeline wiring, plus the rest of a production Collector setup — is in Monitoring with the OpenTelemetry Collector.

What Sluicio detects

Metrics whose names start with otelcol — otelcol_exporter_queue_size, otelcol_exporter_send_failed_spans, otelcol_processor_dropped_spans, otelcol_receiver_refused_spans, otelcol_process_memory_rss, and the rest of the internal telemetry.

Starter checks

Check	Condition	Severity	Why it matters
Exporter queue backlog	otelcol_exporter_queue_size > 5,000	Warning	The sending queue is filling — the backend is slow or unreachable; when the queue is full, new data is dropped.
High memory	Process memory > 1 GiB	Warning	A Collector near its memory limit starts refusing data (memory_limiter) or gets OOM-killed — either way, telemetry stops.
Send failures	otelcol_exporter_send_failed_spans increasing (delta)	Warning	Exports to the backend are failing — auth, network, or the backend rejecting data. Retries mask it until the queue fills.
Dropped spans	otelcol_processor_dropped_spans increasing (delta)	Warning	A processor is discarding data — typically the memory_limiter shedding load. Whatever it dropped never reaches Sluicio.
Enqueue failures	Enqueue-failed count increasing (delta)	Warning	Data arrived but couldn’t even enter the sending queue — the queue is already full. Loss is happening now.
Refused spans	otelcol_receiver_refused_spans increasing (delta)	Warning	The Collector is pushing back on your apps — back-pressure has reached the sources.
Error-level logs	Error-level log records from the Collector	Warning	The Collector narrates its own failures well — config errors, scrape failures, exporter errors all land here first.

Tuning notes

• Queue backlog should be read against your configured sending_queue.queue_size (default 1,000 per exporter — if you kept the default, the 5,000 threshold can never fire; either raise the queue for real burst tolerance or lower the check to ~80% of your actual queue size).
• Memory — match the threshold to the environment: comfortably below the container’s limit and above the memory_limiter’s limit_percentage trip point, so the warning lands before shedding starts.
• The delta checks are span-flavoured (_spans); the same failure counters exist for metrics and logs (otelcol_exporter_send_failed_metric_points, _log_records). If a Collector carries mostly metrics, clone the checks for those names.
• Error-level logs requires the Collector’s logs to reach Sluicio at all — on a container platform that usually means a filelog receiver or your log shipper forwarding the Collector’s own output.
• One instance of this type per Collector deployment: gateways, daemonset agents and per-team Collectors each deserve their own service (distinct service.name), or one noisy agent hides in the aggregate.