Kubernetes

Kubernetes is where a growing share of integration workloads actually run, and its failures are rarely subtle once you can see them: a node goes NotReady and takes its pods with it, a deployment drops to zero available replicas, a container settles into a restart loop, a nightly Job fails and leaves nothing behind to say so. This system type has starter checks on each of those, and deliberately none on the control plane.

Key: kubernetes · applied automatically to services emitting k8s.-prefixed metrics.

One type for every distribution

There is no separate type for k0s, k3s, RKE2, MicroK8s, kubeadm, EKS, AKS, GKE or OpenShift. Every check below reads the Kubernetes API or the kubelet, and conformance guarantees those interfaces on all of them, so one type covers the lot and behaves the same on each.

What genuinely differs between distributions is the control plane, which is exactly why nothing here watches it:

• On a managed cluster (EKS, AKS, GKE) the provider runs the control plane and does not expose its metrics to you at all.
• On k3s backed by kine, etcd is replaced by SQLite or a SQL database, so etcd metrics do not exist.
• On k0s and kubeadm they usually do exist, but a check that only works on some clusters is worse than no check: it sits in the health list looking armed and stays silent for ever.

If you run a distribution that does expose its control plane and you want to watch it, that is a good reason to create your own type - see overriding a built-in.

Getting the telemetry

Two Collector receivers, both in the contrib distribution. The k8s_cluster receiver watches cluster objects through the Kubernetes API, and the kubelet_stats receiver reads node and volume resource usage from the kubelet:

otel-collector-config.yaml (receivers)

receivers:
  k8s_cluster:
    auth_type: serviceAccount
    collection_interval: 30s
  kubelet_stats:
    auth_type: serviceAccount
    endpoint: "https://${env:K8S_NODE_NAME}:10250"
    insecure_skip_verify: true
    collection_interval: 30s

k8s_cluster runs once per cluster (a Deployment with a ServiceAccount that can list and watch cluster objects). kubelet_stats runs on every node (a DaemonSet, with K8S_NODE_NAME from the downward API), because each kubelet only knows about its own node.

kubelet_stats needs Collector v0.152.0 or later; before it the receiver is called kubeletstats. k8s_cluster has that name on v0.141.0 and later; on a Collector that does not accept it, use k8scluster.

The full pipeline - the resource processor that names the service, wiring into a metrics pipeline, verification - is worked through in Monitoring with the OpenTelemetry Collector.

What Sluicio detects

Metrics whose names start with k8s. - k8s.node.condition_ready, k8s.pod.phase, k8s.container.restarts, k8s.deployment.available, k8s.job.failed_pods and the rest of the two receivers’ export.

kube-state-metrics is a different path. Scraped through the prometheus receiver it arrives as kube_*, with entirely different metric names, so it is not detected as this type. Matching it here would give the service the right label and then apply a set of checks that could never fire. If that is your setup, create your own type carrying your own metric names.

Starter checks

Check	Condition	Severity	Why it matters
Node not ready	k8s.node.condition_ready < 1, split by node	Critical	The node is NotReady or unreachable — its pods are being evicted, or are already gone.
Deployment has no available replicas	k8s.deployment.available < 1, split by deployment	Critical	Every replica is unavailable: that workload is down, not degraded.
Container restart loop	k8s.container.restarts rising by more than 3 in the window, split by container	Warning	CrashLoopBackOff, an OOM kill, or a failing probe.
Pod failed	k8s.pod.phase ≥ 4, split by pod	Warning	The pod is in Failed or Unknown phase.
Job failing	k8s.job.failed_pods > 0, split by job	Warning	A Job has failed pods — the one that matters most on a cluster running scheduled integration work.
DaemonSet not running anywhere	k8s.daemonset.ready_nodes < 1, split by DaemonSet	Warning	Often the log or metric agent itself, which is how a cluster goes quiet without anything reporting that it did.
Node disk nearly full	k8s.node.filesystem.available < 2 GiB, split by node	Warning	The kubelet starts evicting pods under disk pressure.

Every check splits by the object it concerns, so a firing alert names the node, deployment, pod or job rather than telling you the cluster is unwell.

Tuning notes

• Deployment availability is a floor, not a ratio. The starter check fires only when a deployment reaches zero available replicas. To catch partial loss, raise the threshold to your replica count - a 3-replica deployment alerting below 3 tells you the moment one pod goes missing, at the cost of firing during ordinary rolling updates.
• Node disk needs your node size. 2 GiB is a reasonable floor for a small node and meaningless on a large one. Set it against the disk your nodes actually have.
• Pod failed uses an enum, not a count. k8s.pod.phase is 1 Pending, 2 Running, 3 Succeeded, 4 Failed, 5 Unknown, so the threshold is the phase value. ≥ 4 covers both Failed and Unknown.
• Node not ready reads a tri-state. k8s.node.condition_ready is 1 true, 0 false and -1 unknown, so < 1 catches both a node that reports NotReady and one the control plane has lost track of.
• k8s.node.condition is disabled by default, but you still get this metric. The receiver’s node_conditions_to_report option defaults to ["Ready"], and each reported condition becomes its own metric - hence k8s.node.condition_ready. Add conditions such as MemoryPressure or DiskPressure to that list if you want checks on them.
• Container restart loop counts restarts in the window, not the lifetime total, so a container that crashed a hundred times last month and has been stable since does not fire. A pod recreated mid-window resets its counter, which undercounts rather than over-reports.
• Job failing is worth keeping even on a quiet cluster. A nightly export that dies inside a Job produces no traffic and no errors anywhere else; the failed pod is the only evidence.