Paperless-ngx

Paperless-ngx is an open-source document management system: it watches a folder, an inbox and an API for incoming documents, runs OCR, classifies them, and files them into a searchable archive. That intake is a pipeline, and pipelines fail in ways a “is it up” check never sees - a scanner feeding unreadable PDFs, an OCR stage that quietly doubles in duration, a queue that stops draining while the web UI stays perfectly responsive.

Key: paperless-ngx · applied automatically to services emitting paperless_-prefixed metrics.

Getting the telemetry

Paperless-ngx ships no OpenTelemetry support and no metrics endpoint. Both signals come from outside it, and both are needed - they answer different questions.

Traces come from Python auto-instrumentation layered onto the published image. Nothing in paperless changes; a sitecustomize.py on PYTHONPATH instruments every process the container starts, and each reports as its own service (paperless-webserver, -worker, -consumer, -scheduler).

Metrics come from prometheus-paperless-exporter, which reads paperless’ REST API with least-privilege view permissions. Scrape it with the Collector’s prometheus receiver:

otel-collector.yaml (excerpt)

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: paperless-exporter    # becomes service.name
          scrape_interval: 30s
          static_configs:
            - targets: [exporter:8081]

processors:
  # paperless_task_status is one series per task id. Aggregate the id away
  # and sum: pending becomes queue depth, failure becomes an error count.
  metricstransform/tasks:
    transforms:
      - include: paperless_task_status
        action: update
        operations:
          - action: aggregate_labels
            label_set: [status]
            aggregation_type: sum

A working end-to-end setup - compose file, instrumented image, scenario scripts that drive every ingest outcome - is published at github.com/syron/paperless-ngx-otel.

What Sluicio detects

Metrics whose names start with paperless_, emitted by the exporter. Auto-instrumentation alone produces only process.*, system.* and flower.* - far too generic to claim a system type from, so detection requires the exporter. Traces alone won’t identify the system.

Starter checks

Check	Condition	Severity	Why it matters
Document ingest failed	Error traces in 15 minutes	Warning	A document failed to consume. The trace names the document and the stage.
New ingest failures	paperless_task_status{status="failure"} increasing (delta)	Warning	Overlaps the trace check deliberately - this one still fires when the worker’s telemetry pipeline is down.
Slow document ingest	p95 consume time > 60,000 ms	Warning	OCR dominates ingest time; a step change usually means bigger scans or a starved worker.
Ingest backlog	paperless_task_status{status="pending"} > 25	Warning	Tasks are queued and not draining - a wedged worker, or intake outrunning OCR.
Documents unfiled	paperless_statistics_documents_inbox_count > 100	Warning	Documents consumed fine but nobody has filed them. A business backlog, not a fault.
Celery worker unhealthy	paperless_status_celery_status < 1	Critical	Nothing will consume - and no trace will be emitted to tell you so.
Redis unhealthy	paperless_status_redis_status < 1	Critical	The broker paperless queues through is unreachable.
Database unhealthy	paperless_status_database_status < 1	Critical	Paperless cannot reach its database.
Unapplied migrations	paperless_status_database_unapplied_migrations > 0	Warning	The image was upgraded but migrations never ran - a half-upgraded instance.
Low document storage	paperless_status_storage_available_bytes < 2 GiB	Warning	Paperless fails ingest hard when document storage runs out.
Classifier stale	Last trained more than 7 days ago	Warning	Auto-tagging quality drifts silently, with no failure anywhere.

Two prerequisites, or half these checks are dead

The paperless_task_status aggregation shown above is required. Without it the metric is one series per task id - unbounded, and not a count of anything, so the backlog and failure-count checks never evaluate. Do not substitute paperless_task_status_info: despite the name it is a label registry the exporter sets to 1 once per status string it has seen, so every series reads 1 forever.

The Celery worker must not recycle its pool children. Paperless hardcodes worker_max_tasks_per_child = 1, so each child is killed the moment its task returns - while the span processor still holds that task’s spans, waiting on a 5-second batch interval. They die with the child, silently: no error, no dropped-span counter. Every trace-based check goes permanently quiet while ingest itself looks healthy. Paperless doesn’t expose the setting, so run the worker with --pool solo (no children to recycle) until it’s fixed in opentelemetry-python. The metric checks are unaffected - which is precisely why this type carries both kinds.

Tuning notes

• Ingest latency. 60 seconds is generous because it has to cover OCR on a multi-page scan. If you ingest mostly digital PDFs with a text layer, tighten it hard - those complete in well under a second, and 60 s would hide a tenfold regression.
• Backlog threshold. 25 pending tasks suits a household or small office. A bulk import legitimately queues thousands; raise it, or silence the check during migrations.
• Unfiled documents. This one tracks a human habit, not system health. If you file in weekly batches it will nag every week - raise it or disable it.
• Duplicate rejections do not appear as failures in traces. consume_file returns a duplicate result rather than raising, so the span looks identical to a success. Paperless does record the task as a failure, so the metric check catches it - expect the two failure checks to disagree, and treat the metric as the broader signal.
• A silent instance is normal. No dead-man check ships with this type on purpose: most paperless instances legitimately go hours without a document. If yours has continuous intake, add a trace-volume check yourself.

Related

• github.com/syron/paperless-ngx-otel - the instrumentation recipe, with measured findings about what auto-instrumentation can and cannot see, and a minimal reproduction of the span-loss bug behind the caution above.
• prometheus-paperless-exporter - the metrics source.