Monitor a file-drop integration (hot folder) with OpenTelemetry

File drops are still everywhere in integration: an SFTP upload, an export written to a share, a partner dropping a nightly batch. They fail quietly — a file lands and just sits there. This guide makes a hot folder observable two ways: a trace per file so you can follow one document end-to-end, and a folder-depth metric so a growing backlog is impossible to miss.

Before you start

You’ll need the folder your integration watches, and either a small hook in the code that processes files (Option A) or the OpenTelemetry Collector (contrib) for the no-code path (Option B). The two are complementary — most teams ship both.

Choose your signal

• A trace per file — best when you control the processor and want per-file timing, attributes (size, partner, type) and failures you can search.
• Folder metrics — best when you can’t touch the code: just watch how many files are waiting and how old the oldest one is.

Option A — Emit a trace per file

Wrap the per-file handler in a span. The attributes you set here become searchable fields in Sluicio, so add the ones you’d want to filter a failure by. (For SDK setup, see Instrument a service.)

Java

FileProcessor.java

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;

Tracer tracer = GlobalOpenTelemetry.getTracer("file-drop");

void processFile(Path file) throws IOException {
  Span span = tracer.spanBuilder("file.process").startSpan();
  try (var scope = span.makeCurrent()) {
    span.setAttribute("file.name", file.getFileName().toString());
    span.setAttribute("file.size", Files.size(file));
    handle(file);
  } catch (Exception e) {
    span.recordException(e);
    span.setStatus(StatusCode.ERROR);   // shows as a failed message
    throw e;
  } finally {
    span.end();
  }
}

.NET

FileProcessor.cs

using System.Diagnostics;

// Registered via .AddSource("file-drop") in your tracing config
static readonly ActivitySource Source = new("file-drop");

void ProcessFile(FileInfo file)
{
    using var activity = Source.StartActivity("file.process");
    activity?.SetTag("file.name", file.Name);
    activity?.SetTag("file.size", file.Length);
    try
    {
        Handle(file);
    }
    catch (Exception ex)
    {
        activity?.SetStatus(ActivityStatusCode.Error, ex.Message);  // failed message
        throw;
    }
}

JavaScript

processFile.js

const path = require('node:path');
const { trace, SpanStatusCode } = require('@opentelemetry/api');
const tracer = trace.getTracer('file-drop');

async function processFile(filePath, stat) {
  await tracer.startActiveSpan('file.process', async (span) => {
    span.setAttribute('file.name', path.basename(filePath));
    span.setAttribute('file.size', stat.size);
    try {
      await handle(filePath);
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR });   // shows as a failed message
      throw err;
    } finally {
      span.end();
    }
  });
}

Each file becomes one trace. A failed file is a failed span you can find — and re-drive — from message search.

Option B — Folder metrics with the Collector

No code path: the Collector’s file_stats receiver reports on the files matching a glob, so you get a live count (the backlog) and each file’s modification time (to find the oldest, stuck file).

This config targets Collector v0.152.0 or later. On older Collectors the receiver is called filestats and the exporter otlphttp.

otel-collector-config.yaml

receivers:
  file_stats:
    include: '/data/inbound/*'
    collection_interval: 60s

exporters:
  otlp_http/sluicio:
    endpoint: https://your-instance-name.sluicio.com
    headers:
      authorization: Bearer ${env:SLUICIO_INGEST_KEY}

service:
  pipelines:
    metrics:
      receivers: [file_stats]
      exporters: [otlp_http/sluicio]

docker run --rm \
  -v "$(pwd)/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml" \
  -v /data/inbound:/data/inbound:ro \
  -e SLUICIO_INGEST_KEY \
  otel/opentelemetry-collector-contrib:latest

Metric	What it tells you
file.count	Folder depth — how many files are waiting.
file.mtime	Per-file modification time — the oldest one is your stuck file.
file.size	File size — useful to spot a truncated or zero-byte drop.

Watch the age, not just the count

A folder with five files is fine — unless one of them has been sitting there since last night. Derive oldest-file age from the minimum file.mtime; a file older than your batch window means the integration stalled.

Alert on a growing backlog

• Backlog building — file.count stays above your threshold for N minutes.
• Stuck file — the oldest file’s age (now − min file.mtime) exceeds the batch window. This fires even when the count is low, which is exactly when a single wedged file would otherwise go unnoticed.

Verify in Sluicio

Where to go next

• Track RabbitMQ queue depth and consumer lag
• Monitor ActiveMQ Artemis queue depth and dead-letter growth
• Instrument a service — the SDK setup behind Option A.