Monitor a file-drop integration (hot folder) with OpenTelemetry

File drops are still everywhere in integration: an SFTP upload, an export written to a share, a partner dropping a nightly batch. They fail quietly — a file lands and just sits there. This guide makes a hot folder observable two ways: a trace per file so you can follow one document end-to-end, and a folder-depth metric so a growing backlog is impossible to miss.

Before you start

You’ll need the folder your integration watches, and either a small hook in the code that processes files (Option A) or the OpenTelemetry Collector (contrib) for the no-code path (Option B). The two are complementary — most teams ship both.

Choose your signal

• A trace per file — best when you control the processor and want per-file timing, attributes (size, partner, type) and failures you can search.
• Folder metrics — best when you can’t touch the code: just watch how many files are waiting and how old the oldest one is.

Option A — Emit a trace per file

Wrap the per-file handler in a span. The attributes you set here become searchable fields in Sluicio, so add the ones you’d want to filter a failure by. (For SDK setup, see Instrument a service.)

Java

FileProcessor.java

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;

Tracer tracer = GlobalOpenTelemetry.getTracer("file-drop");

void processFile(Path file) throws IOException {
  Span span = tracer.spanBuilder("file.process").startSpan();
  try (var scope = span.makeCurrent()) {
    span.setAttribute("file.name", file.getFileName().toString());
    span.setAttribute("file.size", Files.size(file));
    handle(file);
  } catch (Exception e) {
    span.recordException(e);
    span.setStatus(StatusCode.ERROR);   // shows as a failed message
    throw e;
  } finally {
    span.end();
  }
}

.NET

FileProcessor.cs

using System.Diagnostics;

// Registered via .AddSource("file-drop") in your tracing config
static readonly ActivitySource Source = new("file-drop");

void ProcessFile(FileInfo file)
{
    using var activity = Source.StartActivity("file.process");
    activity?.SetTag("file.name", file.Name);
    activity?.SetTag("file.size", file.Length);
    try
    {
        Handle(file);
    }
    catch (Exception ex)
    {
        activity?.SetStatus(ActivityStatusCode.Error, ex.Message);  // failed message
        throw;
    }
}

JavaScript

processFile.js

const path = require('node:path');
const { trace, SpanStatusCode } = require('@opentelemetry/api');
const tracer = trace.getTracer('file-drop');

async function processFile(filePath, stat) {
  await tracer.startActiveSpan('file.process', async (span) => {
    span.setAttribute('file.name', path.basename(filePath));
    span.setAttribute('file.size', stat.size);
    try {
      await handle(filePath);
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR });   // shows as a failed message
      throw err;
    } finally {
      span.end();
    }
  });
}

Each file becomes one trace. A failed file is a failed span you can find — and re-drive — from message search.

Option B — Folder metrics with the Collector

No code path: the Collector’s filestats receiver reports on the files matching a glob, so you get a live count (the backlog) and each file’s modification time (to find the oldest, stuck file).

otel-collector-config.yaml

receivers:
  filestats:
    include: '/data/inbound/*'
    collection_interval: 60s

exporters:
  otlphttp/sluicio:
    endpoint: https://your-instance-name.sluicio.com
    headers:
      authorization: Bearer ${env:SLUICIO_INGEST_KEY}

service:
  pipelines:
    metrics:
      receivers: [filestats]
      exporters: [otlphttp/sluicio]

docker run --rm \
  -v "$(pwd)/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml" \
  -v /data/inbound:/data/inbound:ro \
  -e SLUICIO_INGEST_KEY \
  otel/opentelemetry-collector-contrib:latest

Metric	What it tells you
file.count	Folder depth — how many files are waiting.
file.mtime	Per-file modification time — the oldest one is your stuck file.
file.size	File size — useful to spot a truncated or zero-byte drop.

Watch the age, not just the count

A folder with five files is fine — unless one of them has been sitting there since last night. Derive oldest-file age from the minimum file.mtime; a file older than your batch window means the integration stalled.

Alert on a growing backlog

• Backlog building — file.count stays above your threshold for N minutes.
• Stuck file — the oldest file’s age (now − min file.mtime) exceeds the batch window. This fires even when the count is low, which is exactly when a single wedged file would otherwise go unnoticed.

Verify in Sluicio

Where to go next

• Track RabbitMQ queue depth and consumer lag
• Monitor ActiveMQ Artemis queue depth and dead-letter growth
• Instrument a service — the SDK setup behind Option A.