Background Services
Making invisible background jobs observable
Background workers rarely have a UI. If you don’t invest in observability, the only debugging tool you have is “did it run?” and “is the queue growing?”.
1. Start with simple metrics
I like to begin with:
- Processed count (by outcome: success/failure).
- Processing duration (histogram or percentiles).
- Queue depth or pending-items gauges.
2. Add structured logging
Instead of plain text, log structured properties:
- Job name / worker name.
- Correlation ID / trace ID.
- Key identifiers (user, order, account).
3. Health checks that reflect reality
A health check that only says “process is running” isn’t helpful. I prefer checks that:
- Verify the worker processed something recently.
- Check latency thresholds (e.g., oldest message age).
- Fail fast if error rates spike.