Operations

What to watch, what to alert on, and what to do at three in the morning when the notifications stop.

Health surface

A worker reports { status: "unknown", message: "No report received from worker" } when its heartbeat is missing. In a split deployment that is expected: the API process cannot see workers you did not deploy. In a services: ["all"] process it means that worker died.

Metrics

Default Node.js process metrics (heap, event loop lag, GC) are exported alongside these.

Alerts worth having

When nothing is arriving

The dead-letter queue

A message that exhausts its attempts is moved to notifkit:stream:dlq. Nothing is retried from there automatically: a poison payload would just loop.

# Most recent 50
curl -s http://localhost:3000/v1/dlq -H "Authorization: Bearer $NK_KEY" | jq

# Put one back at the front of the pipeline
curl -X POST http://localhost:3000/v1/dlq/replay \
  -H "Authorization: Bearer $NK_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id":"1737054981234-0"}'

# Discard one
curl -X DELETE http://localhost:3000/v1/dlq/1737054981234-0 -H "Authorization: Bearer $NK_KEY"

A replay re-publishes the entry to the inbound stream at its original priority and removes it from the DLQ. Fix the underlying cause first (a bad template, a missing transport, an expired credential), or the same message lands right back.

Configuration

Constructor options beat environment variables, which beat .env, which beats the defaults.

Per-project limits

Each project has its own API request ceiling: 600 requests per minute by default, enforced as a sliding window in Redis. Over it, callers get 429 with Retry-After: 60.

curl -X PATCH http://localhost:3000/v1/projects/{projectId} \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"rateLimitRpm": 3000, "throttleLimit": 50, "throttleWindowHours": 24}'

Requests made with the admin key are allowed 6 000 per minute.

Changes take effect on the next notification: the engine caches these per project for 60 seconds, and a PATCH publishes an invalidation so every worker drops its copy immediately. Setting throttleLimit to 0 stops all non-critical sends for that project: a usable kill switch. critical priority still bypasses the throttle regardless.

Key rotation

# Mint the replacement
curl -X POST http://localhost:3000/v1/projects/{projectId}/keys \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" -d '{"role":"admin"}'

# List what exists — returns id, role, and createdAt. Never the key itself.
# Take the id of the key you are retiring; that is the {keyId} below.
curl -s http://localhost:3000/v1/projects/{projectId}/keys -H "Authorization: Bearer $ADMIN_API_KEY"

# Revoke the old one
curl -X DELETE http://localhost:3000/v1/projects/{projectId}/keys/{keyId} \
  -H "Authorization: Bearer $ADMIN_API_KEY"

Deploy the new key, then revoke the old one. Revocation publishes a cache invalidation to every API process, so it takes effect immediately instead of after the 60-second auth cache expires.

Give read_only keys to anything that only reads: dashboards, status pages, reporting jobs. They are rejected with 403 on any non-GET request.

Suppressions and deliverability

The suppression list is the record of people who asked to stop hearing from you, plus addresses that no longer exist. It is consulted before every dispatch and priority: "critical" does not override it.

# Everything suppressed, newest first
curl -s "http://localhost:3000/v1/suppressions?limit=200" -H "Authorization: Bearer $NK_KEY"

# Only the hard bounces — a spike here usually means a stale or bought list
curl -s "http://localhost:3000/v1/suppressions?reason=bounced" -H "Authorization: Bearer $NK_KEY"

# Someone asked to be removed by phone or reply
curl -X POST http://localhost:3000/v1/suppressions -H "Authorization: Bearer $NK_KEY" \
  -H "Content-Type: application/json" \
  -d '{"channel":"email","target":"person@example.com","reason":"manual"}'

Two numbers are worth watching per campaign, both from GET /v1/campaigns/{campaign}/stats:

Shutdown

SIGINT and SIGTERM trigger a graceful stop: the HTTP server closes, consumers stop reading, in-flight messages are allowed to finish (up to 30 seconds), buffered log rows are flushed, and connections close. Anything unfinished at the deadline is left unacked and reclaimed by another worker: nothing is lost.

Give your orchestrator a termination grace period above 30 seconds so it does not SIGKILL mid-flush.

// Shutting down explicitly, e.g. from a test harness.
await server.stop();

Production checklist

• NODE_ENV=production, with REDIS_URL and DATABASE_URL set explicitly.
• autoMigrate: false everywhere except the one deployment that owns migrations.
• /health and /metrics unreachable from the public internet.
• ADMIN_API_KEY unset on deployments that do not manage projects.
• Postgres max_connections sized for services × processes × DB_MAX_CONNECTIONS.
• Recommended: Redis persistence on (AOF). In-flight messages and pending lists live there, so without it a Redis restart loses whatever is still queued. Managed Redis without AOF, such as ElastiCache, works with that trade-off.
• Termination grace period above 30 seconds.
• QUEUE_MAX_LEN above your worst expected backlog, since anything past it is trimmed unsent.
• TRUST_PROXY=true if, and only if, notifkit sits behind a reverse proxy.
• Alerts on stream depth, delivery failure ratio, and DLQ length.
• Transports registered on every delivery deployment, and on api if you use provider webhooks.