Production hardening

Why this matters

The Collector holds three sensitive things at once: device credentials, encrypted backend secrets, and the live data path between your network and your downstream systems. A "we'll harden it later" instance is the one that ends up exposed when later turns into never. The checklist below is short on purpose — every item is something that has bitten real deployments.

The checklist

Walk this once for each Collector instance before it carries production load.

1. TLS on the API and UI

• HTTPS is always on. For production, replace the auto-generated self-signed certificate with a real one: set both COLLECTOR_TLS_CERT and COLLECTOR_TLS_KEY to the in-container paths of your fullchain certificate and private key (do not rely on the self-signed cert beyond a trusted network). If either is unset, the collector falls back to the self-signed cert.
• Front the Collector with a reverse proxy if you need to terminate TLS centrally and forward to port 4030.
• Renew certificates on a schedule that runs without you. The day a cert silently expires is the day you find out which dashboards depended on the Collector.

2. TLS to every Output

• NATS — use tls:// URLs in the NATS output backend's connection settings (configured in the web UI under Output Settings → NATS). Treat the bus like any other production data plane.
• Prometheus remote-write — HTTPS with a credential, not a cleartext push.
• InfluxDB, Kafka, webhook — TLS on every connection. Most outages traced to "we forgot to encrypt this one hop" happen on the output side, not on the API side.

3. TLS to devices

• gNMI sessions should run over TLS with proper cert validation.
• NETCONF should run over SSH with host-key pinning where the device supports it.
• SNMP should be v3 with authPriv for anything outside an isolated management VLAN. SNMPv2c is for lab and inventory only.

4. Encryption key hygiene

The AES-256 key at storage/keys/prelude-collector.key encrypts device passwords and Output credentials in the Collector database. It is generated automatically on first start. If you lose it, those rows become unreadable.

• Persist the storage/ volume and back up storage/keys/.
• Keep an offline copy somewhere you can find at 3 a.m. on a weekend.
• Do not delete or replace it after data has been written. Existing encrypted values cannot be re-encrypted in place; replacing the key requires re-entering credentials for every Device and Output.

5. API token rotation

API tokens are bearer credentials. Treat them like passwords.

• Issue tokens per consumer (one per service, one per CI pipeline, one per human operator), never a shared token.
• Store tokens in your secret manager. Never commit them. Examples in the docs use <your-api-token> as a placeholder for a reason.
• Rotate on a calendar (90 days is a reasonable starting point) and on departures.
• Revoke immediately when a token leaks; the Collector will reject the next request that uses it.

6. Container resource limits

The Collector tolerates being squeezed; it does not tolerate being unpredictable. Set both requests and limits so the platform can schedule honestly.

• CPU — a 2 vCPU limit fits a typical install up to a few hundred Subscriptions. Raise before you push toward max-subscriptions = 1000.
• Memory — start at 1 GiB. Watch the memory_alloc field on GET /api/v1/metrics; raise the limit if you see steady growth toward the ceiling.
• Disk — the Collector itself uses little disk; PostgreSQL, YANG models in storage/yang/, and MIBs in storage/mibs/vendor/ do. Mount the storage folder on a volume you can resize.
• Restart policy — unless-stopped (or its Kubernetes equivalent). The Collector recovers cleanly from restarts; do not fight it with manual interventions.

7. Tune the concurrency knobs you actually use

Do not leave the defaults if your fleet is bigger than the example fleet:

• max-subscriptions caps how many Subscriptions the Collector will run at once.
• worker-pool-size is the SNMP / NETCONF / CLI pool. Raise it if those protocols are your dominant load.
• gnmi.subscription-stagger-delay and gnmi.max-concurrent-subscriptions keep streaming subscriptions from overwhelming a single device on startup.

If you are not sure where your bottleneck is, look at the runtime metrics dashboard (web UI) or GET /api/v1/metrics before you guess.

8. Monitor the Collector itself

Self-monitoring is not optional. Scrape the Collector the same way you scrape every other production service.

• Hit GET /api/v1/health (Bearer-token authenticated) on :4030 for the evaluated severity of every subscription — this is what you alert on.
• Read GET /api/v1/metrics (or scrape the JSON with a sidecar) for Subscription health, queue depths, drop counters, and process-level RSS / CPU. To get device telemetry into Prometheus, attach the Prometheus output backend (default port 9090, path /metrics/collector).
• Alert on: system-severity going critical, Output queue depth increasing for more than five minutes, and license expiring inside 30 days.
• Forward Collector logs into the same log stack you already run. Set debug = false in production — debug logging is verbose and the signal-to-noise ratio in production is worse than you expect.

9. Back up the database

The Collector's PostgreSQL database holds Devices, Models, Mappings, custom Transforms, Subscriptions, and Output configurations. Rebuilding it from scratch is days of work.

• Run a nightly pg_dump (or your platform's managed equivalent).
• Keep at least seven daily snapshots and one monthly snapshot.
• Test the restore on a non-production Collector at least quarterly. Backups you have not restored from are aspirational, not operational.
• Back up the storage/ folder on the same cadence — YANG and MIB files matter for replays.

10. Document the runbook

Hardening that lives only in someone's head is fragile.

• Where the encryption key is stored and how to retrieve it.
• Who issues and rotates API tokens.
• How to scale (vertical first, then a second Collector with the fleet partitioned).
• Where logs and metrics land, and which dashboards are canonical.
• Restore procedure for the database.

Trade-offs

What you give up by adopting this checklist:

• Speed of change. TLS, secret management, and rotation policies add friction to "just spin up another Collector." That friction is the point in production.
• Tooling familiarity. If your team has never run PostgreSQL backups, NATS over TLS, or Prometheus self-scrape, you will spend a week getting comfortable. Spend it.
• Sample density. Resource limits force you to think about the cost of a tighter polling interval. That is correct, but it is a change from PoC, where you were free to over-poll.

When to deviate

• You are running an isolated lab. Skip TLS on internal links if the network is genuinely isolated and disposable. Do not skip the encryption key backup; lab data still costs effort to recreate.
• You are running a multi-tenant Collector. Add per-tenant API tokens and per-tenant Subscriptions; the checklist above is the minimum, not the ceiling.
• You are in a regulated environment. Your compliance team almost certainly has stronger requirements (HSM-backed keys, audit log shipping, change-control workflow). Treat this checklist as a starting point and let the regulator's checklist override it where they conflict.
• You are running the Collector as a systemd unit instead of a container. Container resource limits become systemd directives (MemoryMax, CPUQuota); the principles do not change.