Architecture overview
How Prelude Collector turns raw device protocol responses into structured snapshots delivered to your backends.
Prelude Collector is a long-running service that reaches out to your network devices, pulls operational state over the protocol you tell it to use, reshapes that raw data into a model you control, and forwards the result to one or more output backends. You configure it once, point it at your fleet, and it keeps a steady stream of structured telemetry flowing to wherever you need it — a time-series store, a message bus, a webhook, or a file on disk.
At the highest level, every piece of data that flows through the collector goes through the same four stages:
- Collect — open a session to a device using gNMI, NETCONF, SNMP, or CLI and pull a response.
- Map — bind values from that response onto the fields of a model.
- Transform — run optional functions on each field to clean, convert, or enrich the value.
- Output — package the result as a snapshot and publish it to each configured backend.
You manage all of this through a REST API and a built-in web UI. You do not need direct database access, and you do not need to recompile the collector to add a new device, model, or transform.
How it fits together
Collection: Devices, Protocols, and Subscriptions
A Device is the network element you want to monitor — a router, a switch, a firewall, an optical line system, anything that exposes operational data. Each device has an IP address and at least one Protocol record that describes how to reach it. The protocol record carries credentials, the port, and any protocol-specific options such as SNMP community strings, NETCONF capabilities, or gNMI TLS settings.
A Subscription binds a device to a model and tells the collector how often to pull data. When you enable a subscription, the collector starts a worker that wakes up on the configured interval, opens a session to the device using the protocol you chose, and asks for the fields the model declares. When you disable it, the worker stops. You can run many subscriptions per device — one per model — and the same device can be polled with different intervals for different sets of data.
Supported protocols cover the typical multi-vendor mix you find in a production network: gNMI for streaming telemetry on modern platforms, NETCONF for YANG-driven configuration and state, SNMP for legacy and read-only access, and CLI as a last resort when no programmatic interface is available. The same model can be served by different protocols on different devices, which lets you normalize data across Cisco, Juniper, Nokia, Arista, and other vendors without changing what your downstream systems consume.
Mapping and transforms: from raw bytes to structured fields
Raw protocol responses are not directly useful. A gNMI subscription returns paths and values that follow a vendor's YANG tree. An SNMP walk returns OIDs and counters. A CLI session returns lines of text. Before any of that is published, the collector reshapes it into a Model — a named schema you control that describes exactly which fields you care about and what they mean.
Each model has a set of Fields. Each field has a name, a type, and one or more Mappings that tell the collector where to find the value in the protocol response. A mapping can be a YANG path, an XPath, an SNMP OID, or a regular expression against CLI output, depending on the protocol. You author mappings once per device family and reuse the same model across the fleet.
Once a value is mapped, you can run a Transform on it. A transform is a small named function that takes a value in and produces a value out. The collector ships a library of built-in transforms for common conversions — units, scales, string parsing, type coercions, OID lookups against a MIB browser. When the built-ins are not enough, you write a custom transform in Starlark, a safe and Python-like scripting language, and the collector hot-reloads it without a restart. Transforms are how you keep the data shape stable even when the underlying protocol response changes.
Outputs: where the data goes
A complete pass through a subscription produces a Snapshot: the fully mapped, fully transformed instance of the model, tagged with the device, the timestamp, and the subscription that produced it. Snapshots are what the collector publishes downstream.
Every snapshot is sent to each configured Output. The collector supports several backends out of the box:
- NATS — the default backend, suitable for fan-out to multiple consumers and for chaining into the rest of your platform.
- Prometheus — for direct scrape or remote-write into a time-series store.
- InfluxDB — for line-protocol ingestion.
- Kafka — for high-throughput streaming pipelines.
- Webhook — for HTTP POST integrations into ticketing, observability, or custom systems.
- File — for archive, replay, and offline analysis.
You can enable any combination of outputs. The same snapshot fans out to all of them in parallel, so adding a new destination does not disturb existing ones. Output settings live in the database, which means you can change them through the API or the web UI without editing files on the host.
Putting it all together
A typical deployment looks like this:
- One collector instance per site or per region, packaged as an OCI image and run under Docker, Podman, or systemd.
- A handful of models that describe the operational data you care about — interface counters, BGP state, optical levels, environment sensors, and so on.
- One subscription per (device, model) pair, with the interval tuned to the rate of change you actually need.
- One or two outputs that match your existing observability stack.
From there, the day-to-day work is adding new devices, refining models as your needs evolve, and writing the occasional custom transform when a vendor returns data in an unusual shape.