File output

The File output writes collected telemetry to local files on the collector host. It supports JSON Lines (jsonl) and CSV formats and optional size-based rotation. After the first mention this page refers to Prelude Collector as "the collector".

When to use

Pick File when you want a local copy of collected data for debugging, archival, or feeding a file-based ingestion pipeline (Filebeat, Vector, Fluent Bit, log shippers). It is not the right Output for production query — pair it with NATS, InfluxDB, or Kafka for that. See Output selection.

How it works

One file is created per model name in the configured output directory.

When rotation is disabled (rotate-after-mb: 0, the default), filenames are stable across restarts and the collector keeps appending to one file per model:

/data/collector/cpu.jsonl
/data/collector/interface.jsonl

When rotation is enabled (rotate-after-mb > 0), every file — including the very first one for a given model — is written under a timestamped name (see File rotation below).

Files are opened in append mode, so existing data is never overwritten on restart. Model names are passed through filepath.Base before use as a filename, which strips any path separators in the input — strict path-traversal prevention (rejecting .., drive letters, etc.) is not implemented; rely on filesystem permissions on output-dir.

Formats

JSON Lines (jsonl)

Each line is a JSON-serialised ParsedData envelope followed by a newline. Field names are kebab-case (device-id, model-name, data-hash). The format is compatible with jq, ClickHouse, BigQuery, DuckDB, and most log ingestion tools.

{"device":"router-01","device-id":42,"model-name":"cpu","path":"…","key":"cpu","timestamp":"2024-01-01T12:00:00Z","data-hash":"sha256:8f3c…","data":{"usage":45.2,"cores":8}}
{"device":"router-02","device-id":43,"model-name":"cpu","path":"…","key":"cpu","timestamp":"2024-01-01T12:00:05Z","data-hash":"sha256:b2e1…","data":{"usage":12.1,"cores":4}}

CSV

The first row is a header with timestamp, device, device-id, model, key, followed by all data field names sorted alphabetically. Subsequent rows contain the corresponding values.

Only top-level scalar fields are written. Nested maps and slices are skipped.

timestamp,device,device-id,model,key,cores,usage
2024-01-01T12:00:00Z,router-01,42,cpu,cpu,8,45.2
2024-01-01T12:00:05Z,router-02,43,cpu,cpu,4,12.1

CSV column headers are determined from the first record written for each model. New fields appearing in later records are not added retroactively.

File rotation

When rotate-after-mb is 0 (the default), no rotation occurs and a single stable filename per model is used indefinitely ({model}.jsonl or {model}.csv).

When rotate-after-mb is set to a non-zero value, every file for a given model is written under a timestamped name — including the first one. The pattern is {model}-YYYYMMDD-HHMMSS.{ext}, with the timestamp captured at file-open time:

/data/collector/cpu-20240101-120000.jsonl
/data/collector/cpu-20240101-130512.jsonl

A new file is opened on the next write that finds the current file's size at or above the threshold, so the actual on-disk size of a rotated file may overshoot the threshold by one record.

Connection settings

Configure this backend through PUT /api/v1/outputs/file.

Authentication

Not applicable — this Output writes to the local filesystem. Apply standard filesystem permissions on output-dir so only the collector process and authorized operators can read it.

Configuration example

{
  "enabled": true,
  "config": {
    "output-dir": "/var/lib/prelude/collector-output",
    "format": "jsonl",
    "rotate-after-mb": 100
  }
}

Apply it with:

export BASE="https://collector.example.com"
export TOKEN="<your-api-token>"

curl -s -X PUT "$BASE/api/v1/outputs/file" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d @file-output.json

Bruno: 08 Outputs / Update output config

Behavior on failure

Writes go through a buffered writer per file. Flush() flushes any open CSV writer and calls Sync() on each file handle, ensuring data reaches the OS — this runs on graceful shutdown. Close() flushes and closes all open file handles; the output directory is not removed.

A failed write (full disk, permission error, removed directory) is counted as a failure on /api/v1/outputs/metrics and logged. There is no automatic retry — fix the underlying condition and reload the Output.

To validate writability proactively, use POST /api/v1/outputs/file/detect, which probes dir-exists and writable against the configured output-dir.

Limitations

• Only jsonl and csv formats are supported.
• CSV column set is locked to the first record for each model; later fields are dropped.
• For CSV, only top-level scalar fields are written; nested data is skipped.
• Rotation is size-based only; there is no time-based rotation and no built-in compression of rotated files.
• The Output does not bound total disk usage beyond per-file rotation. Monitor available disk space, especially under high collection frequency.
• The collector does not delete old rotated files — pair with logrotate or a cron-based cleanup if you need a retention cap.

See also

• Output backends
• Output selection
• API reference