Your question
I am demo-ing enterprise, and would like to know how to look up an email that made it to an inbox and see all the logs for it from end to end. Is there some sort of unique ID for that? How are others doing this with a cluster or even a single server?
This might be the queueID perhaps?
With a cluster I need to send logs from all three servers to something like VictoriaLogs and metrics to Prometheus or something similar.
How do I get the dashboards of each server to see the data from all three servers? Any way to see logs combined in the GUI of each server?
I know this is a big question, the assistance is appreciated.
Ken
I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.
on
Good — queueId is indeed the trace field. Let me check the Trace object reference and search.md for trace queries, and also the metrics/opentelemetry docs regarding multi-server dashboards.
Stalwart (Enterprise) has a built-in answer for this: the Trace object.
Finding a message end-to-end
Every message accepted for delivery gets a Trace record (Enterprise feature) that bundles together the inbound reception, queueing, outbound delivery attempts, and outcome of each hop as a set of associated events. Traces are queryable by id, account, remote host, domain, and time range, and surface in the WebUI under Emails › History › Inbound Delivery / Outbound Delivery. One of the indexed/searchable fields on a trace is queueId, so yes, the queue ID is effectively the correlator you’re thinking of, you can search history by it, or just look up the message by sender/recipient/time in that History view and see the whole trace at once. Retention of trace history is controlled by holdTracesFor on the DataRetention object (default 30 days), and where traces are stored is controlled by the TracingStore singleton.
Docs: /docs/telemetry/management, /docs/telemetry/history, /docs/ref/object/trace
Cluster: combining logs/metrics/dashboards across nodes
Two separate mechanisms matter here:
- Metrics and trace history: The
TracingStore and MetricsStore singletons can each be set to Default, meaning they reuse the main shared data store. In a cluster where all nodes point at the same data store, trace and metric history is naturally shared, so the WebUI Dashboard/History view on any node in the cluster will show the same combined data for the whole cluster, no extra work needed as long as they share a data store. (/docs/telemetry/history)
- Live tracing/logging output: Each node also has its own set of
Tracer objects (per-node config) that can push traces/logs out in real time to an OpenTelemetry collector (OTLP over HTTP or gRPC), a log file, systemd journal, or stdout. To centralize logs into something like VictoriaLogs, or metrics into Prometheus, you’d configure each node’s Tracer with the OtelHttp/OtelGrpc variant pointing at your OTel collector, which can then fan out to VictoriaLogs/Prometheus/Jaeger/etc, or use the dedicated Prometheus metrics exporter variant. This is configured per node (not automatically shared), so all three servers in your cluster would each need their own Tracer entries pointing at the same collector endpoint. (/docs/telemetry/tracing/, /docs/telemetry/tracing/opentelemetry, /docs/telemetry/metrics/opentelemetry, /docs/telemetry/metrics/prometheus)
So: for querying “what happened to this email”, use the Trace/History view (shared across the cluster if using a shared data store), and for live streaming logs/metrics into your own observability stack, point each node’s Tracer/OTel exporter at your collector.
This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.
I have set up a VictoriaLogs server, and configured an OpenTelemetry Tracer and pointed it towards the VL HTTP endpoint. I have log data coming in and it looks good, but theres not a queueID field in any of the data.
The id you want is queueId (a 64-bit number). Every accepted message is enqueued for delivery, so it gets a queueId, and both the intake session and the later delivery log it, which lets you follow a message across the two. Within a single connection there’s also spanId, the session id, on every event of that session.
The reason you don’t see queueId on most lines is that it’s only attached to queue and delivery-stage events, not to every event (a connection or auth event carries spanId but not queueId). In VictoriaLogs, filter to the delivery and queue events and you’ll find it, on both the intake and the delivery side with the same value. For a combined cross-node view, point all three servers at the same VictoriaLogs and Prometheus and build the dashboard there.
The per-server WebUI only shows the logs of its own node, so there’s no single combined view inside the GUI itself. Stalwart Enterprise also has the “Delivery History” feature which allows you to trace message delivery and search by recipient, domain or IP. If you want to achieve the same “Delivery History” functionality on your telemetry platform, you need to look at the spans rather than the logs since these group multiple events by spanId and, if your telemetry solution allows it, you can also search spans by queueId, IP or domain.