Issue Description
In a multi-node cluster with the Redis coordinator, the broadcast subscriber loop in crates/services/src/broadcast/subscriber.rs processes exactly one message per subscription: the tokio::select! on stream.next() is not wrapped in an inner read loop, so after each received broadcast the outer loop calls coordinator.subscribe() again. For the Redis backend (crates/coordinator/src/backend/redis/pubsub.rs), redis_subscribe opens a brand-new pub/sub connection (get_async_pubsub()) on every call. Since Redis pub/sub is fire-and-forget, every broadcast received tears down the subscription on all nodes simultaneously, and any message published during the reconnect window is lost cluster-wide. A single mail delivery publishes a short burst of separate broadcast messages (cache invalidation, push notification), so the first message routinely causes the rest of the burst to be dropped.
Suggested fix: keep one long-lived pub/sub subscription per node and read messages from it in an inner loop, only re-subscribing when stream.next() returns None (disconnect) or on error. The current per-message re-subscribe also affects the other coordinator backends structurally, but only Redis loses messages because the transport has no replay.
Expected Behavior
A mailbox change on node B is broadcast via the coordinator and node A delivers an IMAP IDLE/push notification within milliseconds. The pub/sub subscriber connection stays established for the lifetime of the process.
Actual Behavior
Notifications are intermittently lost; clients idling on the other node only see new mail at their next poll (minutes later). On the Redis server, each node’s subscriber connection is recreated after every received broadcast: CLIENT LIST shows long-lived pooled connections (age 6–17 h) but subscriber connections (flags=P) only minutes old, with idle == age, and all nodes’ subscribers reconnect in the same second.
Reproduction Steps
- Run a 2-node Stalwart cluster with the Redis coordinator (standalone Redis or Dragonfly).
- Connect an IMAP client to node A and enter IDLE on INBOX.
- Deliver a message via SMTP that is ingested on node B.
- Observe that the IDLE notification frequently does not arrive (client sees the message only on its next poll). Meanwhile redis-cli CLIENT LIST shows the flags=P subscriber connections from both nodes being replaced each time a message is published to stwt.agora.
Relevant Log Output
Node B (ingest side) delivery completes normally:
2026-07-24T07:04:27Z INFO Queued message for delivery (queue.message-queued) listenerId = “smtp”, localPort = 25, remoteIp = 209.85.160.49, from = “…@gmail.com”, to = [“test@…”]
2026-07-24T07:04:28Z INFO Message ingested (message-ingest.ham) queueId = 320018405522407937, accountId = 1, documentId = 52, mailboxId = [0], elapsed = 30ms
2026-07-24T07:04:28Z INFO Delivery completed (delivery.completed) elapsed = 1000ms
Node A (IDLE side) no cluster.message-received for the ingest; nothing logged at all (the subscriber was between subscriptions; message silently lost). Subscriber start is logged only once at boot:
2026-07-23T19:52:12Z INFO Clustering enabled (cluster.startup) id = 0
2026-07-23T19:52:12Z INFO PubSub subscriber started (cluster.subscriber-start)
Redis-side evidence (CLIENT LIST, both Stalwart pods 11–15 h old; subscriber connections recreated together, no traffic since creation):
id=29271 addr=10.1.1.152:40910 age=349 idle=349 flags=P lib-name=redis-rs lib-ver=1.2.0
id=29272 addr=10.1.1.164:35644 age=349 idle=349 flags=P lib-name=redis-rs lib-ver=1.2.0
id=17885 addr=10.1.1.152:37916 age=56081 idle=0 (regular pooled connection, same pod)
Stalwart Version
v0.16.x
Installation Method
Built from source
Database Backend
FoundationDB
Blob Storage
S3-compatible
Search Engine
Elasticsearch
Directory Backend
Internal
I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.
on
I have searched this support forum (open and closed topics) and confirm this is not a duplicate.
on
I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.
on