Intermittent IMAP IDLE delays: remote EmailPush is processed against an older cache state

Issues
1

Issue Description

Clients in IMAP IDLE state are experiencing delays in notifications about new email

Expected Behavior

After receiving a push for committed mailbox changes, the IMAP frontend promptly reports the updated INBOX count and new-message data without requiring another delivery or polling.

Actual Behavior

  1. 18:42:31.532–18:42:31.567: One external message produced two local deliveries, 35 ms apart, for Accounts A and B in the same Thunderbird instance. All three frontends received both EmailPush events.
  2. 18:42:31: Account A followed the expected path on stalwart-2:
    EmailPush(documentId=77, changeId=236) → store.cache-update changeId=236 → imap.raw-output * 5 EXISTS → imap.fetch documentId=77.
  3. 18:42:31: Account B followed the failing path on stalwart-0: EmailPush(documentId=374, changeId=766) → store.cache-hit changeId=765. Stalwart sent no EXISTS or FETCH for document 374.
  4. 18:47:15–18:47:19: Without user action, Thunderbird automatically ended IDLE after 591.100 seconds. The same Account B session then logged store.cache-update changeId=766, imap.raw-output * 316 EXISTS, imap.fetch documentId=374, and * 316 FETCH (UID 316 ...), before returning to IDLE. The message therefore appeared about 4 minutes 44 seconds after ingest.

Reproduction Steps

  1. Run four Stalwart processes against the same FoundationDB data store and Redis
    coordinator:

    Three frontend replicas accepting SMTP and IMAP traffic, with outboundMta disabled. One separate outbound worker with outboundMta enabled.

  2. create two accounts: A and B and connect them to thunderbird - enable ‘Allow
    immediate server notifications when new messages arrive’ in thunderbird

  3. send one email to A and with CC to B - make sure thunderbird connected to different IMAP service instances

  4. observe notification discrepancies

Relevant Log Output

2026-09-13T18:42:31.532655772Z stalwart-outbound message-ingest.ham       account=A
documentId=77  mailboxId=[0] changeId=236
2026-09-13T18:42:31.567203324Z stalwart-outbound message-ingest.ham       account=B
documentId=374 mailboxId=[0] changeId=766
2026-09-13T18:42:31Z     stalwart-0/1/2  cluster.message-received   EmailPush(account=A,
documentId=77,  changeId=236)
2026-09-13T18:42:31Z     stalwart-0/1/2  cluster.message-received   EmailPush(account=B,
documentId=374, changeId=766)
2026-09-13T18:42:31Z     stalwart-2      store.cache-update         account=A changeId=236
elapsed=4ms
2026-09-13T18:42:31Z     stalwart-2      imap.raw-output            account=A "* 5 EXISTS"
2026-09-13T18:42:31Z     stalwart-2      imap.fetch                 account=A documentId=77
mailboxId=0
2026-09-13T18:42:31Z     stalwart-0      store.cache-hit            account=B changeId=765
elapsed=0ms
                                      [no EXISTS or FETCH for documentId=374]
2026-09-13T18:47:15Z     stalwart-0      imap.idle-stop             account=B elapsed=591100ms
2026-09-13T18:47:15Z     stalwart-0      store.cache-update         account=B changeId=766
elapsed=1ms
2026-09-13T18:47:15Z     stalwart-0      imap.raw-output            account=B "* 316 EXISTS"
2026-09-13T18:47:15Z     stalwart-0      imap.fetch                 account=B documentId=374
mailboxId=0
2026-09-13T18:47:15Z     stalwart-0      imap.noop                  account=B elapsed=3ms
2026-09-13T18:47:16Z     stalwart-0      imap.raw-output            account=B "* 316 FETCH
(UID 316 ...)"
2026-09-13T18:47:19Z     stalwart-0      imap.idle-start            account=B

Stalwart Version

v0.16.x

Installation Method

Built from source

Database Backend

FoundationDB

Blob Storage

S3-compatible

Search Engine

Internal

Directory Backend

Internal

Additional Context

Observed on Stalwart v0.16.21
The relevant source is unchanged in v0.16.22

Separate SMTP frontend and outbound-MTA worker processes
Redis coordinator; no subscriber or publisher errors during the observation window
Envoy proxy in front of stalwart

seems Related to the FoundationDB queue-refresh report

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Your trace lines up with what the IMAP IDLE code actually does. In handle_idle (crates/imap/src/op/idle.rs), an EmailPush notification just sets has_email_changes/has_mailbox_changes flags and calls write_mailbox_changes, which in turn queries the FoundationDB change log for the account via store.changes(..., Query::Since(modseq)). That query is served through the same account-level cache (crates/email/src/cache/mod.rs) that emits the store.cache-hit / store.cache-update events you’re seeing.

The cache-hit at 18:42:31 on stalwart-0 means that when it queried FDB for changes since its last known changeId=765, it got back an empty change set even though the write for changeId=766 had already been committed and broadcast via the cluster/Redis push. That is a read-side visibility race between the near-instant cross-process push notification and the FDB change-log read used to refresh the cache; the cache only picked up the new state on the next trigger (here, the IDLE timeout forcing a resync), which is why you saw a ~4m44s delay matching the IDLE cycle rather than the actual push latency.

This is consistent with the mechanism in the FoundationDB queue-scan report you linked: a cluster-wide push notification can outrun the change log becoming visible to a changes() read on a different frontend process, and there’s no retry/backoff on a stale cache-hit result to catch up before the next scheduled resync. There’s nothing in the current docs describing a workaround for this race, so I can’t point you to a documented fix or tuning knob for it.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

Fixed in v0.16.23 (yet to be released).