FoundationDB queue scan can miss cross-process enqueue until five-minute fallback

Issues
1

Issue Description

We are observing delayed local delivery in a clustered Stalwart deployment using
FoundationDB.

Environment

  • Observed on Stalwart v0.16.19
  • The relevant source is unchanged in v0.16.21
  • FoundationDB data store
  • Separate SMTP frontend and outbound-MTA worker processes
  • Redis coordinator; no subscriber or publisher errors during the observation window

Observed behavior

Across 335 inbound messages correlated from SMTP acceptance to local mailbox ingest:

  • median latency: 28.6 seconds

  • p95: 300.4 seconds

  • 54 messages were delivered between 299 and 301 seconds

  • 205 of 206 delayed messages were processed either at the five-minute fallback or
    immediately after another enqueue

  • once the worker found a message, local ingest completed within approximately 0.2 seconds

  • no rate-limit, quota, concurrency, reschedule, or recipient-rejection events preceded the
    delayed attempts

Example:

SMTP accepted: 16:19:18.765
Local delivery attempt: 16:24:19.235
Mailbox ingest completed: 16:24:19.392
Total delay: 300.627 seconds

Expected Behavior

A queue item committed by one process should become visible when QueueRefresh wakes the outbound worker, without waiting for another enqueue or the periodic queue scan.

Actual Behavior

After committing a queue item, Stalwart broadcasts QueueRefresh.
The worker’s FoundationDB transaction may reuse a process-local read version for up to four seconds:

stalwart/crates/store/src/backend/foundationdb/mod.rs at v0.16.21 · stalwartlabs/stalwart · GitHub

The committing process advances only its own read-version floor.
Consequently, the separate worker can scan at a pre-commit read version and miss the item that triggered the refresh. It then discovers it after another enqueue or the 5 * 60 second fallback:

stalwart/crates/smtp/src/queue/spool.rs at v0.16.21 · stalwartlabs/stalwart · GitHub

A potential fix would be to obtain a fresh FoundationDB read version for externally triggered
queue scans, or propagate the producer’s commit-version floor with QueueRefresh.

Reproduction Steps

  1. Run four Stalwart processes against the same FoundationDB data store and Redis
    coordinator:

    • Three frontend replicas accepting SMTP and IMAP traffic, with outboundMta disabled.
    • One separate outbound worker with outboundMta enabled.

  2. Configure a local hosted domain and mailbox.

  3. Allow the outbound worker to become idle and perform an empty queue scan, populating its cached FoundationDB read version.

  4. Submit mail through one frontend to the local mailbox. To increase the reproduction rate, submit messages in pairs a few seconds apart, then stop traffic.

  5. Record the frontend’s queue.message-queued timestamp and correlate its queue ID with the
    outbound worker’s:

    • delivery.attempt-start
    • message-ingest.ham or message-ingest.spam
    • delivery.completed

  6. Observe that some messages are not processed when their QueueRefresh is received. They become visible either immediately after a later enqueue or at the periodic queue scan approximately 300 seconds later.

Relevant Log Output

Case 1 — five-minute fallback

queue.message-queued 2026-09-07T16:19:18.765Z
delivery.attempt-start 2026-09-07T16:24:19.235Z queueName=local
message-ingest 2026-09-07T16:24:19.392Z
delivery.completed 2026-09-07T16:24:19.392Z

Acceptance-to-ingest: 300.627 seconds

Case 2 — a later enqueue wakes the worker

original message queued 2026-09-07T17:24:26.294Z
another message queued 2026-09-07T17:28:24.660Z
original delivery attempt 2026-09-07T17:28:24.843Z queueName=local
message-ingest 2026-09-07T17:28:24.941Z
delivery.completed 2026-09-07T17:28:25.021Z

The original message became visible 0.184 seconds after the later enqueue.
Total acceptance-to-ingest: 238.646 seconds

For all 206 messages delayed by more than one second:

  • no rate-limit events
  • no quota or concurrency-limit events
  • no reschedule events
  • no recipient rejection
  • no Redis subscriber or publisher errors

Once a message became visible to the outbound worker, local ingest completed
within approximately 0.06–0.16 seconds.

Stalwart Version

v0.16.x

Installation Method

Built from source

Database Backend

FoundationDB

Blob Storage

S3-compatible

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

2

Thanks for the detailed report and the code-level analysis. What you’ve described in crates/store/src/backend/foundationdb/mod.rs is accurate: the FDB backend caches a read version per-process and only refreshes it after REFRESH_READ_VERSION_AFTER (1s) has passed since it was last obtained, up to MAX_READ_VERSION_AGE (4s), and raise_floor/store_max only ratchet the version known to that process’s own ReadVersion state. The QueueRefresh broadcast (crates/services/src/broadcast/subscriber.rs) simply wakes the local queue manager’s event loop (QueueEvent::Refresh) on processes with outbound_mta enabled; it doesn’t carry or propagate the committing process’s actual FDB commit version, and the woken worker’s subsequent queue scan (crates/smtp/src/queue/spool.rs) can still read from its own cached, potentially stale, read version rather than one guaranteed to include the just-committed write. So a wakeup can legitimately race ahead of visibility until either the cache naturally expires/refreshes or the 5-minute (QUEUE_REFRESH) fallback scan runs, which matches the correlated timings you observed.

This is a legitimate finding rather than a documented/expected behavior, so I can’t point you to docs confirming or denying it as intended design, nor confirm any planned fix timeline. A staff or engineering team member will need to follow up on whether the read-version handling should be bypassed or a fresh version forced for externally-triggered queue scans.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

Fixed in v0.16.22 (yet to be released).