Task-generated report message (DMARC/TLS-RPT) never gets its first delivery retry attempted (queue: default)

Issues
1

Issue Description

A task-generated report message (DMARC/TLS-RPT) sitting in the default
queue never gets its first delivery retry attempted, even long after its
scheduled nextRetry time has passed.

Stalwart version: v0.16.19 (Docker image, digest-pinned)
Platform: rootless Podman on a Debian-based VPS

Expected Behavior

Once a queued message’s nextRetry time is reached, the queue scheduler
attempts delivery on its next pass and increments Retry Count — whether
that attempt succeeds or fails.

Actual Behavior

A TLS-RPT report (returnPath noreply-tls@, recipient
[email protected], queue: default, flagged “Message is an automated
report (DMARC, TLS-RPT, etc.)”) had its nextRetry time pass and then
stayed overdue by more than an hour, with Retry Count remaining 0 the
entire time — not one delivery attempt was made, successful or failed.

Manually forcing the field via stalwart-cli update QueuedMessage <id> --field nextRetry=<now> triggered immediate, successful delivery on the
very next pass. So the delivery mechanism itself works fine once
triggered — something upstream of it (the wakeup/scheduling signal)

Reproduction Steps

I don’t have a minimal reproduction sequence — this was noticed live, not
deliberately triggered. Observed sequence, not a confirmed trigger:

  1. Server running normally, handling regular SMTP traffic.
  2. An internal task generates a TLS-RPT aggregate report for delivery.
  3. Its nextRetry time passes at some point before I noticed.
  4. Found over an hour overdue with Retry Count still 0.

Unknown what specifically would make this reproducible on demand. Will
follow up with a tighter sequence if it recurs.

Relevant Log Output

Not available. The Tracer was running at info level at the time of the
incident, and the affected message is gone now (delivered after the
manual nextRetry fix), so I can’t retroactively extract trace output for
this specific occurrence. Happy to raise the Tracer to trace and follow
up with logs if/when this recurs.

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

Additional Context

One possibly related prior report, same subsystem (queue scheduler
timing), already fixed:

Mails queue cannot be changed —
next_delivery_event() only considered retry.due while the queue index
uses min(retry.due, notify.due, expires), fixed in v0.16.9. I’m on
v0.16.19 so this specific fix should be present; the failure mode looks
like the same family of bug, possibly an edge case the fix didn’t cover.

There’s also a thread titled “BlockedIp/AllowedIp changes via settings
API don’t take effect until restart” (couldn’t link it, 2-link limit for
new users) confirming a broader pattern in the codebase: in-memory state
populated at startup that some write paths don’t trigger a reload/notify
for.

My unverified guess (haven’t checked the source): report messages, which go through a
separate staging queue before a background task assembles and inserts
them into the main queue, take an insertion path that doesn’t trigger
whatever wakeup signal the live queue scheduler needs — unlike a normal
SMTP-submitted message or an explicit API update.

Impact: only Stalwart’s own generated reports seem affected so far, not
normal SMTP-submitted mail — but I haven’t confirmed that boundary, just
haven’t seen it happen to user mail.

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Thanks for the detailed report and reproduction context. I looked at the queue scheduler and report-delivery code path in the current source, and there is a plausible mechanical explanation worth confirming with logs, though I can’t be certain it’s the exact root cause without a captured trace.

What the code does:

  • When a DMARC/TLS-RPT report is generated with deliver_now = false, its recipient’s retry.due (and notify.due/expires) are pushed out by a random offset of up to 3 hours (crates/smtp/src/reporting/send.rs), specifically to spread report delivery load.
  • When the message is queued, the spool code does send a QueueEvent::Refresh to the live queue manager (crates/smtp/src/queue/spool.rs), which forces scan_from = 0 and an immediate urgent rescan.
  • However, that rescan (next_event() in crates/smtp/src/queue/spool.rs) only iterates queue index keys in the range [scan_from, now + QUEUE_REFRESH), where QUEUE_REFRESH is 5 minutes. A report whose retry.due is, say, 2 hours in the future falls completely outside that iteration window on the initial refresh, so the manager has no way to learn about it and schedule its own wakeup for that specific time from that scan alone.
  • Subsequent periodic scans (driven by next_refresh, recomputed each pass) should, in principle, walk forward in ~5‑minute increments and eventually enter the window containing the report’s due time, self-correcting without needing an external nudge. If that “walk forward” logic stalls for some reason (e.g. scan_from getting stuck, or the full-scan reset logic not kicking in as expected), a message queued far in the future could be silently skipped indefinitely until something else (like your manual nextRetry edit, or a full scan reset via QueueEvent::Refresh from unrelated activity) forces scan_from back to 0.

This lines up with your observation that only report messages (which get the multi-hour randomized delay) are affected, while normal SMTP mail (queued with near-term retry times) isn’t.

Given the complexity here, capturing trace-level logs around a stuck report next time it happens, specifically anything logged under the Queue event category and the scheduler’s next_refresh/scan_from behavior, would be very valuable for confirming this. A staff member will follow up on this thread to dig further into the scan-window logic.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

If this occurs again please provide trace logs as the bot suggested.

4

It recurred today and claude and I captured it end-to-end at trace level. Here’s the full sequence for one DMARC aggregate report, timestamps in UTC:

1. Report generation (10:18:37Z)

2026-08-30T10:18:37Z INFO DMARC aggregate is being prepared (outgoing-report.dmarc-aggregate-report)
  reportId = 1787998717, domain = "google.com",
  rangeFrom = 2026-08-29T10:18:37Z, rangeTo = 2026-08-30T10:18:37Z

2. Strategy lookup fails (10:18:37Z) – WARN

2026-08-30T10:18:37Z WARN Strategy not found (smtp.id-not-found)
  id = "report", details = "Queue strategy not found"

3. Report is queued anyway – but nextRetry pushed ~42 min into the future (10:18:37Z)

2026-08-30T10:18:37Z INFO Queued report for delivery (queue.report-queued)
  queueId = 326747013865539072,
  from = "[email protected]",
  to = ["[email protected]"],
  size = 2461,
  nextRetry = 2026-08-30T11:01:00Z,
  nextDsn = 2026-08-31T11:01:00Z,
  expires = 2026-09-04T11:01:00Z

The report sat in the queue from 10:18 until 11:01 with zero delivery attempts – exactly the behaviour from the original report (Retry Count stuck at 0, long overdue).

4. First delivery attempt only at 11:01:00Z – on queue "default", not "report" (11:01:00Z)

2026-08-30T11:01:00Z INFO Delivery attempt started (delivery.attempt-start)
  queueId = 326747013865539072, queueName = "default",
  from = "[email protected]", to = ["[email protected]"],
  size = 2461, total = 1

2026-08-30T11:01:02Z INFO Message delivered (delivery.delivered)
  queueId = 326747013865539072, queueName = "default",
  hostname = "smtp.google.com",
  to = "[email protected]",
  code = 250, details = "OK ... gsmtp", elapsed = 810ms

2026-08-30T11:01:02Z INFO Delivery completed (delivery.completed)
  queueId = 326747013865539072, queueName = "default",
  elapsed = 2545000ms           <- this is the wall-clock age, ~42 min overdue

5. For contrast: a normal authenticated message queued at 08:39 was delivered instantly

2026-08-30T08:39:21Z INFO Queued message submission for delivery (queue.authenticated-message-queued)
  queueId = 326734523934965760, from = "[email protected]",
  to = ["[email protected]"], nextRetry = 2026-08-30T08:39:21Z

2026-08-30T08:39:22Z INFO Message delivered (delivery.delivered)
  queueId = 326734523934965760, queueName = "remote",
  hostname = "gmail-smtp-in.l.google.com", code = 250, elapsed = 347ms

Normal mail gets immediate delivery on queue "remote". The report got a 42-minute delayed first attempt on queue "default".

Configuration context

My setup has an MtaVirtualQueue named report (id: d, 5 threads) but no MtaDeliverySchedule named report – only local, remote, and dsn. This is the default state after a clean v0.16.19 install; report virtual queue is a built-in, but the matching delivery schedule is not auto-provisioned.

The WARN smtp.id-not-found id = "report" confirms Stalwart looks for a report delivery schedule at enqueue time, doesn’t find one, and falls back to "default" – which appears to use a coarser delivery cadence (next-hour-boundary?) rather than immediate retry.

This is likely the same underlying cause as topic #1349 (marked “Fixed in v0.16.18” by @mdecimus). The fix seems to have addressed the routing-expression evaluation (source == 'report'), but there’s still a gap: if no MtaDeliverySchedule named report exists (which is the default), the strategy lookup falls through with the warning above and the message lands on "default" instead of "remote".

Two separable issues:

  1. Missing default report schedule: The built-in MtaVirtualQueue named report is auto-provisioned, but no matching MtaDeliverySchedule is – so a fresh install hits smtp.id-not-found on every report. Either auto-provisioning a report schedule or falling through to remote (instead of default) would fix it.

  2. Delayed first delivery on default queue: Even after landing on "default", the 42-minute gap between nextRetry and the actual first attempt suggests the queue scheduler doesn’t pick up messages on "default" as eagerly as "remote" – possibly the scan-window / wakeup issue the helpbot described in post #2.

Stalwart version: v0.16.19 (Docker image stalwartlabs/stalwart-mail:v0.16.19)
Platform: rootless Podman on Debian-based VPS

Happy to provide full object dumps or run further instrumented tests.

5

Follow-up from our side (self-hosted single-node, v0.16.20):

We applied the workaround for the missing schedule - added an explicit report MtaDeliverySchedule (mirroring dsn: 15m/30m/1h/2h retries, max 10 attempts) via our IaC on 2026-08-30. Before the fix, report messages fell back to default’s coarser cadence exactly as described here, delaying first delivery ~42min.

This fixed the cadence and routing issue, but we’ve since observed a separate, more severe symptom that matches the “retry count stuck at 0 indefinitely” root cause this thread originally suspected - distinct from the schedule-lookup issue:

  • 2026-08-30: one queued outbound message (automated TLS-RPT report to [email protected]) sat with Retry Count: 0, over an hour past its scheduled nextRetry. DNS/network ruled out. Action/ResumeMtaQueue via CLI had zero effect - retry count and next-retry time were unchanged, ruling out a simple paused-queue state.
  • Recurrence 2026-08-31, 01:00-01:45 UTC: our watchdog (polls queue age every 15min) alerted 4x in lockstep (age growing 4170s → 5068s → 5966s → 6867s, no retry attempted in between - same signature as before). Self-resolved at 01:50 UTC with Delivery completed ... elapsed = 7185000ms (~2h queue residency). Same message type/pattern: [email protected] → [email protected], queueName = "default" at the time.
Log snippet: 2026-08-31 delayed delivery on "default" queue (~2h residency) 
2026-08-31T01:50:18Z INFO Delivery attempt started (delivery.attempt-start) queueId = 326849179942915072, queueName = "default", from = "[email protected]", to = ["[email protected]"]
2026-08-31T01:50:18Z INFO Fetched MTA-STS policy (mta-sts.policy-fetch) queueId = 326849179942915072, queueName = "default", domain = "google.com", strict = true
2026-08-31T01:50:18Z INFO Connecting to remote server (delivery.connect) hostname = "smtp.google.com", remoteIp = 142.251.127.26, remotePort = 25
2026-08-31T01:50:18Z INFO SMTP STARTTLS command (delivery.start-tls) version = "TLSv1_3", details = "TLS13_AES_256_GCM_SHA384"
2026-08-31T01:50:19Z INFO Message delivered (delivery.delivered) hostname = "smtp.google.com", to = "[email protected]", code = 250, details = "OK ... gsmtp"
2026-08-31T01:50:19Z INFO Delivery completed (delivery.completed) queueId = 326849179942915072, queueName = "default", elapsed = 7185000ms
  • For comparison, 2026-09-01, 16:43-17:12 UTC: a normal run, now correctly landing in the report queue (schedule fix applied): queued at 16:43:57, first scheduled retry at 17:12:48 (29min later, matching the report schedule’s first tier), delivered successfully on that first attempt (250 OK, DSN success). No stuck symptoms this time.
Log snippet: 2026-09-01 normal run on "report" queue (schedule fix active) 
2026-09-01T16:43:57Z INFO TLS aggregate report is being prepared (outgoing-report.tls-aggregate) reportId = 1788194637, domain = "gmail.com"
2026-09-01T16:43:57Z INFO Queued report for delivery (queue.report-queued) queueId = 327157887572909568, from = "[email protected]", to = ["[email protected]"], nextRetry = 2026-09-01T17:12:48Z
2026-09-01T17:12:48Z INFO Delivery attempt started (delivery.attempt-start) queueId = 327157887572909568, queueName = "report", from = "[email protected]", to = ["[email protected]"]
2026-09-01T17:12:48Z INFO Connecting to remote server (delivery.connect) hostname = "smtp.google.com", remoteIp = 142.251.127.27, remotePort = 25
2026-09-01T17:12:48Z INFO SMTP STARTTLS command (delivery.start-tls) version = "TLSv1_3", details = "TLS13_AES_256_GCM_SHA384"
2026-09-01T17:12:50Z INFO Message delivered (delivery.delivered) hostname = "smtp.google.com", to = "[email protected]", code = 250, details = "OK ... gsmtp"
2026-09-01T17:12:50Z INFO Delivery completed (delivery.completed) queueId = 327157887572909568, queueName = "report", elapsed = 1733000ms

So the schedule fix resolves the “delayed first attempt due to wrong cadence” issue, but doesn’t touch whatever causes the scheduler to occasionally skip a due message entirely on default (retry count never increments past 0, ResumeMtaQueue doesn’t unstick it). Every instance we’ve observed so far has been an automated TLS-RPT/DMARC report, never user-addressed mail - but we treat that as coincidence of low volume, not evidence it’s report-specific.

Happy to provide trace-level logs from the next recurrence if that helps narrow it down - it seems to recur roughly every 1-2 days on our instance.

6

Nothing was overdue in either of those traces. In both, the first delivery attempt fired at exactly the nextRetry printed in the queue.report-queued line, 11:01:00 in one and 17:12:48 in the other. What you are measuring is queue age, not lateness, and the retry count is 0 until the first attempt by definition, so a report sitting there is not evidence of a skip.

Aggregate DMARC and TLS-RPT reports are queued with a random delay of up to three hours, deliberately, so that every report on the server does not leave at the same instant. Failure reports go out immediately; aggregates do not. Your 42 minutes, two hours and 28 minutes all fall inside that window, including the 29 minute gap on the 1st, which was the same jitter and not the first tier of the report schedule. Schedule retry intervals only come into play after an attempt has failed.

ResumeMtaQueue only unpauses a paused queue; it never reschedules a message, so seeing no effect on a queue that was never paused is correct.

That randomisation is documented nowhere, I will get it onto the report pages. Separately, a report delivery schedule is seeded on a fresh install alongside the report virtual queue, and you have the queue but not the schedule, so something removed it after seeding. Since you mentioned schedules are managed by IaC, that is the first place I would look.

7

@stalwart one problem with the default.rs of seeding the database is that when new defaults are added, they are not automatically added when upgrading so config starts to drift from how the official version would look.

That said, the remote Schedule has been in default.rs since I installed it, but in my case too I don’t have it… I added it manually based on the values from:

After applying my config via my infrastructure-as-code it is still there, so I really don’t know why it was not present.

8

I found the real issue, there was an invalid property value in the report delivery schedule default and it was not being crated on fresh install. This is now fixed.

@kevinvalk When new features are added, Stalwart will automatically use the safe default for that new setting, even if it does not exist in your old schema.

9

Thanks for digging into this and fixing the property in default.rs! We’ll test it out as soon as the next release is tagged and report back.

10

Note that this last fix applies to new deployments only where the registry is empty. You will have to manually create the report delivery schedule if it is missing in your server.