Message delivered again when the save after a successful delivery fails

Issues
1

Issue Description

Stalwart 1.0.3 (8b32686), 3 nodes, PostgreSQL + Patroni through a local HAProxy, Valkey, S3 blobs.

The remote answers 250 and Stalwart logs delivery.delivered. The write that should remove the message from the queue fails on a database error. It’s logged and not retried. The lock is released and the message is still due, so a later queue scan sends it again. The node is alive throughout. The remote gets the message twice.

Expected Behavior

A delivered message isn’t sent again when the final queue save fails: the save retries, keeping the lock, for long enough to cover a database failover.

Actual Behavior

Result, 2 runs, recipients counted at the receiver, each run the same:

Messages Delivered twice
Control: same hold, about 40 s, database up 3 per run 0
Database refused when the 250 arrives 5 per run 5 per run
Same, 1 message with 2 recipients 2 recipients per run 2 per run

Resends came 80 to 140 s after the 250, from the same node. Nothing was lost.

Reproduction Steps

  1. Run an SMTP receiver that accepts the message but holds its reply after the final dot.
  2. Submit a message and wait until the receiver has it.
  3. Make the database unreachable from the node. We refused 127.0.0.1:5433 with nft and closed open sockets with ss -K. PostgreSQL itself kept running.
  4. Have the receiver answer 250.
  5. Restore database access (we used 75 s).

Relevant Log Output

Log, one message (shortened):

02:18:37 queue.authenticated-message-queued queueId=338826606726351360
02:19:14 delivery.delivered code=250 elapsed=36861ms
02:19:14 store.postgresql-error reason="error connecting to server: Connection refused (os error 111)"
         details="Failed to write to update queue." causedBy=crates/smtp/src/queue/spool.rs:1005
02:20:43 delivery.delivered code=250 elapsed=476ms

Stalwart Version

v1.x

Installation Method

Binary (Linux)

Database Backend

PostgreSQL

Blob Storage

S3-compatible

Search Engine

PostgreSQL

Directory Backend

Internal

Additional Context

Source as I read it:

  • crates/smtp/src/outbound/delivery.rs:1664-1683: save_changes and remove return a bool, and it’s ignored. remove still returns QueueEventStatus::Completed.
  • crates/smtp/src/queue/spool.rs:1001-1010: on error, log and return false. One attempt.
  • delivery.rs:109: the lock is released after the attempt either way.

Same fault without any block, PostgreSQL primary powered off under outbound load: 470 messages, 3 delivered twice. Delivered +2 to +3 s, spool.rs:1005 “connection closed” +13 s, delivered again +34 s, same node.

Any write error at that point should do the same: I also saw it with cannot execute DELETE in a read-only transaction (25006) at spool.rs:1005, when a pooled connection was on a standby.

I’m not asking about a node that dies right after the 250; I know that can’t be closed completely. Here the node stays up and knows the save failed.

Related, cause looks different: Redis lock handoff reuses a FoundationDB read version and leaves negative queue-quota counters also reports duplicates on v1.0.3, from a stale FoundationDB read after the delete committed. Here the delete fails, on PostgreSQL.

Question: could that final save retry for long enough to cover a database failover, keeping the lock while it does? Is there a setting I’ve missed?

Tested and written with AI help.

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

I found no setting that retries this save or keeps the lock while it retries. Your reading of the code matches what I see in crates/smtp/src/queue/spool.rs, near line 1005.

The final write runs once through write_batch. If the error is an assertion failure, the code clears the events and tries once more. That is a different case from a connection error. Any other error is logged as “Failed to write to update queue.” and the function returns false. There is no backoff loop around it. I couldn’t open delivery.rs through the search tool, so I haven’t confirmed your notes on lines 1664-1683 or on the lock release at line 109. That includes the claim that the false result is ignored and remove still reports Completed. If those lines are as you describe, the sequence you saw follows: the message stays due and the lock is released, so the next queue scan sends it again.

I found nothing in the docs or in spool.rs that tunes this. I also found nothing that would make a delivered message be tracked separately from the queue entry. The docs don’t cover a way to avoid the duplicate. A retry that keeps the lock for the length of a failover would need a code change.

I’m leaving this for a human to confirm and follow up. Your reproduction and the related topic (2213) will help with that.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

This has been fixed, the fix will be included in the next weekly release.