Issue Description
Stalwart 0.16.26 (official stalwartlabs/stalwart Docker image, Linux arm64, RocksDB) can reject incoming mail with “452 4.3.1 Mail system full, try again later.” even when the queue is empty and configured limits are 100,000 messages / 10 GiB.
A single message delivered through two virtual queues leaves negative per-recipient-domain quota counters after both deliveries succeed. Subsequent small messages are rejected, while larger messages can still be accepted.
We encountered this after upgrading from 0.16.20 to 0.16.26 with an existing empty-condition quota rule. I reproduced it on fresh test databases with synthetic email and a loopback SMTP sink, without SMTP2GO or any external mail service. The backend selections below describe this isolated reproduction. Version controls show that the accounting defect already exists in 0.16.20 when the quota condition is explicitly true; the 0.16.25 empty-condition enforcement change appears to have exposed it.
Expected Behavior
After every recipient has been delivered and QueuedMessage/query reports total: 0, the corresponding queue-quota count and byte counters should return to zero. New mail below the configured limits should receive SMTP 250 regardless of message size. A quota reservation should be released exactly once when recipients span multiple virtual queues.
Actual Behavior
After a 2,175-byte message is delivered to recipients in two virtual queues, both recipient-domain counters end at count -1 and bytes -2,214, despite the final queue total being 0. The counted size includes the server-generated Date header.
A subsequent 175-byte probe to either domain receives SMTP 452 (Mail system full). A 4,175-byte probe receives SMTP 250. Routing both recipients through one virtual queue instead leaves all counters at zero and accepts the probes.
Reproduction Steps
- Start the official 0.16.26 Docker image with a fresh RocksDB database. Use disposable, isolated test containers (–network none, no published ports) and synthetic email.
- Configure an enabled MtaQueueQuota keyed on rcptDomain, with messages: 100000, size: 10737418240, and the default empty match condition. Read it back by ID to verify that it is enabled with those keys, limits and condition.
- Create virtual queues q1 and q2 and corresponding delivery schedules. Route a.example recipients to q1 and b.example recipients to q2.
- Route both domains to a loopback SMTP sink (127.0.0.1:2526 inside the test network namespace); delay acceptance for b.example by 3 seconds. Configure the isolated test to avoid external DNS/authentication or spam lookups.
- Submit single-recipient baseline messages to [email protected] and [email protected] separately. Confirm SMTP 250, successful sink deliveries, and a drained queue.
- Submit one 2,175-byte message to BOTH [email protected] and [email protected] in the same SMTP transaction. Both deliveries succeed. Wait for x:QueuedMessage/query to report total: 0.
- Submit a 175-byte probe to each domain: both receive SMTP 452. Submit a 4,175-byte probe: it receives SMTP 250. Drain all accepted mail.
- Stop only the disposable test instance and inspect its queue-quota counters via the database console: both per-domain counts are -1 and byte counters are -2,214.
- Repeat with both recipients assigned to q1: probes receive 250 and counters return to 0. Repeat on 0.16.20 with default empty condition (no reservations, probes accepted), then explicit true condition (negative counters and 452 reproduced).
Relevant Log Output
Synthetic reproduction harness output (0.16.26, two queues, default empty condition):
SMTP_RESULT {"label": "mixed", "code": 250, "response": "2.0.0 Message queued with id 4a8d68cacc00400.", "messageBytes": 2175}
SMTP_RESULT {"label": "probe-a-small", "code": 452, "response": "4.3.1 Mail system full, try again later.", "messageBytes": 175}
SMTP_RESULT {"label": "probe-b-small", "code": 452, "response": "4.3.1 Mail system full, try again later.", "messageBytes": 175}
SMTP_RESULT {"label": "probe-a-large", "code": 250, "response": "2.0.0 Message queued with id 4a8d68e65200a00.", "messageBytes": 4175}
The final QueuedMessage/query total was 0. Decoded database-console counters after stopping the disposable instance: count -1 and bytes -2214 for each of a.example and b.example. These are test results, not production message logs.
Stalwart Version
v0.16.x
Installation Method
Docker
Database Backend
RocksDB
Blob Storage
RocksDB
Search Engine
Internal
Directory Backend
Internal
Additional Context
Version/configuration controls (fresh database for each case; quota rule read back and checked before sending):
| Version | Virtual queues | Match condition | Small probe | Final count / bytes per domain |
|---|---|---|---|---|
| 0.16.26 | Two | Empty | 452 | -1 / -2214 |
| 0.16.26 | One | Empty | 250 | 0 / 0 |
| 0.16.20 | Two | Empty | 250 | No reservations |
| 0.16.20 | Two | true | 452 | -1 / -2214 |
Suspected implementation defect (source analysis, rather than a confirmed upstream diagnosis):
In MessageWrapper::save_changes, release_quota removes released reservations from the worker metadata and appends counter decrements. The is_multi_queue merge updates recipients on the persisted message but retains its old metadata. Subsequent processing/final removal can therefore release the same reservation again.
save_changes source (v0.16.26, spool.rs lines 674–790)
check_quota casts signed counters to u64. A negative value becomes huge; wrapping addition appears to explain the size-dependent acceptance.
check_quota source (v0.16.26, quota.rs lines 121–177)
The 0.16.25 empty-condition enforcement fix appears to have exposed this older accounting defect:
Commit a674be967aa7ceec2036e3bab4421166d6589745 in stalwartlabs/stalwart (included in 0.16.25).
The existing topic about empty-condition quotas never being enforced concerns that separate activation issue:
Stalwart Support topic #1852: “Default global queue quota is enabled but never enforced because its match expression is empty”.
Workaround: disabling the affected queue-quota rule and reloading settings restored receiving. Is there a supported way to repair already-negative quota counters once the reservation-release logic is fixed?
A self-contained Python reproducer, full synthetic SMTP results, raw/decoded quota-counter dumps and exact image digests are available. All tests used official images without source modifications, isolated Docker containers with a loopback-only sink, fresh RocksDB databases, and no external relay. No production messages or credentials are included in this report.
I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.
on
I have searched this support forum (open and closed topics) and confirm this is not a duplicate.
on
I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.
on