Outbound delivery to all .org and .io recipients stalls with "DNSSEC validation failed"

Issues
1

Issue Description

What happened?

With the default System DNS resolver, every outbound message to a recipient whose MX
hosts sit under .org (and .io) stalls in the queue. Each attempt logs:

INFO IP address lookup failed (delivery.ip-lookup-failed) queueId = ..., domain = "<>",
hostname = "mxext1.mailbox.org", details = "Temporary Failure for mxext1.mailbox.org:
Connection failed: lookup error: DNS resolution error: DNS error: DNSSEC validation failed",
elapsed = 3ms

The recipient’s DNS is entirely healthy — the full chain validates with AD set from any
validating resolver, and the DANE TLSA records are present and signed. Only Stalwart’s
DNSSEC resolver fails, and it fails in 1–11 ms, i.e. without a network round trip.

MX, TLS-RPT and MTA-STS lookups continue to work throughout, because those go through the
non-validating resolver. Only delivery.ip-lookup-failed is affected.

The underlying defect is in hickory-dns and is reported separately:
hickory-dns/hickory-dns#3963. In short: when several nameservers are queried
concurrently and more than one returns a truncated response, hickory aborts its TCP retry
and gives up. .org’s DNSKEY RRset grew to 1447 bytes during a key rollover on
2026-09-17, past the 1232-byte UDP ceiling, which is what exposed it.

This issue is about Stalwart’s exposure to it, and two things on the Stalwart side that
made it very hard to diagnose.

Why Stalwart is exposed by default

  • DnsResolver::System calls read_system_conf(), which maps every nameserver in
    /etc/resolv.conf through NameServerConfig::udp_and_tcp. In a container this is
    routinely two entries — Podman writes both an IPv4 and an IPv6 address for
    aardvark-dns by default.
  • opts.num_concurrent_reqs is taken from the resolver settings, so those two
    nameservers are queried in parallel.
  • opts_dnssec.validate = true is set for the DANE resolver.

That combination is enough to trigger it, with nothing configured unusually.

A one-line mitigation available now, independent of hickory: force
opts_dnssec.num_concurrent_reqs = 1 for the DNSSEC resolver in
crates/common/src/config/smtp/resolver.rs. With a single in-flight query there is no
second truncated response to race, and the TCP retry proceeds normally. I verified this
distinction directly — same host, same upstreams, one nameserver resolves Proof::Secure
while two fail 8/8.

Two Stalwart-side problems this exposed

1. The error message points at the wrong party.

DNSSEC validation failed reads as “the recipient’s zone is bogus”. The actual condition
is “the local resolver could not retrieve a DNSKEY RRset because the response was
truncated”. Those need different responses from an operator, and the current wording sends
you to audit the recipient’s DNS, where everything is correct. Surfacing truncation
distinctly — or including the record type and zone that failed to validate — would have
cut the diagnosis from hours to minutes.

2. Resolver settings are bootstrap-only, but the UI suggests otherwise.

Resolvers::parse is only ever called from SmtpConfig::parse(bp: &mut Bootstrap).
Changing the DNS resolver in the admin UI and reloading settings therefore has no effect
on the running server; the old resolver stays live until the process restarts. During
diagnosis this looked exactly like “the new resolver setting doesn’t help”, when it had
simply never been applied. Either rebuilding the resolver on reload, or telling the
operator in the UI that this particular setting needs a restart, would avoid that.

Workaround

Set the resolver to Custom DNS with a single server and Protocol: TCP
(pointing at the same address the container already uses), then restart. TCP has no
truncation to race, so delivery resumes immediately. A single nameserver with
udp_and_tcp also works and keeps the UDP fast path.

Version

v0.16.21, and v0.16.22 — the relevant hickory code is byte-identical in
hickory-resolver 0.26.2 and 0.26.3, so upgrading does not help.

What database are you using?

RocksDB

What blob storage are you using?

RocksDB

Where is your directory located?

Internal

What operating system are you using?

Linux (NixOS, aarch64), running under Podman

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

2

This is hickory bug, added a workaround to limit concurrency to 1 in v0.16.23 (yet to be released).