Support.stalw.art

yoda2026 · September 21, 2026, 7:23pm

Issue Description

Version: 0.16.23 (Docker, ghcr.io/stalwartlabs/stalwart:v0.16.23), upgraded from 0.16.22 on 2026-09-21. Resolver: system. The container’s resolv.conf has a single nameserver (our provider’s validating resolver).

What happens
Outbound delivery to one recipient domain under .org that is DNSSEC-signed and whose MX hosts are Google Workspace (aspmx.l.google.com, alt1/alt2.aspmx.l.google.com) fails on every attempt:

INFO IP address lookup failed (delivery.ip-lookup-failed) ... hostname = "aspmx.l.google.com",
details = "Temporary Failure for aspmx.l.google.com: Connection failed: lookup error:
DNS resolution error: DNS error: DNSSEC validation failed", elapsed = 880ms

The message stays queued and bounces at expiry. It started 2026-09-17/18, the same days as the .org DNSKEY rollover described in Stalwart GitHub issue 3316 and hickory-dns issue 3963. In the same period the same Google MX hosts delivered normally (83 deliveries) for recipient domains that are NOT signed. Every DNSSEC failure since 09-17 is this one recipient domain.

Why the 0.16.23 fix does not seem to cover it
0.16.23 makes the DNSSEC resolver query one nameserver at a time, to avoid the parallel truncated-response race. Here only one nameserver is configured, so there is no second response to race with. After upgrading to 0.16.23 and force-retrying the queued message, it fails with exactly the same error.

What I verified (from the host and from inside the container’s network namespace)

dig +dnssec +bufsize=1232 DNSKEY org via the configured resolver → tc set (truncated), as expected.
Same query over TCP → NOERROR, ad set, 1319 bytes. TCP/53 to the resolver is reachable from inside the Stalwart container.
The recipient domain’s MX RRset validates (ad from the resolver; delv says “fully validated”).
DS google.com returns a correct opt-out NSEC3 denial with RRSIGs (so aspmx.l.google.com should be provably insecure).

Expected
The A/AAAA lookup of an MX host in an unsigned zone should resolve as Insecure and delivery should proceed (no DANE, since the MX zone is unsigned).

Notes

Not every signed .org domain is affected: another DNSSEC-signed .org domain delivered fine in the same window, so it may depend on how the .org DNSKEY fetch (truncated over UDP) is cached or ordered.
Looks like the same hickory truncation / TCP-retry path as Stalwart GitHub issue 3316, but reachable with a single nameserver.

Reproduce
A DNSSEC-signed .org recipient domain with MX pointing to Google Workspace, sent from 0.16.23 with one upstream validating resolver.

I can share the recipient domain and a full trace privately if useful.

Expected Behavior

The A/AAAA lookup of an MX host that lives in an unsigned zone (aspmx.l.google.com, under google.com) should resolve as DNSSEC “Insecure”, and delivery to the DNSSEC-signed .org recipient domain should proceed normally, as it does for unsigned recipient domains using the same MX hosts. No DANE applies, since the MX zone is unsigned.

Actual Behavior

Every delivery attempt to that recipient fails before connecting: “delivery.ip-lookup-failed … hostname = aspmx.l.google.com … DNS error: DNSSEC validation failed” (also alt1/alt2.aspmx.l.google.com). The message stays in the queue, DSN delay warnings go to the sender, and it bounces at expiry. The same Google MX hosts deliver fine in the same period for recipient domains that are not DNSSEC-signed. Upgrading 0.16.22 → 0.16.23 and force-retrying did not change it. Started 2026-09-17/18 (.org DNSKEY rollover).

Reproduction Steps

Run Stalwart 0.16.23 in Docker with the default (system) resolver, where the container’s resolv.conf lists ONE validating upstream resolver.
Send a message to a recipient whose domain is under .org, is DNSSEC-signed (DS at .org), and has its MX on Google Workspace (aspmx.l.google.com, alt1/alt2.aspmx.l.google.com).
Watch the delivery log: every attempt ends in delivery.ip-lookup-failed “DNSSEC validation failed” for the Google MX hosts, and the message stays queued.
For comparison, send to an UNSIGNED domain that uses the same Google MX hosts: it delivers normally.

Relevant Log Output

2026-09-21T19:08:10Z INFO Delivery attempt started (delivery.attempt-start) queueId = 330564860025141760, queueName = "remote", from = "[email protected]", to = ["[email protected]"], size = 1975, total = 1
2026-09-21T19:08:10Z INFO TLS-RPT record not found (tls-rpt.record-not-found) ... domain = "signed-recipient.org", elapsed = 31ms
2026-09-21T19:08:10Z INFO MTA-STS policy not found (mta-sts.policy-not-found) ... domain = "signed-recipient.org", strict = false, elapsed = 106ms
2026-09-21T19:08:11Z INFO IP address lookup failed (delivery.ip-lookup-failed) ... domain = "signed-recipient.org", hostname = "aspmx.l.google.com", details = "Temporary Failure for aspmx.l.google.com: Connection failed: lookup error: DNS resolution error: DNS error: DNSSEC validation failed", elapsed = 880ms
2026-09-21T19:08:11Z INFO IP address lookup failed (delivery.ip-lookup-failed) ... domain = "signed-recipient.org", hostname = "alt1.aspmx.l.google.com", details = "Temporary Failure for alt1.aspmx.l.google.com: Connection failed: lookup error: DNS resolution error: DNS error: DNSSEC validation failed", elapsed = 139ms

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

helpbot · September 21, 2026, 7:25pm

Based on the 0.16.23 source, the fix that shipped only changes num_concurrent_reqs on the DNSSEC resolver to 1, which prevents the parallel nameserver race where two servers both return a truncated response and hickory’s retry gets cancelled (server/crates/common/src/config/smtp/resolver.rs, and the changelog entry “DNS: The DNSSEC resolver queries a single nameserver at a time…”). That change has no effect on a single-nameserver setup like yours, because there was never a second response to race with — it only fixes the specific multi-nameserver race in issue 3316.

Your case looks like a distinct manifestation of the same underlying hickory truncation-handling problem: tcpOnError (the DnsResolver option that retries a query over TCP) only fires when the UDP query itself returns an error. A truncated response with the tc bit set but NOERROR (exactly what you observed with dig) is not treated as an error by the resolver, so no TCP retry is triggered for that DNSKEY fetch — leaving the resolver with an incomplete RRset for .org’s DNSKEY set, which it then reports as bogus/failed validation instead of falling through to “Insecure” for the (unsigned) Google MX zone.

This is not something the current docs address as a workaround, and I can’t confirm a fix for it in the fetched source, so:

This isn’t covered in the current docs. A community member or staff member will follow up.

Please do share the recipient domain and trace privately as you offered — that will help pin down whether it’s exactly this truncation/TCP-retry gap in the current DNSSEC resolver path.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

trinitrion · September 22, 2026, 12:30pm

+1, same root cause, different trigger path, reproducible with zero DNSSEC-signed involvement

Confirmed the same behavior on Stalwart v0.16.23 (Docker), resolver preset: Quad9. Deterministic delivery.ip-lookup-failed / DNSSEC validation failed on every attempt to aspmx.l.google.com and alt2.aspmx.l.google.com, for a recipient domain that has no DNSSEC signing at all and no MTA-STS policy.

dnsviz confirms the actual issue:

google.com to l.google.com: An SOA RR with owner name (l.google.com) not
matching the zone name (google.com) was returned with the NODATA response.
See RFC 1034, Sec. 4.3.4, RFC 2308, Sec. 2.2.

l.google.com (the MX zone) is legitimately unsigned. External resolvers (Quad9 directly via dig/delv) return a clean NOERROR / unsigned answer, the delegation-boundary NODATA response just carries an SOA owner name that doesn’t match RFC 2308 Sec. 2.2. Stalwart’s internal resolver treats this non-compliant-but-harmless response as bogus instead of Insecure.

What we ruled out:

Not a resolver cache issue, reproduced immediately after a full container restart (fresh cache), same result.
Not DANE-related, TLSA lookups against the same MX hosts correctly log dane.tlsa-record-not-dnssec-signed and fall back cleanly. The failure happens one layer earlier, at plain A/AAAA resolution.
Not deterministic per-domain, recipients whose domain has its own MTA-STS policy succeed, because Stalwart then targets the MTA-STS-preferred hostname (smtp.google.com) instead of falling back to the raw MX host (aspmx.l.google.com), and that hostname resolves cleanly. Recipients without MTA-STS hit the raw MX host and fail 100% of the time.

Expected: A/AAAA lookup of an MX host in a zone that provably has no DS record (here: l.google.com under google.com) should resolve as DNSSEC Insecure, and delivery should proceed without DANE — exactly as it does for the MTA-STS-covered path using the same MX hosts.

Happy to share full container logs / dnsviz output privately if useful for triage.

DodoLeDev · September 23, 2026, 12:43pm

Some new interesting findings when digging into the Stalwart DNS queries (in my case):

First, Stalwart fetches the mailserver for the target domain, and the public key for the DNSSEC-protected domain.

server.tld               DNSKEY +dnssec  => OK (DNSKEY, fully validated)
server.tld               MX     +dnssec  => OK (MX: aspmx.l.google.com, fully validated)

Then, it tries to retrieve MTA-STS and TLSRPT data directly from the target domain.

_mta-sts.server.tld      TXT             => NXDOMAIN (there was no MTA-STS policy)
_smtp._tls.server.tld    TXT             => NXDOMAIN (there was no TLS-RPT address)

Then the problematic part: Stalwart checks the security of the MX domain:

l.google.com             NS     +dnssec  => Unsigned answer
l.google.com             DS     +dnssec  => NXDOMAIN
_tcp.aspmx.l.google.com  NS     +dnssec  => NXDOMAIN
aspmx.l.google.com       NS     +dnssec  => NXDOMAIN

Stalwart have interrupted it’s lookup to l.google.com, while the missing piece was provided by google.com (which gives a NSEC3 entry saying that DNSSEC is not included in subdomains).

So I suppose that the program reached a situation that prevents it from continuing:

server.tld is DNSSEC-ed, so it assumed aspmx.l.google.com (and l.google.com) was too but it wasn’t
When looking in each upper domain, the fact that l.google.com returned NXDOMAIN when asking for its DS turned into an error
l.google.com does not mention google.com anywhere in its responses

Please note that I am not an expert in DNS and mail behaviour. All these interaction make it really difficult to find the source of the error…

ideer2 · September 23, 2026, 1:10pm

So.. we still cant upgrade from 16.20 if we need reliable email transport?

trinitrion · September 23, 2026, 1:26pm

Wait, hold on, if I’m reading this right, this would mean: any Google Workspace customer whose own domain doesn’t have an MTA-STS policy configured (which is probably the vast majority of small-to-mid-size Workspace customers, MTA-STS on the recipient side isn’t exactly a default anyone sets up) cannot receive mail from Stalwart at all, full stop?

To clarify scope on our end: so far this has only hit automated DMARC/TLS-RPT report deliveries for us, not any real transactional mail, nothing else has gotten stuck in the queue. So either we’ve just been lucky with which Workspace domains our actual correspondents use (i.e. they do have MTA-STS), or there’s a difference in severity here I don’t understand yet. Curious whether others are seeing real mail affected too, or if it’s mostly report traffic hitting this path first because reports go out more frequently/automatically

DodoLeDev · September 24, 2026, 5:14pm

Wait, hold on, if I’m reading this right, this would mean: any Google Workspace customer whose own domain doesn’t have an MTA-STS policy configured cannot receive mail from Stalwart at all

I have worked around this problem (kind of) by creating a rule that disables DANE for google workspace MX domains.

Seems to work, but not in every environments (it works in production but not on a throw-able container, don’t ask me why ‍:male_sign:)

stalwart · September 24, 2026, 6:11pm

Please track these two hickory bugs:

github.com/hickory-dns/hickory-dns

DNSSEC "exceeded max validation depth" after upgrading to 0.26.3

opened 12:33PM - 19 Sep 26 UTC

kevincox

**Describe the bug** Resolution fails with "DNS error: DNSSEC validation failed"…. An error is also logged from hickory_net::dnssec with "exceeded max validation depth". **To Reproduce** Upgrade from 0.26.1 to 0.26.3 **Expected behavior** DNS resolution continues to work. **System:** - OS: Linux - Architecture: x86_64 - Version - rustc version: ? **Version:** Crate: resolver and dependencies. Version: 0.26.3 **Additional context** I am still looking into this issue and will update. I mostly filed this so that others encountering the same issue can help investigate. Quite possibly this is a configuration issue on my end but the configuration wasn't changed as part of the update. ~~So far I am only seeing this error on `.com` domains that **don't** support DNSSEC. (Mostly Google domains)~~ but that may just be based on the distribution of domains that our service resolves.

github.com/hickory-dns/hickory-dns

DNSSEC: NSEC3 opt-out DS denial is treated as Bogus when the next closer name is not QNAME

opened 06:10PM - 24 Sep 26 UTC

mdecimus

**Describe the bug** A validating lookup of a name in an insecure zone fails wi…th `DnsError::DnssecBogus` when that zone is delegated two or more labels below the closest encloser in an NSEC3 opt-out parent zone. The parent's DS NODATA answer is a valid RFC 5155 section 8.6 proof, but hickory only looks for an opt-out NSEC3 covering H(QNAME) and never for the one covering the next closer name. A live example is `port.venice.it`, which is delegated directly from `it.`. `venice.it` is not a zone and has no NSEC3 record of its own: it is an empty non-terminal that exists only because of an insecure delegation inside an opt-out span, and RFC 5155 section 7.1 allows it to be left out. The `it.` servers answer `DS port.venice.it` with this authority section (RRSIGs omitted): ``` it. 3600 IN SOA dns.nic.it. hostmaster.nic.it. 2026092417 10800 900 604800 3600 RS1N3N7M54PDEM5EUNV9NPKH3B6CGPJC.it. 3600 IN NSEC3 1 1 0 - RS1PG43S35R5V1KF1V10FT5SP409IUBV NS SOA RRSIG DNSKEY NSEC3PARAM 6EQIAO6UF3HPKVP4A8704T8QV4NV63RL.it. 3600 IN NSEC3 1 1 0 - 6ERRV534H8JG5VFJ7B5IG4SMVMTKVGOA NS DS RRSIG ``` With an empty salt and 0 iterations the hashes are: | Name | H(name) | In the response | |---|---|---| | `it.` (closest encloser) | `RS1N3N7M54PDEM5EUNV9NPKH3B6CGPJC` | matched by the apex NSEC3 | | `venice.it.` (next closer) | `6ERNUJ3UKNAC1SIO0BB8OPAM3QSHJC7T` | covered by `6EQIAO6U...`, Opt-Out set | | `port.venice.it.` (QNAME) | `IE297EM4H4IQ0305K7H1ED4AQNHRBJUU` | neither matched nor covered | That is a closest provable encloser proof plus an opt-out NSEC3 covering the next closer name, which is exactly what section 8.6 requires when no NSEC3 matches QNAME. The user who ran into this confirmed with BIND's `delv @1.1.1.1 port.venice.it MX` that the answer validates as unsigned, and both 1.1.1.1 and 8.8.8.8 return NOERROR without AD for it. `comune.venezia.it`, an ordinary insecure delegation one label below `it.`, validates fine because its next closer name is QNAME itself. **To Reproduce** ```toml [dependencies] hickory-resolver = { version = "0.26.3", features = ["dnssec-aws-lc-rs"] } tokio = { version = "1", features = ["full"] } tracing-subscriber = { version = "0.3", features = ["env-filter"] } ``` ```rust use hickory_resolver::{ TokioResolver, config::{CLOUDFLARE, ResolverConfig, ResolverOpts}, net::runtime::TokioRuntimeProvider, }; #[tokio::main] async fn main() { tracing_subscriber::fmt() .with_env_filter("hickory_net::dnssec=debug") .without_time() .with_ansi(false) .init(); let mut opts = ResolverOpts::default(); opts.validate = true; opts.num_concurrent_reqs = 1; let resolver = TokioResolver::builder_with_config( ResolverConfig::udp_and_tcp(&CLOUDFLARE), TokioRuntimeProvider::default(), ) .with_options(opts) .build() .unwrap(); for name in ["port.venice.it.", "comune.venezia.it."] { match resolver.mx_lookup(name).await { Ok(lookup) => lookup .answers() .iter() .for_each(|r| println!("{name} {} {}", r.data, r.proof)), Err(err) => println!("{name} error: {err:?}"), } } } ``` `num_concurrent_reqs = 1` only keeps an unrelated failure out of the way: with the default, one of my runs failed earlier on the root DNSKEY fetch with "response was truncated; TCP already tried or no other transport available", which looks like #3963. Relevant output, identical over three runs: ``` DEBUG hickory_net::dnssec: nsec3 proof for port.venice.it., returning Bogus: no valid servicing wildcard proof DEBUG hickory_net::dnssec: returning Nsec error for port.venice.it. Bogus DEBUG hickory_net::dnssec: failed to verify: port.venice.it. record_type: MX: communication failure for query: port.venice.it. IN DS: DNS error: DNSSEC Negative Record Response for port.venice.it. IN DS, Bogus port.venice.it. error: Dns(DnssecBogus) DEBUG hickory_net::dnssec: nsec3 proof for comune.venezia.it., returning Secure: DS query covered by opt-out proof comune.venezia.it. 5 venis01.esvacloud.com. Insecure comune.venezia.it. 5 venis02.esvacloud.com. Insecure ``` **Expected behavior** The MX lookup for `port.venice.it` succeeds and the records carry `Proof::Insecure`. **Suggested fix** Disclaimer: Fix proposed by Claude Opus 5.5, against main at f486d9e: ```diff @@ -220,7 +220,18 @@ pub(super) fn verify_nsec3_insecure_delegation(zone: &Name, nsec3s: &[(&Name, &N return info.nsec3_data.opt_out(); } - false + let Some(parent) = records.first().map(|record| &record.zone_name) else { + return false; + }; + if records.iter().any(|record| &record.zone_name != parent) { + return false; + } + let Ok(cx) = Context::new(&query, Some(parent), &records) else { + return false; + }; + cx.closest_encloser_proof() + .next_closer + .is_some_and(|(_, record)| record.nsec3_data.opt_out()) } /// Convert pairs of record names and NSEC3 RDATA to structures containing hashed record names, zone @@ -425,6 +436,18 @@ fn validate_nodata_response( return cx.proof(Proof::Secure, "DS query covered by opt-out proof"); } + if query_type == RecordType::DS + && cx + .closest_encloser_proof() + .next_closer + .is_some_and(|(_, record)| record.nsec3_data.opt_out()) + { + return cx.proof( + Proof::Secure, + "DS query next closer covered by opt-out proof", + ); + } + let (proof, reason) = match wildcard_encloser_num_labels { // Case 4: // Name is serviced by wildcard that has a record of this type ``` `verify_nsec3_insecure_delegation` has no SOA to work from, so the patch bounds the encloser walk with the parent of the NSEC3 owner names and gives up if they disagree. The change does not accept anything RFC 5155 rejects. `closest_encloser_proof()` already refuses an ancestor-delegation NSEC3 as the closest encloser (RFC 5155 section 8.3, RFC 6840 section 4.1). In a correctly signed zone no existing name falls inside an NSEC3 span, and section 7.1 requires the empty non-terminal above a signed delegation to have its own NSEC3, so without forged signatures this proof cannot be produced for a delegation that has a DS. With the patch applied to main, `cargo test -p hickory-net --features dnssec-aws-lc-rs --lib dnssec::` passes, 31 tests including the new one. The same patch on 0.26.3 makes the program above return `Insecure` for `port.venice.it`, and `comune.venezia.it`, `gmail.com` and `example.com` resolve with the same proofs as before. I can send it as a PR together with the test. **System:** - OS: macOS 26.6.2 (also seen on Linux in Docker by the user who reported it) - Architecture: arm64 - rustc version: 1.98.1 **Version:** Crate: hickory-net, hickory-resolver Version: 0.26.3, and main at f486d9e

trinitrion · October 5, 2026, 7:58am

Is this still unresolved?

mdecimus · October 5, 2026, 11:46am

We’re waiting hickory to merge the fix and release.

stalwart · October 11, 2026, 3:49pm

Hi everyone,

This is an update on the DNSSEC and DANE delivery failures reported across several threads here. They come from bugs in hickory, the DNS library Stalwart uses to validate DNSSEC, and each one now has a fix submitted upstream:

Google Workspace MX hosts (aspmx.l.google.com and the altN hosts) failing with “exceeded max validation depth”: hickory#3974, fixed in #3976 and #4007. Both are merged but not yet in a hickory release.
Domains delegated two or more labels below an NSEC3 opt-out zone, such as port.venice.it and the comune.*.fr.it municipality domains: hickory#3985, fix in #4001.
Unsigned zones without an NS record at their apex, for example akokan.pl: hickory#4027, fix in #4028.
Microsoft 365 hosts (*.mail.protection.outlook.com) failing when the upstream resolver answers SERVFAIL for them, which Google Public DNS and many local resolvers do: hickory#4063, fix in #4064.
Truncated UDP responses never retried over TCP when several name servers are queried at the same time: hickory#3963, fix in #4062.
Valid answers rejected because the response also carries an unrelated NSEC record, plus some correctly signed wildcard answers rejected: hickory#4047, fix in #4055.
Validating queries sent without the CD bit, which makes a validating upstream resolver answer SERVFAIL: hickory#3966, fix in #4014 and #4036.

DANE delivery has been broken for some of you for weeks now, and the hickory maintainers seem to be busy at the moment, so most of these PRs are still waiting for review. We have forked hickory at stalwartlabs/hickory-dns with all of the fixes above applied, and Stalwart will build against that fork until hickory publishes a release that contains all of them. We ran the patched resolver against every domain reported in these threads, with Cloudflare, Google and Quad9 as upstream resolvers, and they all validate correctly now.

v0.16.26 ships with the patched resolver and will be out today or tomorrow. After upgrading you can remove any TLS strategy you added to turn DANE off for specific domains. If a domain still fails after the upgrade, post it in this thread together with the DNS resolver you use and we’ll look into it.

Topic		Replies	Views
Outbound delivery to all .org and .io recipients stalls with "DNSSEC validation failed" Issues no-ai	1	117	September 18, 2026
The configured DNS resolver cannot validate DNSSEC Questions	2	78	September 28, 2026
Transmission problem to outlook.com Questions	20	336	October 11, 2026
Outbound: "DNSSEC validation failed" on MX host lookup when the recipient domain is signed but its MX hosts are in an unsigned zone (Google Workspace). Still present in 0.16.24 Issues	4	120	October 12, 2026
MX lookup fails with "DNSSEC validation failed" for signed zone (mail.de, RSA/SHA-512) on 0.16.21, regardless of DANE setting Issues	13	388	October 11, 2026