DNSSEC validation failed

Issue Description

when sending an email to some recipient the DNSSEC validation fails.
Try to run trace to info at mollie dot com or hallo at junda dot nl (or any other google-hosted domains)
aspmx.l.google.com
alt1.aspmx.l.google.com
and all the others fail DNSSEC

Using 1.1.1.1 or 8.8.8.8 doesn’t change.

Expected Behavior

DNSSEC pass, trace succeeds, emails sent without issues.

Actual Behavior

Trace fails.
Regular emails too.

Reproduction Steps

configure stalwart to use 1.1.1.1 DNS
try a delivery test to info at mollie dot com

Relevant Log Output

Full DNS decode (tshark) available

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

This looks like a known resolver bug that was already fixed. Google’s DNS responses for those domains are large enough to get truncated, which should trigger a TCP retry for DNSSEC validation, but a race in the DNSSEC resolver could cancel that retry when two nameservers both returned a truncated UDP response in parallel, causing DNSSEC validation to fail.

This was fixed in 0.16.23: “DNS: The DNSSEC resolver queries a single nameserver at a time, working around a hickory-resolver race that cancels the TCP retry when two nameservers return a truncated response in parallel.” If you’re on an older 0.16.x build, upgrading to 0.16.23 or later should resolve it.

If you’re already on 0.16.23+ and still see this, check your DnsResolver settings (Settings > Network > DNS > DNS Resolver): enableEdns must be true and tcpOnError should be true as well, since DNSSEC depends on larger EDNS0 responses and TCP fallback.

https://docs.stalw.art/docs/server/dns/resolver

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

I am on stalwartlabs/stalwart:latest (release 0.16.25) and the problem is still there (even with the settings you suggested)

We are waiting for a hickory-resolver fix. Search the forum for details.

Hi.
Thank you for the reply.

I saw that some fixes have been applied but apparently my issue is different than the ones that have been fixed because the problem is still there.

Momentarily I rolled back to v0.16.20 and the problems are gone.

Please let me know if you need support to reproduce my issue on your side.

Thank you.

cant the hickory crate just be rolled back to the version in 0.16.20 till the problem is fixed?

I can imagine that the guys at stalwart already explored this option…It shouldn’t have entered into a published product at all, if there were proper regression tests..also…in the .21 the GitHub changelog for the release mentions nothing about this “package update”.

We can’t include regression tests for the hundreds of libraries that Stalwart depends on. Also this bug is triggered by certain hosts only.

I didn’t mean from your side, I meant from the side of the package you are using.
But knowing that hundreds of poorly tested libraries run on my system doesn’t really make me feel safe.

It was better not to know.

This is an update on the DNSSEC and DANE delivery failures reported across several threads here. They come from bugs in hickory, the DNS library Stalwart uses to validate DNSSEC, and each one now has a fix submitted upstream:

  • Google Workspace MX hosts (aspmx.l.google.com and the altN hosts) failing with “exceeded max validation depth”: hickory#3974, fixed in #3976 and #4007. Both are merged but not yet in a hickory release.
  • Domains delegated two or more labels below an NSEC3 opt-out zone, such as port.venice.it and the comune.*.fr.it municipality domains: hickory#3985, fix in #4001.
  • Unsigned zones without an NS record at their apex, for example akokan.pl: hickory#4027, fix in #4028.
  • Microsoft 365 hosts (*.mail.protection.outlook.com) failing when the upstream resolver answers SERVFAIL for them, which Google Public DNS and many local resolvers do: hickory#4063, fix in #4064.
  • Truncated UDP responses never retried over TCP when several name servers are queried at the same time: hickory#3963, fix in #4062.
  • Valid answers rejected because the response also carries an unrelated NSEC record, plus some correctly signed wildcard answers rejected: hickory#4047, fix in #4055.
  • Validating queries sent without the CD bit, which makes a validating upstream resolver answer SERVFAIL: hickory#3966, fix in #4014 and #4036.

DANE delivery has been broken for some of you for weeks now, and the hickory maintainers seem to be busy at the moment, so most of these PRs are still waiting for review. We have forked hickory at stalwartlabs/hickory-dns with all of the fixes above applied, and Stalwart will build against that fork until hickory publishes a release that contains all of them. We ran the patched resolver against every domain reported in these threads, with Cloudflare, Google and Quad9 as upstream resolvers, and they all validate correctly now.

v0.16.26 ships with the patched resolver and will be out today or tomorrow. After upgrading you can remove any TLS strategy you added to turn DANE off for specific domains. If a domain still fails after the upgrade, post it in this thread together with the DNS resolver you use and we’ll look into it.