Both settings can point at the very same resolver (127.0.0.1:53), but Stalwart decides whether DNSSEC/DANE is usable differently depending on how the connection to that resolver is described internally, not just whether TCP fallback is enabled.
When you use System Resolver, Stalwart parses /etc/resolv.conf and builds the name-server connection list from it. Entries read this way are configured as plain UDP connections; the “Try TCP on Error” flag only makes Stalwart retry a query over TCP after a UDP failure, it does not change the underlying connection type that was registered for that name server.
When you use Custom DNS and explicitly select protocol tcp for the server entry, Stalwart registers that name server with a Tcp connection type from the start.
Internally, at startup Stalwart runs a DNSSEC capability check (ensure_dnssec) that requires two things:
- At least one configured name-server connection whose protocol is not UDP (i.e., TCP or TLS), and
- A successful, cryptographically validated DNSKEY lookup against that resolver.
With the System Resolver, condition 1 fails, because the connections read from resolv.conf are all UDP, even though tcp_on_error is true. That is why you get the “cannot validate DNSSEC … DANE has been disabled” warning, and DANE is disabled to avoid deferring mail.
With Custom DNS and protocol explicitly set to tcp, condition 1 is satisfied (a genuine TCP connection is registered), and if the DNSKEY lookup then validates successfully, DANE stays enabled and no warning is shown.
So functionally, if your 127.0.0.1:53 resolver is DNSSEC-validating and reachable over TCP, the Custom DNS setup with protocol: tcp is the way to get DANE enabled; the System Resolver path currently cannot satisfy the TCP-connection check regardless of tcp_on_error.
Reference: DNS Resolver docs and DANE docs.
This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.