v0.16.12→v0.16.19: every TLS-terminated listener accepts then silently closes with zero bytes (survives recreate + upgrade); no log output anywhere

Issues
1

Issue Description

Summary
Every TLS-terminated listener (IMAP/993, JMAP+HTTPS/443, SMTP submission) accepts the TCP connection and then closes it with zero bytes sent back — no TLS ServerHello, no plaintext banner, nothing — before any protocol handshake completes. This is not intermittent: it is 100% reproducible on every connection attempt, from every network path tested (public internet, host loopback, and container-to-container within the same Docker network).

The server has been in this state continuously. Docker’s own HEALTHCHECK (curl -k https://127.0.0.1:443/healthz/live) intermittently reports “healthy,” which is misleading — it is not representative of whether real protocol traffic (IMAP LOGIN, JMAP session, SMTP) can complete.

Environment
Deployed via the official stalwartlabs/stalwart Docker image, using the standard docker-compose.yml pattern from the Stalwart install docs (RocksDB backend, config.json as a datastore-pointer bootstrap file, all real config/domains/certificates stored in the RocksDB-backed registry).
Started on v0.16.12, running continuously for ~7 weeks before the fault was noticed.
Docker: docker compose v2, host is a standard Linux VPS (no unusual kernel/network config — verified: no fail2ban, no ufw, no custom iptables DROP/REJECT rules; iptables -L DOCKER-USER is empty; packet counters confirm inbound SYNs are correctly DNAT’d to the container).
What I ruled out, with direct evidence, before filing this
Resource exhaustion — checked /proc//limits and open FD count on the main process directly: 54 of 65,535 file descriptors in use. Memory ~360MB/7.7GB. CPU idle. Disk 17% used. None of these are close to any limit.

OS/network-namespace state — did a full docker compose down && up (not restart), which tears down and recreates the network namespace, port bindings, and iptables NAT rules from scratch. Identical failure afterward.

Binary version — upgraded from stalwartlabs/stalwart:v0.16.12 to :latest (resolved to v0.16.19) via the same procedure (down, swap image tag, up), same data volume. Identical failure on the new binary, immediately after a completely fresh process start.

Host firewall / connection filtering — fail2ban is inactive (not even installed as a running service), ufw is inactive, iptables -L DOCKER-USER -n -v is empty (0 rules), and the NAT table’s DNAT rules for 993/443/etc. show real packet/byte counters incrementing on test connections, confirming traffic reaches the container’s actual listening socket.

Certificate problems — used the built-in --console Data Store CLI to scan the SUBSPACE_REGISTRY (‘s’) keyspace directly (source-verified subspace byte from crates/store/src/lib.rs), extracted all 88 stored leaf certificates + chains (352 total BEGIN/END CERTIFICATE blocks), and decoded every one with openssl x509 -noout -subject -ext subjectAltName -dates. Findings:

No expired certificates. All notAfter dates are in the future relative to server time.
No duplicate/conflicting certificates for the same SAN set (checked specifically, since this pattern is described in a support thread as a known failure mode — not present here).
The specific certificate needed for the actual IMAP/JMAP hostname (CN=mx1.maillan.com, SAN mail.maillan.com, mx1.maillan.com) is present, singular, and currently valid (notBefore=Jul 9 2026, notAfter=Oct 7 2026).
A server.proxy.trusted-networks misconfiguration — this failure signature (connection accepted, then dropped silently expecting a PROXY-protocol header) is described in Stalwart’s own troubleshooting docs as an exact match. I searched the registry keyspace for any literal or namespaced form of this setting — nothing found. This deployment has no reverse proxy in front of Stalwart’s own published ports (Caddy in this stack only proxies two unrelated services on different ports; Stalwart’s 443/993/etc. are published directly).

No visibility from logging, at any level — this is the part I can’t get past without your help:

docker logs stalwart has been completely empty since the container was first created 7+ weeks ago (0 lines), across both v0.16.12 and v0.16.19.
Re-ran the server attached, with RUST_LOG=trace and RUST_BACKTRACE=full explicitly set, on a fresh container instance — made a real connection attempt against it — still zero stdout/stderr output, before or after the failed connection.
Checked whether telemetry/tracing is instead being written to Stalwart’s own internal store (SUBSPACE_TELEMETRY_SPAN, byte ‘o’, per crates/store/src/lib.rs) — scanned that entire keyspace directly via the console — it’s completely empty. Tracing does not appear to be enabled by default, and I have no way to enable it (the WebUI/JMAP admin API needed to configure it is the exact thing that’s unreachable due to this bug — a hard catch-22).
What this points to
Given the data is verified clean and the fault survives both a full environment recreate and a version upgrade, this looks like a runtime-level fault in how the server handles TLS/connection setup against this specific account/domain dataset (~280 mailboxes, ~20 domains with Automatic ACME certificate management via Let’s Encrypt, TlsAlpn01 challenge type) — not a configuration or data-corruption issue on my end. I’m not able to narrow it further without either the WebUI (blocked by this exact bug) or guidance on capturing meaningful diagnostics for a genuinely silent failure like this.

What would help
Is there a way to force verbose logging to stdout that doesn’t depend on RUST_LOG/RUST_BACKTRACE (e.g. a config value that must be set inside the RocksDB-backed config itself, which I can’t reach right now)?
Is there a known failure mode where a large number of Automatic-managed ACME domains (each with 3 auxiliary SANs: mta-sts., autoconfig., autodiscover.) could cause the TLS acceptor to hang/panic per-connection during certificate selection, in a way that wouldn’t show up as data corruption?
Any recommended way to safely inspect/repair this from the CLI/console alone, without the WebUI, given the exact SUBSPACE_REGISTRY key structure for Certificate/Domain objects (I reverse-engineered the subspace byte from source but not the full object-key encoding within it)?
Happy to provide anything else — I have a full backup of the RocksDB data taken immediately before any of this troubleshooting, plus the complete decoded certificate inventory, if that’s useful for reproduction.

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

This is exactly what I need. Let me pull the full recovery-mode doc to give a precise path forward.

This gives a very solid path forward. Let me check the environment-variables page for recovery mode log level details already covered, and I have enough now.

Recovery mode is exactly the way out of this catch‑22. Setting STALWART_RECOVERY_MODE=1 (and optionally STALWART_RECOVERY_ADMIN=user:pass if you can’t sign in normally) makes Stalwart start with every mail/TLS listener disabled and expose only a plain HTTP management endpoint on port 8080 (STALWART_RECOVERY_MODE_PORT to change it) serving the WebUI and full JMAP management API. From there you can inspect and fix the TLS/certificate/listener configuration through the WebUI or stalwart-cli without needing any of the broken 443/993 listeners to work at all. Logging in recovery mode also goes to stderr at the level set by STALWART_RECOVERY_MODE_LOG_LEVEL (default info), so you’ll get visibility even before you touch tracer settings.

Once you’re in via recovery mode:

  • Check whether a Stdout tracer object exists and is enabled; by default Stalwart does not automatically ship a console tracer with trace/debug level turned on, which explains why RUST_LOG/RUST_BACKTRACE alone produced nothing — those env vars aren’t how Stalwart’s own tracing subsystem is configured; tracer configuration lives in the database as a Tracer object (@type: "Stdout") with enable/level fields, editable from Settings › Telemetry › Tracers.
  • Look specifically for the tls.not-configured and tls.certificate-not-found events once tracing is on — these are logged precisely for the “accepts then drops silently” signature when a listener’s TLS handshake can’t find a matching certificate for the SNI/hostname requested, which matches your symptom description more closely than a proxy-protocol misconfiguration would.

Docs used: Recovery mode, Environment variables, Console tracer, Database Console.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

I can’t speak to why the logs are missing, but…

This behavior will occur when the internal Stalwart security module has blocked an IP. There are a few triggers by default that can cause this to happen.

If you’re running inside of Docker, my guess would be that Stalwart has blocked the Docker subnet.

It works like fail2ban would, but it’s all inside of the binary.

Put Stalwart into Recovery Mode as the bot suggested, and try log in with HTTP 8080.

If you get the WebUI, then head to Settings → Security → Blocked IPs. Find what is stopping you, and add it to the allowed list, Reload Settings → Blocked IP list, then restart the server in normal mode.

/live only tells you if the server has a heartbeat. If you’re looking for an indicator of functionality /ready is better.

Hope that helps.

4

Start with the logging, because the silence is not evidence of anything. We ignore RUST_LOG entirely (telemetry is configured through Tracer objects), and the setup wizard’s default tracer writes a rotating file under /var/log/stalwart/. The official image neither creates that directory nor runs as a user who can write it, so the tracer opens nothing and quietly exits, and docker logs has been empty since day one. Your RUST_LOG experiment was a no-op.

To get output without the WebUI, restart with STALWART_RECOVERY_MODE=1, STALWART_RECOVERY_ADMIN=admin:<password> and optionally STALWART_RECOVERY_MODE_LOG_LEVEL=trace. That forces a console tracer on stderr and opens plain HTTP on 8080 (Recovery mode | Stalwart). Recovery mode also ignores the blocked-IP list, so it stays reachable even if a ban is what is shutting you out.

Which I think it is. A blocked source address is dropped in build_session right after accept(), before the TLS acceptor ever runs, so you get an accepted socket, zero bytes, no ServerHello and no banner: your signature exactly. Under Docker with the userland proxy every external connection usually arrives from the bridge gateway address, so a single ban takes out every client at once. Ban periods are unset by default, which means bans never expire, so one bad day seven weeks ago is still in force today. Once you are in, check Settings > Security > Blocked IPs (or stalwart-cli query BlockedIp), delete the entry, add the address or network to Allowed IPs, and set authBanPeriod and loiterBanPeriod so future ones expire (Auto-banning | Stalwart). Then switch the tracer to the Stdout variant so docker logs works from here on.

On ACME: no, a large number of domains cannot hang the acceptor. The ALPN challenge path only applies to HTTPS listeners, and certificate selection falls back to a wildcard and then to any available cert, so it cannot block. That is also why IMAP on 993 fails identically, which an ACME fault could not cause. Your healthcheck reporting healthy over 443 says the same thing: TLS termination completes fine for at least one source address.