ACME LetsEncrypt failures are permanent and terminal

Issues
1

Issue Description

For some unknown reason, the ACME LetsEncrypt renewal failed on my server with “HTTP error: builder error”, exactly 1 month ago. I had not changed anything on the server when the problem appeared, it’s just the normal renewal that failed with “HTTP error: builder error”.

However, what’s far more severe than the original error: the task was never re-tried. I see a failed “Task” in the web UI, last month, and no new attempts.

I had tried all kinds of things, restarted the server, even changing the ACME provider from TLS to HTTP method, but to no avail.

The only thing that helped was to change the domain from automatic TLS to manual, and then back to Automatic (and configure the ACME provider again). Only then did Stalwart re-try. And it worked immediately.

So, there are 2 issues here:

  • The “builder error”. Given that it had worked before, and I didn’t change any config, and it now works again with the same config, that is not a misconfigured URL. (BTW: I also checked the URL and it was correct the whole time.)

  • A single renewal failure is fatal. It never retries. Which then breaks the entire server.

I could not even get to the web UI of the server, because of a) HSTS (which makes override impossible) and b) the “risks” of it being an actual attack and entering the admin password and c) the ssh -L trick doesn’t work either, due to how stalwart does its own authentication with OAuth2 and keeps going back to the original domain instead of localhost.

This took me several hours to debug and figure out.

Expected Behavior

a) No “builder error”
b) Temporary ACME failures are re-tried after a few days. (That’s why we renew a month earlier than necessary.)

Actual Behavior

a) Unprovoked “HTTP error: builder error”
b) Temporary ACME failures become fatal and kill the entire server and the admin UI.

Relevant Log Output

No log output, because no ACME was even attempted.

The task failed with “HTTP error: builder error” (no other info).

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

Additional Context

Retries (today) with Stalwart v0.16.19

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Fixed in v0.16.18 (yet to be released).

3

Here’s what Claude disected from the commit log:

Issue (b) — failure is fatal, never retried: b1983c88 (2026-08-16, “Fix Task manager: totalDeadline is measured from the time a task was created instead of its first failed attempt”). Committed at 17:32 CEST, about half an hour after mdecimus posted the reply on the thread. It does two things in crates/services/src/task_manager/manager.rs: the retry deadline clock now starts at the first failed attempt rather than at task creation, and it adds a perpetual_retry_time() function that applies to TaskType::AcmeRenewal and TaskType::DkimManagement. When the normal retry schedule or totalDeadline is exhausted, those two task types get rescheduled anyway with exponential backoff capped between 1 hour and 6 hours, instead of moving into a dead Failed state. That is exactly the reported symptom.

Issue (a) — “HTTP error: builder error”: 73d27006 (2026-08-14, “HTTP: Shared reqwest clients”). This is inference on my part, not something the maintainer stated. Before it, crates/common/src/network/acme/http.rs constructed a fresh reqwest::Client for every ACME request, and each build re-ran TLS setup including loading the platform root certificate store. A failure there is reqwest’s builder error kind, which Stalwart renders as “HTTP error: builder error” with no further detail — matching the vague, unreproducible message. The commit replaces that with a single rustls ClientConfig built once at startup (utils::http::init_shared_tls_configs() is now called from main.rs) and reused through use_preconfigured_tls, so per-request client construction can no longer fail that way. It’s in v0.16.18, but it has no CHANGELOG entry and isn’t described as a bug fix, so treat the link as plausible rather than confirmed.

Follow-up in v0.16.19: 0e19a60b (2026-08-23, “ACME fixes”). Changelog credits it with “Order and authorization failures are never logged” and “An order rejected by the CA marks the renewal task as permanently failed.” It adds trc::event!(Acme(AcmeEvent::Error)) and RenewBackoff logging, and introduces a TaskFailureType::Perpetual variant, moving OrderInvalid, Json and Registry errors out of the permanent bucket. This one primarily targets the related topic #1422 (CAA rejection silent retry loop), but it reinforces the same retry fix and adds the logging that would have made the original builder error diagnosable.

So: the “never retries” half is properly fixed and then hardened again a week later. The “builder error” half was never root-caused publicly — the code path that can produce it was refactored away as a side effect, and the logging added in 0.16.19 means a recurrence would at least be visible next time.

4

Both halves shipped, so 0.16.19 has everything: b1983c88 went out in 0.16.18 and 0e19a60b in 0.16.19.

The bit you had to infer is worth stating outright, since it is the actual cause of “one failure is fatal”. On successful issuance the next renewal task is created straight away but scheduled ~30 days out, and the retry deadline used to be measured from when the task was created rather than from the first failed attempt. With totalDeadline defaulting to 6 hours, the deadline had already passed a month before the renewal ever ran, so the very first failure returned no next-retry time and the task went to Failed on attempt zero. That is also why toggling to Manual and back was the only cure: it created a new task with a new timestamp.

The builder-error link can be confirmed now. Before 73d27006 every ACME request built its own reqwest client, and in reqwest 0.13 that build step re-reads the platform certificate verifier, which is the one thing in that path that returns the builder error kind. It is constructed once at startup now.

Since it will come up again: AcmeError::Http still prints only reqwest’s top-level Display, so whatever really failed is dropped on the floor. The ACME error events added in 0.16.19 are what would surface a recurrence, and even those stay terse. Post-fix, AcmeRenewal and DkimManagement run the normal backoff and then retry perpetually at 1h doubling to a 6h cap, so neither reaches Failed any more; Permanent now covers only Crypto, Invalid, NotDue and ChallengeNotSupported.