Server not accessible in recovery mode after a blob store misconfiguration

Issues
1

Issue Description

I recently got locked out by making a typo in the blob store configuration.

I lost access to the server in recovery mode via the WebUI and stawart-cli (I did not try the apply command since I was playing around locally and did not have a snapshot file).

Expected Behavior

Server accessible via the WebUI and stawart-cli in recovery mode when the blob storage is misconfigured.

Actual Behavior

Error in the WebUI and stalwart-cli.

Reproduction Steps

  1. Configure Stalwart v0.16.23 with a blob store and ensure the connection succeeds.
  2. Update the blob store configuration and restart Stalwart in recovery mode.

Relevant Log Output

ERROR S3 error (store.s3-error) reason = "<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message></Message><RequestId>...</RequestId><HostId>...</HostId></Error>", code = 403, causedBy = "crates/store/src/dispatch/blob.rs:52", causedBy = "crates/common/src/network/mta.rs:452", details = "Failed to load spam filter model", causedBy = "crates/common/src/manager/boot.rs:222"
ERROR Resource error (resource.error) reason = S3 error (store.s3-error) { reason = "<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message></Message><RequestId>...</RequestId><HostId>...</HostId></Error>", code = 403, causedBy = "crates/store/src/dispatch/blob.rs:52" }, url = "https://github.com/stalwartlabs/webui/releases/latest/download/webui.zip", details = "Failed to unpack application for prefixes: admin, account, no bundle is available to serve"

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

PostgreSQL

Blob Storage

S3-compatible

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Recovery mode does not fully isolate the server from a broken blob store. The webui bundle is treated as an Application object, and unpacking it is not gated on recovery mode: start_services() always calls applications.unpack_all(), which first checks the blob store for a cached copy of the bundle before falling back to downloading it from GitHub. When the blob store is misconfigured, that cache lookup fails with the S3 error you saw, and if no bundle is cached yet (or the cached entry can’t be read), the server has nothing to serve for /admin or /account, so the WebUI itself comes up blank in recovery mode too.

The JMAP management API (which stalwart-cli talks to directly, not through the webui bundle) is not supposed to depend on the blob store being healthy though, so a BlobStore object can normally still be corrected via stalwart-cli even while the WebUI bundle fails to load. If you found the CLI itself refusing to connect or authenticate under the same conditions, that’s beyond what’s documented and would need a closer look at your specific setup (recovery admin credentials, listener state, etc.) rather than something covered by current docs.

For reference, the recovery mode mechanism and the STALWART_RECOVERY_ADMIN fallback credential are described here: Recovery mode | Stalwart

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

Fixed in v0.16.25 (will be released today).