Bug Report: S3 'Purge blob store' task fails with 403 SignatureDoesNotMatch on Cloudflare R2

Issues
1

Issue Description

After upgrading to the v0.16.x branch, the background maintenance task “Purge blob store” fails consistently with an S3 SignatureDoesNotMatch (403) error when using Cloudflare R2 as the blob backend.

The underlying problem is that Stalwart’s S3 dispatcher implementation is including an empty “content-length:” and a “content-type:text/plain” header inside the CanonicalRequest payload during a HTTP DELETE operation. Because a standard DELETE request has no body payload, Cloudflare R2 calculates the signature according to strict AWS specs without these malformed entries, leading to an authentication signature mismatch.

All other operations (such as initial writes and verification reads) function flawlessly; the failure is isolated strictly to the blob deletion/purge code path introduced in the rewrite.

Expected Behavior

Stalwart should successfully purge orphaned objects from the S3/R2 bucket during maintenance. The internal S3 client should construct a standard, bodyless HTTP DELETE request and sign it without injecting empty content-length or default text content-type headers into the CanonicalRequest signature string.

Actual Behavior

The maintenance task fails with a 403 Forbidden error originating from crates/store/src/dispatch/blob.rs:221. The R2 storage engine returns a SignatureDoesNotMatch XML response because Stalwart’s calculated signature does not match the signature calculated by Cloudflare.

Reproduction Steps

  1. Deploy Stalwart Mail Server v0.16.7.
  2. Configure an S3 Blob Storage backend pointing to a Cloudflare R2 bucket.
  3. Ensure the API keys have full administrative read/write/delete privileges and IP whitelisting allows connections.
  4. Trigger or wait for the automatic “Purge blob store” maintenance task to execute.
  5. Check the logs or the administration panel to see the failed task state.

Relevant Log Output

S3 error (store.s3-error): reason = <?xml version="1.0" encoding="UTF-8"?>SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your secret access key and signing method.AWS4-HMAC-SHA256
20260526T041058Z
20260526/auto/s3/aws4_request
7182721d7d7680ad930210d1c8ee3f14bc078bece4d38b90b70404717346709441 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 36 30 35 32 36 54 30 34 31 30 35 38 5a 0a 32 30 32 36 30 35 32 36 2f 61 75 74 6f 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 37 31 38 32 37 32 31 64 37 64 37 36 38 30 61 64 39 33 30 32 31 30 64 31 63 38 65 65 33 66 31 34 62 63 30 37 38 62 65 63 65 34 64 33 38 62 39 30 62 37 30 34 30 34 37 31 37 33 34 36 37 30 39 34DELETE
/lewsion-stalwart-blobs-wnam/agsutgzdcsudtiygpfhellacnz3zeajxiqlkbx3l37vqy17k9o1q

content-length:
content-type:text/plain
host:9d1aa3bcce298960a2fc322768d4f019.r2.cloudflarestorage.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20260526T041058Z

content-length;content-type;host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b85544 45 4c 45 54 45 0a 2f 6c 65 77 73 69 6f 6e 2d 73 74 61 6c 77 61 72 74 2d 62 6c 6f 62 73 2d 77 6e 61 6d 2f 61 67 73 75 74 67 7a 64 63 73 75 64 74 69 79 67 70 66 68 65 6c 6c 61 63 6e 7a 33 7a 65 61 6a 78 69 71 6c 6b 62 78 33 6c 33 37 76 71 79 31 37 6b 39 6f 31 71 0a 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 0a 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 74 65 78 74 2f 70 6c 61 69 6e 0a 68 6f 73 74 3a 39 64 31 61 61 33 62 63 63 65 32 39 38 39 36 30 61 32 66 63 33 32 32 37 36 38 64 34 66 30 31 39 2e 72 32 2e 63 6c 6f 75 64 66 6c 61 72 65 73 74 6f 72 61 67 65 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 36 30 35 32 36 54 30 34 31 30 35 38 5a 0a 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3b 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3b 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 1 63 1 34 49 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35dcb39a0d353f3744f92369f48db810956b8ec118b62d7f13c234c4ba839112b9, code = 403, causedBy = crates/store/src/dispatch/blob.rs:221, causedBy = crates/store/src/write/blob.rs:141, causedBy = crates/services/src/task_manager/maintenance.rs:273

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

PostgreSQL

Blob Storage

S3-compatible

Search Engine

PostgreSQL

Directory Backend

Internal

Additional Context

  • The deployment environment is a standard Linux VPS.
  • Credentials and bucket access permissions have been verified independently using standalone S3 clients with IP whitelisting active; manual object operations (including deletions) are fully functional outside of Stalwart.
  • The signature calculation logic mismatch became apparent immediately following the platform migration to the v0.16 release stream.

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Let me look at the relevant source code to understand the issue.

The Stalwart S3 backend calls self.bucket.delete_object(...) from the rust-s3 library. The signature canonicalization issue (empty content-length: and content-type:text/plain headers being included in the DELETE request’s CanonicalRequest) is happening inside the rust-s3 crate, not directly in Stalwart’s own code.

This is a confirmed bug in the S3 DELETE path introduced in the v0.16.x rewrite. The CanonicalRequest your log shows clearly includes:

content-length:
content-type:text/plain

…for what should be a bodyless DELETE request. This causes Cloudflare R2 (which strictly follows AWS SigV4 specs) to reject the request with SignatureDoesNotMatch, while AWS itself is apparently more lenient.

Root cause: The rust-s3 library (the dependency Stalwart uses) is incorrectly injecting content-length (empty) and content-type: text/plain headers into the canonical request for DELETE operations. Since DELETE has no body, these headers should not be present in the signed headers list at all. Stalwart’s own delete_blob code at crates/store/src/backend/s3/mod.rs just calls bucket.delete_object(...) – the defect is in how rust-s3 constructs and signs that request.

Workaround options until a fix is released:

  • There is no configuration-level workaround for this within Stalwart, since the signing logic is in the rust-s3 dependency.
  • You can temporarily disable the “Purge blob store” maintenance task in the admin panel to stop the errors, with the understanding that orphaned blobs will accumulate until a fix is applied.
  • Track the Stalwart server repository for a dependency bump or patch to rust-s3 that corrects the DELETE signing behavior.

This is a genuine bug that will need to be fixed at the code level (either by patching the rust-s3 dependency or by manually stripping those headers before signing in delete_blob). The maintainers will need to address it in a future release.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

I can confirm this exact same behavior on v0.16.x using Cloudflare R2. My logs show the exact same CanonicalRequest hash mismatch due to the content-type:text/plain header on the empty DELETE.

Matt

4

Same here, had to shutdown my cluster due to corruption

5

This is a bug in the rust-s3 crate, track this issue DeleteObject signs `content-length` and `content-type` headers, causing 403 SignatureDoesNotMatch on Cloudflare R2 · Issue #466 · durch/rust-s3 · GitHub

6

I can confirm this also applies for all sort of s3 signatures when using ceph rgw as s3 store.

Since upgrading Ceph RGW to Ceph 20.2.2 Tentacle, Stalwart 0.16.11 fails against a Ceph-backed S3-compatible blob store with SignatureDoesNotMatch generally.

The same Ceph RGW endpoint, bucket, credentials, and path-style configuration work correctly with other S3 clients, for example the PHP AWS SDK. This appears to be a SigV4 canonical request / signed-header compatibility issue between Stalwart’s S3 blob backend, rust-s3, and stricter Ceph RGW request validation/canonicalization in Ceph 20.2.2.

A working PHP AWS SDK request signs only the stable SigV4 headers:
SignedHeaders=host;x-amz-content-sha256;x-amz-date

The failing Stalwart / rust-s3 request appears to include additional HTTP/client headers in SignedHeaders, for example:
SignedHeaders=accept;host;x-amz-content-sha256;x-amz-date
or similar volatile transport/client headers.

This matters because Ceph RGW verifies S3 requests using AWS SigV4-compatible authentication. In SigV4, every header listed in SignedHeaders must match exactly when the server reconstructs the canonical request.

Headers such as the following are not required for S3 SigV4 authentication and may be injected, normalized, removed, or changed by the HTTP stack or proxies:
accept, accept-encoding, user-agent, connection, amz-sdk-request, amz-sdk-invocation-id

If one of these volatile headers is included in SignedHeaders, Ceph RGW can calculate a different canonical request and reject the request with SignatureDoesNotMatch.

7

We’re waiting for the fix upstream.

8

I also have this issue, and this preventing me to upgrade the webui

9

Also have this issue with SeaweedFS S3 service

It seems perhaps rust-s3 is not really a mature S3 implementation if a bug blocking a basic operations goes unfixed for many months.

10

I completely agree with the sentiment about rust-s3. If a fundamental operation like a bodyless DELETE request is being signed incorrectly and goes unfixed for months, it is hard to trust it as mature enough for critical mail infrastructure.

Because of this, I decided to completely drop Cloudflare R2 and migrate my entire blob storage to native Amazon S3. My philosophy going forward is that “S3 is S3” really means it belongs to Amazon. While third-party implementations like R2 or Backblaze are great for raw storage, AWS is the reference standard. Sticking to native AWS S3 means I don’t have to worry about strict SigV4 parsing failures from third-party endpoints rejecting malformed headers that AWS itself handles gracefully.

To fix this once and for all:

  1. I used rclone to sync my entire blob bucket directly from Cloudflare R2 to an AWS us-west-1 S3 bucket.

  2. I put Stalwart into recovery mode, updated the BlobStore singleton via the stalwart-cli to point to AWS S3, and injected my new IAM credentials.

  3. I restarted the server.

Overall, I had to sacrifice about 5 to 10 minutes of downtime for the mail server, but the result is 100% stability. The “Purge blob store” maintenance task now runs flawlessly without a single 403 SignatureDoesNotMatch error. Considering how incredibly cheap native AWS S3 is for standard mail volumes, getting rid of this headache and achieving full compatibility was absolutely worth the switch.