S3 blob store (Hetzner Object Store) fails reads and writes, no useful debug information

Issues
1

Issue Description

Any operation (read or write) on the S3 bucket fails with an opaque error message.

Expected Behavior

Stalwart should be able to read and write from and to the configured Hetzner Object store.

Actual Behavior

Reads and writes get an error with reason “reqwest: error sending request for url” No status codes, no other information other than the backtrace (and the S3 URL which I am restricted from pasting here, due to an URL limit).

Reproduction Steps

Configure stalwart with a BlobStore to Hetzner:

{
“region”: {
“customEndpoint”: “https://nbg1.your-objectstorage.com”,
“customRegion”: “nbg1”,
“@type”: “Custom”
},
“bucket”: “REDACTED-mail”,
“accessKey”: “REDACTED”,
“secretKey”: {
“secret”: “****”,
“@type”: “Value”
},
“securityToken”: {
“@type”: “None”
},
“sessionToken”: {
“@type”: “None”
},
“profile”: null,
“timeout”: 15000,
“maxRetries”: 3,
“keyPrefix”: “cc_”,
“allowInvalidCerts”: true,
“verifyAfterWrite”: true,
“@type”: “S3”,
“id”: “singleton”
}

Relevant Log Output

2026-06-23T04:10:34Z ERROR S3 error (store.s3-error) reason = “reqwest: error sending request for url (URL_REDACTED)”, causedBy = “crates/store/src/dispatch/blob.rs:221”, causedBy = “crates/store/src/write/blob.rs:141”, causedBy = “crates/services/src/task_manager/maintenance.rs:273”, details = “Failed to perform store maintenance task”
2026-06-23T04:10:34Z WARN Task failed during processing (task-manager.task-failed) id = 314378544350272517, details = “StoreMaintenance”, reason = “S3 error (store.s3-error): reason = reqwest: error sending request for url (URL_REDACTED_BUT_CORRECT), causedBy = crates/store/src/dispatch/blob.rs:221, causedBy = crates/store/src/write/blob.rs:141, causedBy = crates/services/src/task_manager/maintenance.rs:273”
2026-06-23T04:10:40Z ERROR S3 error (store.s3-error) reason = “reqwest: error sending request for url (URL_REDACTED)”, causedBy = “crates/store/src/dispatch/blob.rs:221”, causedBy = “crates/store/src/write/blob.rs:141”, causedBy = “crates/services/src/task_manager/maintenance.rs:273”, details = “Failed to perform store maintenance task”
2026-06-23T04:10:40Z WARN Task failed during processing (task-manager.task-failed) id = 314378544350250501, details = “StoreMaintenance”, reason = “S3 error (store.s3-error): reason = reqwest: error sending request for url (URL_REDACTED_BUT_CORRRECT), causedBy = crates/store/src/dispatch/blob.rs:221, causedBy = crates/store/src/write/blob.rs:141, causedBy = crates/services/src/task_manager/maintenance.rs:273”

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

RocksDB

Blob Storage

S3-compatible

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

The error "reqwest: error sending request for url" is a low-level network/TLS transport error from the reqwest HTTP client – it means the TCP/TLS connection to the Hetzner endpoint itself is failing before any HTTP response is even received. This is distinct from an S3 authentication or permissions error (those return HTTP 4xx/5xx codes). There are a few likely culprits:

1. TLS certificate validation (most likely given allowInvalidCerts: true is already set)

You’ve already set allowInvalidCerts: true, which bypasses certificate validation. However, reqwest can still fail at the TCP/TLS layer if:

  • The Docker container cannot reach nbg1.your-objectstorage.com at all (DNS resolution failure, firewall, missing network egress).
  • The TLS handshake is failing for a reason other than certificate validity (e.g., cipher suite mismatch).

2. Connectivity from within the Docker container

The most common cause in Docker setups is that the container cannot reach the external endpoint. Try:

  • curl -v https://nbg1.your-objectstorage.com from inside the container (or from the host with the same network) to verify basic connectivity.
  • Check that your Docker network has external internet access and that no firewall rules block outbound HTTPS (port 443).

3. Increase log verbosity to get more detail

The error message is truncated because the tracer level is too coarse. In the WebUI under Settings > Telemetry, set your tracer’s level to trace. This will produce much more verbose output from the reqwest/rustls layer including the specific underlying error (e.g., “connection refused”, “DNS error”, “certificate unknown”, etc.).

4. Hetzner Object Store endpoint URL format

Hetzner’s S3-compatible API typically uses path-style or virtual-hosted-style URLs. Some S3 clients default to virtual-hosted style (bucket.endpoint), which may not be supported by all providers. The customEndpoint in your config looks correct (https://nbg1.your-objectstorage.com), but check the Hetzner docs to confirm whether your bucket region/endpoint combination is right – some Hetzner accounts use a different base URL format.

Summary of steps to try:

  1. Set tracer level to trace or debug to expose the underlying reqwest/rustls error.
  2. Verify outbound connectivity to nbg1.your-objectstorage.com:443 from inside the Docker container.
  3. Confirm the exact endpoint URL format required by Hetzner for your account/region.

Docs reference: S3-compatible backend configuration and Tracing/logging levels.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

I have verified on another machine that the URL, credentials, and bucket are correct (, and they were used as they are in previous versions of stalwart).

I have also verified the blobs that are trying to be read do exist and are accessible with these credentials.

For writes, I have pushed the exact files manually without problem (such as the webui package).

I have verified that DNS connectivity is correct within the container.

4

The tracer level is already at trace, and I see other TRACE-level logs. I have seen no further reqwest or rustls logs than variations of the ones I posted.

I have already verified cURL connectivity from within the container. I can connect to the S3 server just fine.

I have checked, rechecked, and copy-and-pasted the URL against my workstation, from which connectivity works just fine (and with the same credentials).

5

FWIW, I have also tried connecting by IP address, to rule out DNS issues, and that also fails in the same manner (obviously, TLS cert validation is disabled for this).

6

Here is the output of the cURL from within the container. It fails with a 403, which is expected, since I am not supplying any authentication material.

$ curl -v https://nbg1.your-objectstorage.com/cycore-mail/cc_rvlcfpc7pomdfyoycd3nm2sq3uwaohrogmfaidwvfr7jygsitijq
*   Trying [2a01:4f8:b000::1]:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* SSL Trust Anchors:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
*   CApath: /etc/ssl/certs
* Host nbg1.your-objectstorage.com:443 was resolved.
* IPv6: 2a01:4f8:b000::1
* IPv4: 88.198.120.0
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*   subject: CN=nbg1.your-objectstorage.com
*   start date: Jun 18 19:47:22 2026 GMT
*   expire date: Sep 16 19:47:21 2026 GMT
*   issuer: C=US; O=Let's Encrypt; CN=YE2
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 3: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
*   subjectAltName: "nbg1.your-objectstorage.com" matches cert's "nbg1.your-objectstorage.com"
* OpenSSL verify result: 0
* SSL certificate verified via OpenSSL.
* Established connection to nbg1.your-objectstorage.com (2a01:4f8:b000::1 port 443) from fdee:4f32:ab0b:c303::d567 port 35974 
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://nbg1.your-objectstorage.com/cycore-mail/cc_rvlcfpc7pomdfyoycd3nm2sq3uwaohrogmfaidwvfr7jygsitijq
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: nbg1.your-objectstorage.com]
* [HTTP/2] [1] [:path: /cycore-mail/cc_rvlcfpc7pomdfyoycd3nm2sq3uwaohrogmfaidwvfr7jygsitijq]
* [HTTP/2] [1] [user-agent: curl/8.20.0]
* [HTTP/2] [1] [accept: */*]
> GET /cycore-mail/cc_rvlcfpc7pomdfyoycd3nm2sq3uwaohrogmfaidwvfr7jygsitijq HTTP/2
> Host: nbg1.your-objectstorage.com
> User-Agent: curl/8.20.0
> Accept: */*
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 403 
< content-length: 263
< x-amz-request-id: tx000007d839ff1a2e6daf1-006a3a6e19-26b7f260-nbg1-prod1-ceph4
< accept-ranges: bytes
< content-type: application/xml
< date: Tue, 23 Jun 2026 11:29:30 GMT
< strict-transport-security: max-age=63072000
< x-debug-bucket: cycore-mail
< 
* Connection #0 to host nbg1.your-objectstorage.com:443 left intact
<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message></Message><BucketName>cycore-mail</BucketName><RequestId>tx000007d839ff1a2e6daf1-006a3a6e19-26b7f260-nbg1-prod1-ceph4</RequestId><HostId>26b7f260-nbg1-prod1-ceph4-nbg1</HostId></Error>/var/lib/stalwart $
7

Let me look into the S3 configuration and any known issues with custom endpoints.

The key findings here are clear. Let me summarize what I found:

The error "reqwest: error sending request for url" is a connection-level failure (not an HTTP error response). It’s produced by the into_error function that wraps any s3 crate error directly – it never reaches an HTTP status code, meaning the request didn’t complete at all. This is a transport/network error from the underlying reqwest HTTP client.

There are two important things to note from the source code:

1. allowInvalidCerts is commented out / not wired in

Looking at the S3 store initialization code:

.with_path_style()
/*.set_dangereous_config(allow_invalid, allow_invalid)
.map_err(|err| {
    format!("Failed to create bucket: {err:?}")
})?*/

The allowInvalidCerts field in your config is currently commented out in the source – it has no effect. If Hetzner’s TLS certificate is not trusted by the system CA bundle in the Docker container, the connection will fail at the TLS handshake with exactly this kind of opaque reqwest transport error.

2. Path-style addressing is hardcoded – .with_path_style() is always applied, which is correct for most S3-compatible services including Hetzner.

Most likely cause: The Docker container’s CA bundle does not trust the TLS certificate on nbg1.your-objectstorage.com, and the allowInvalidCerts: true setting you’ve configured is currently a no-op (the code is commented out). The connection fails before any HTTP response is received, which is why there’s no status code in the error.

Things to try:

  1. Verify connectivity from inside the container – run curl -v https://nbg1.your-objectstorage.com from inside the Stalwart Docker container to confirm whether TLS verification is failing.
  2. Check if the container has a complete CA bundle – some minimal Docker images (Alpine-based, etc.) lack the full ca-certificates package. Ensure ca-certificates is installed and up to date.
  3. Test your credentials and bucket independently – use aws s3 ls s3://REDACTED-mail --endpoint-url https://nbg1.your-objectstorage.com from outside the container to confirm credentials and bucket name are correct.
  4. Check for DNS resolution – confirm that nbg1.your-objectstorage.com resolves correctly from within the container (nslookup nbg1.your-objectstorage.com).

The allowInvalidCerts workaround not working is worth reporting to the Stalwart maintainers, but the most actionable path right now is to ensure the container’s CA trust store is complete so TLS validation succeeds without needing to bypass it.

Reference: S3-compatible backend docs

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

8

1. (Again): I already verified connectivity to the S3 server from within the container, and I posted the output of the curl -v command, as executed from within the container, already, above.

2. This completely validates the CA bundle inside the container, as well as the propriety of the TLS connection as proffered from the server.

3. (Also again): I have already validated the credentials from outside the container and verified (via copy-and-paste, as well as literal search) that the credentials used without exactly match those used within the container.

  1. (Also again): I have already validated the DNS resolution from within the container, and I have also already tried bypassing DNS resolution by using the IP address directly.
9

The cURL output is the key finding here. Look at this line:

* ALPN: server accepted h2

Hetzner Object Store is negotiating HTTP/2 via ALPN. The rust-s3 crate that Stalwart uses (version 0.37 with tokio-rustls-tls) does not support HTTP/2 - it is an HTTP/1.1-only client. When the server accepts h2 and reqwest/hyper tries to speak HTTP/1.1 framing over what the server thinks is an H2 connection, the transport fails before any HTTP response is exchanged, which is exactly what produces a bare "reqwest: error sending request for url" error with no status code.

Also worth noting from the source: the allowInvalidCerts field you set has no effect - it is commented out in the code:

.with_path_style()
/*.set_dangereous_config(allow_invalid, allow_invalid)
.map_err(|err| {
    format!("Failed to create bucket: {err:?}")
})?*/

Workarounds to try:

  1. Place a proxy in front (e.g., nginx or Caddy) configured to forward to Hetzner over HTTP/1.1, and point Stalwart’s customEndpoint at your local proxy. Caddy in particular can do HTTP/1.1 upstream with transport http { versions 1.1 }.

  2. Check if Hetzner offers an HTTP/1.1-only endpoint - some S3-compatible providers have alternate endpoints or support disabling HTTP/2 negotiation server-side.

  3. Use a different blob store backend (Azure Blob, filesystem, etc.) if neither workaround is feasible for your setup.

This is ultimately a limitation of the underlying rust-s3 library used by Stalwart, not a configuration issue on your end. Since the source code confirms allowInvalidCerts is also currently a no-op, you may want a human maintainer to confirm whether HTTP/2 support or the allowInvalidCerts fix is on the roadmap.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

10

As with most servers, it offers both http/2 and http/1.

I further verified this by running curl --http1.1 -v ``https://nbg1.your-objectstorage.com, to force cURL to use HTTP/1, which it did quite happily and successfully.

11

You’re right to be frustrated – the curl output you posted proves TLS is working fine (valid cert, full handshake, HTTP/2, 403 as expected), so the allowInvalidCerts commentary is a red herring in your case.

Looking back at the source code that was already fetched, the critical finding is that allowInvalidCerts is commented out in the current source:

.with_path_style()
/*.set_dangereous_config(allow_invalid, allow_invalid)
.map_err(...)? */

This means the s3 crate is being initialized without any custom TLS configuration at all – it uses whatever default reqwest client the s3 crate constructs internally. The s3 crate (the Rust crate, not AWS’s SDK) builds its own reqwest client, and if there is a mismatch between what that client expects and what Hetzner’s server sends (for example, HTTP/2 vs HTTP/1.1 negotiation, or a specific TLS extension), it could fail at the transport layer with exactly “error sending request for url” and no further detail, even when curl succeeds fine.

The fact that your curl shows ALPN: server accepted h2 (HTTP/2) is worth noting – some versions of the s3 crate or reqwest have had issues with HTTP/2 on certain endpoints.

Since the root cause appears to be a code-level issue (commented-out TLS config, possible HTTP/2 incompatibility in the s3 crate) rather than a configuration problem on your end, this goes beyond what the docs cover.

This isn’t covered in the current docs. A community member or staff member will follow up.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

12

I would not be so quick to rule out a configuration error. Not only are the two suggested code-level issues red herrings (both have been separately validated to not be the problem), but I have seen other logs which may indicate a problem with outbound connections (though I haven’t been able to surface any of these recently). I had quite a number of issues with DNS resolution in my iteration set, for instance.

I can run the email reachability test just fine, so it must be able to send some traffic outbound.

Also, I definitely had this working initially, because I validated that I could send and receive mail. That’s why this is so urgent: I have already switched over to v0.16, and I am currently unable to receive mail (because writes to the blob store are failing).

Since nothing else other than configuration has changed (though I did today update to the latest stalwart release, just to make sure there wasn’t a fix therein), I strongly suspect it is a configuration problem somewhere. I just haven’t been able to figure out where, and there is not much in the way of tooling to be able to test outbound connections from within Stalwart. Everything I try from the shell in the container works, but it’s clear there is at lease some level of difference, since Stalwart can use an entirely different DNS implementation.

13

This may also be a red herring, but the version of rustls which stalwart uses is years out of date: Release 0.23.5 · rustls/rustls · GitHub

It appears all the testing of the (container image) system CA certs was for naught, because rustls is using either rustls-webpki or webpki-roots (I can’t tell which) for the root CA certs. Both of those crates appear to be up to date, though, and since the issuer for the server’s cert is Let’s Encrypt, it seems unlikely that that is a problem.

14

Even this is misleading, because the S3 client actually uses rust-s3, which has its own version of reqwest (old, v0.12) and is configured to use its own rustls (tokio-rustls-tls), and its default is to use v1 of webpki-roots… which again, seems to be fine.

15

All of these various internalisms invalidate both system-level testing (as has been done here) and internal test tooling (since different bits of code use potentially highly distinct upstream tools for what otherwise seem like very common functions: you couldn’t very usefully have an “HTTP reachability tester” intended to be able to validate all types of HTTP reachability, since all these various dependencies have their own bespoke means of making HTTP calls, performing DNS lookups, verifying TLS, etc.

16

Well that is fun: running the debian image instead of the alpine image works.

That must explain why it worked originally and then failed to work. I must have started with the debian image and switched over to the alpine image when I formalised it.

The other weird problems I was having (web UI frequently deciding to not work, pyzor not being able to download, MTA-STS policy fetch always failing, etc) have also disappeared. In fact, there are no logs of WARN or ERROR level any more.

It would appear there is some problem with the alpine image.

17

Although you found a solution, some enhancements were done to the codebase to avoid similar issues in the future:

  • rust-s3 (the S3 crate that Stalwart relies on) has not been updated yet to use the latest HTTP library release (reqwest). As a result, the older reqwest version could be using the webpki roots as a default. This does not explain all other weird problems you described, but just to tighten things, when S3 is enabled now the old request version is imported with the native certs feature enabled.
  • The allowInvalidCerts setting was being ignored: We had to temporarily remove this option some releases ago as rust-s3 did not have a way to configure it but now it does so it was wired in.
  • S3 error messages now include additional details that might have helped you debug the issue earlier.