Issue Description
I am migrating a 4-node cluster of email servers from Stalwart v0.13.4 to v0.16.7 using the official migration guide (including the proxy and Vandelay). Because v0.13.4 does not expose the urn:ietf:params:jmap:principals JMAP capability that Vandelay requires for account name resolution, I first upgraded the source cluster from v0.13.4 to v0.15.5 (following the upgrade guide) to gain that capability before attempting the Vandelay export.
After upgrading to v0.15.5 and running Vandelay’s JMAP import against the source cluster, I received “email blob missing” warnings for the vast majority of emails in the first account I attempted (3,290 out of 4,097). The 807 emails that succeeded were all received on or after 2025-09-03. All emails received before that date failed, but are still visible in Roundcube Webmail and Mail.app on macOS.
After extensive investigation I determined the following:
-
The S3 blob store contains 122,437 objects. The actual email content for the failing emails is present in the bucket under keys derived from their BLAKE3 hashes using Stalwart’s custom Base32 encoding.
-
The MySQL data store k table has only 9,857 entries across all accounts. Encoding these entries using Stalwart’s Base32Writer confirms they all correspond to objects in the S3 bucket.
-
The JMAP blob IDs for the failing emails decode (via BlobId::from_base32) to hashes that do NOT have corresponding BlobOp::Commit entries in the k table.
-
Cross-referencing email metadata entries (EmailField::Metadata, field byte 71) in the p table against the k table and the S3 bucket across all accounts on the cluster reveals:
- 360,361 total email metadata entries
- 6,363 already have valid k table entries (OK)
- 54,166 have blobs present in S3 but NO corresponding k table entry
- 19,924 have no k table entry AND no blob in S3 (these appear to be unrecoverable)
-
The emails missing k table entries are all older emails that predate when blob link tracking was introduced in the codebase. They have valid blob_hash values in their MessageMetadata rkyv archives and their content exists in S3, but the BlobOp::Commit and BlobOp::Link entries were never created for them, not by the original ingestion, and not by the v0.15.5 migration.
-
The v0.15.5 migration’s migrate_emails_v014 correctly carries the blob_hash from LegacyMessageMetadata into the new MessageMetadata struct, but does not create the corresponding k table entries. The migrate_blobs_v014 function only re-migrates existing blob link entries from the old format and cannot create entries that never existed.
Expected Behavior
After upgrading from v0.13.4 to v0.15.5, all emails whose blob content is present in the configured blob store should be accessible via JMAP blob download, and Vandelay should be able to import them successfully into an archive file.
Actual Behavior
After upgrading to v0.15.5, JMAP blob downloads return HTTP 404 for all emails that lack k table entries, even though their content exists in S3. Specifically:
- GET /jmap/download/{accountId}/{blobId}/{name} returns
{"status":404,"title":"Not Found","detail":"The requested resource does not exist on this server."} - Vandelay reports
warning: Email {id} skipped: malformed jmap response: email blob missingfor each affected email - The JMAP Email/get method returns valid
blobIdvalues for these emails (the blob IDs are correctly constructed from the stored hashes), but downloading those blobs fails - Emails ARE visible and browsable in IMAP and Roundcube (which use a different code path), giving the false impression that the mailbox is intact
Reproduction Steps
- Run Stalwart from v0.11.x, accumulating emails over an extended period (months/years). Configure it with MySQL as the data store and Wasabi S3 as the blob store.
- Upgrade through v0.13.4, continuing to receive mail
- Install Vandelay and set up a v0.16.7 deployment with the migration proxy per the official migration guide
- Upgrade source cluster to v0.15.5 to gain the urn:ietf:params:jmap:principals support needed by Vandelay
- Run Vandelay import against the v0.15.5 source:
vandelay import jmap \ --url https://mail.example.com \ --auth-basic "[email protected]" \ --account-name "[email protected]" \ user.sqlite - Observe “email blob missing” warnings for all emails predating blob link tracking introduction
- Confirm via JMAP Email/get that affected emails have valid blobId values
- Confirm via direct S3 listing that blob objects exist in the bucket
- Confirm via the k table that no BlobOp::Commit entries exist for those blob hashes
Relevant Log Output
From Vandelay:
...
warning: Email mwyaaadl2 skipped: malformed jmap response: email blob missing
warning: Email mw2aaadl0 skipped: malformed jmap response: email blob missing
warning: Email mxeaaadl3 skipped: malformed jmap response: email blob missing
warning: Email mxiaaadma skipped: malformed jmap response: email blob missing
warning: Email mxmaaadmb skipped: malformed jmap response: email blob missing
warning: Email mzaaaadmu skipped: malformed jmap response: email blob missing
warning: Email m1eaaadn0 skipped: malformed jmap response: email blob missing
warning: Email m1iaaadn1 skipped: malformed jmap response: email blob missing
warning: Email m3eaaadof skipped: malformed jmap response: email blob missing
warning: Email m3iaaadog skipped: malformed jmap response: email blob missing
warning: Email m3maaadoh skipped: malformed jmap response: email blob missing
warning: Email m3qaaadoi skipped: malformed jmap response: email blob missing
warning: Email m3uaaadoj skipped: malformed jmap response: email blob missing
warning: Email m3yaaadok skipped: malformed jmap response: email blob missing
warning: Email m32aaadol skipped: malformed jmap response: email blob missing
warning: Email naaaaadom skipped: malformed jmap response: email blob missing
warning: Email naeaaadon skipped: malformed jmap response: email blob missing
warning: Email naiaaadoo skipped: malformed jmap response: email blob missing
warning: Email namaaadop skipped: malformed jmap response: email blob missing
import: Email done (fetched=807 deleted=0 skipped=0 failed=6580)
import: ContactCard ...
import: ContactCard done (fetched=0 deleted=0 skipped=0 failed=0)
import: CalendarEvent ...
import: CalendarEvent done (fetched=0 deleted=0 skipped=0 failed=0)
Mailbox: created=0 fetched=7 updated=0 deleted=0 skipped=0 failed=0
AddressBook: created=0 fetched=1 updated=0 deleted=0 skipped=0 failed=0
Calendar: created=0 fetched=1 updated=0 deleted=0 skipped=0 failed=0
FileNode: created=0 fetched=0 updated=0 deleted=0 skipped=0 failed=0
Identity: created=0 fetched=1 updated=0 deleted=0 skipped=0 failed=0
SieveScript: created=0 fetched=0 updated=0 deleted=0 skipped=0 failed=0
ParticipantIdentity: created=0 fetched=1 updated=0 deleted=0 skipped=0 failed=0
Email: created=0 fetched=807 updated=0 deleted=0 skipped=0 failed=6580
ContactCard: created=0 fetched=0 updated=0 deleted=0 skipped=0 failed=0
CalendarEvent: created=0 fetched=0 updated=0 deleted=0 skipped=0 failed=0
error: some objects failed; the archive is consistent and resumable
Stalwart Version
v0.15.x
Installation Method
Binary (Linux)
Database Backend
MySQL
Blob Storage
S3-compatible
Search Engine
MySQL
Directory Backend
Internal
Additional Context
From its inception, Nginx has been configured as a reverse proxy behind the 4-node Stalwart cluster to handle HTTP/HTTPS traffic. The Linode NodeBalancer (load balancing software) has also been configured to handle IMAPS/SMTP/Submission traffic, directing it to one of the four backend Stalwart servers.
To migrate to v0.16, a new deployment was installed via the binary method on a separate Linode that sits outside the cluster (but in the same datacenter). The migration proxy was installed on a second separate Linode that can communicate with both the new deployment and the source over the private network. Vandelay was installed on a MacBook with remote access to all servers.
I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.
on
I have searched this support forum (open and closed topics) and confirm this is not a duplicate.
on
I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.
on