All email lost after upgrading to stalwart 0.16

Issues
1

Issue Description

I upgraded from stalwart 0.15.5 to 0.16.5 following the upgrade guide. All my accounts and domains successfully migrated, however all existing email is lost.

Expected Behavior

Existing email should still be available.

Actual Behavior

All existing emails are lost.

Reproduction Steps

I followed the upgrade guide. stalwart-cli apply failed the first time, because DKIMSignatures could not be imported. The secret referenced a file %{file:/opt/stalwart/etc/dkim/example.com.ed25519-2025-06.key}%, which failed validation. I removed the DKIMSignatures from the import, retried the import, but it failed again, because some parts were already applied. I followed the steps in Recovering from a partial apply. Afterwards, the import was successful. However, I fear that

stalwart-cli query Account --json | jq -r '.id' \
    | stalwart-cli delete Account --stdin

might still have removed all email?

Maybe step 4 in the docker upgrade guide failed, because I did not see any log scrolling. It just showed that the HTTP endpoint is listening immediately.

Wait until the logs stop scrolling and settle on the message indicating the HTTP endpoint is listening.

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

PostgreSQL

Blob Storage

PostgreSQL

Search Engine

PostgreSQL

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Based on the information gathered, here is an assessment of the situation:

The stalwart-cli delete Account --stdin command used during “Recovering from a partial apply” very likely deleted account metadata – but whether it deleted email blobs depends on what was actually stored and how the deletion propagates to blob storage.

Since you are using PostgreSQL for both data and blob storage, all email content lives in that same PostgreSQL database. The critical question is: was the blob data (the actual email content) deleted along with the Account objects, or only the account index/metadata?

Here is what we can say with confidence:

  1. The delete Account step in the recovery procedure is intended to clean up account records so the import can re-create them cleanly. If it also cascaded to email blobs, the emails would be gone from the database.

  2. Step 4 of the Docker upgrade guide (“wait until logs settle”) is important because it is the data migration step that copies existing email content into the new format. If the container was not given enough time to complete that migration before you proceeded, the emails may never have been migrated at all – meaning they were not present to be imported.

  3. If you have a PostgreSQL backup from before the upgrade, that is your best recovery path. Restore the old database, and attempt the migration again, this time ensuring step 4 (migration) runs to full completion before proceeding.

Since this is a potential data loss situation tied to the specific 0.16 upgrade procedure, and the authoritative source is the upgrade guide itself (stalwart/UPGRADING/v0_16.md at main · stalwartlabs/stalwart · GitHub), a human staff member should review your specific case – especially whether the migration step completed and whether the delete cascade affected blobs.

Immediate recommendations:

  • Do not write any new data to the current PostgreSQL instance if you want to attempt recovery.
  • Check whether you have a pre-upgrade database backup.
  • Inspect the PostgreSQL blob tables directly to see if email content rows still exist (they may be orphaned but recoverable).

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong. If you found it useful, mention @helpbot in any reply to ask follow-up questions and the bot will respond again.

4

I did not notice data loss immeditately due to the issue mentioned here: User supplied passphrases for app passwords in 0.16 missing (was available in 0.15) - #4 by P4sca1 .

So I already made a bunch of configuration changes manually after the migration. e.g. I setup automatic TLS, DNS and DKIM management and recreated app passwords. Also, new email arrived in the meantime. Therefore, I cannot just restart the migration from a backup from scratch. I do have a database dump from before the migration. How can I restore old email without loosing email that arrived after the migration and all the configuration I’ve done? @helpbot

My database monitoring revelealed that database size went down from 5.5GB to 5.2GB. So I think email is still stored somewhere in the database but maybe not migrated? Is there any way to retrigger the migration to the new data format?

5

The upgrade guide’s “Step 2: Back up the database” section is the key reference here. From what the upgrade docs say explicitly:

The database layout used to store user data (emails, calendars, contacts, files, blobs, search indexes) is NOT affected by this change. […] No user mail is touched.

The recovery-mode startup wipes specific tables (s, d, r, h, b, g, j, f, u, o, x, w) – but not the email message/blob tables. The delete Account --stdin command in the partial-apply recovery section also only deletes directory/account records, not blob data.

The good news: Your emails are almost certainly still in the PostgreSQL database, just orphaned (no account directory entries pointing to them). The 5.5GB → 5.2GB drop is consistent with the metadata tables being wiped, not blob/message data.

What you need to do is re-run stalwart-cli apply to restore the account directory records that link users back to their mailboxes. Since you’ve already done configuration work since the migration, here is the recommended approach:

  1. Do NOT restore the full database backup – that would overwrite your new configuration.
  2. Your export.json from the migration still contains the account/domain records from the old deployment. Re-apply just that file while the server is running normally: the accounts will be recreated pointing at the same underlying blob data. You can do this with the same stalwart-cli apply --file export.json command used during migration.
  3. Once accounts are re-created, trigger “Recalculate disk quotas” from the WebUI’s Tasks panel.

The upgrade guide confirms accounts use a stable client-ID scheme (restore-{old_numeric_id}) during migration, so re-applying export.json should recreate the same account identities. New mail that arrived since the migration and new configuration (DKIM, TLS, app passwords) are stored in separate tables from the account directory records and should not be affected by re-creating the account objects.

That said, since this is a potential data recovery situation with a production deployment, a human staff member should review and confirm before you proceed.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong. If you found it useful, mention @helpbot in any reply to ask follow-up questions and the bot will respond again.

6

@helpbot Like I said, I already re-applied export.json and all accounts were successfully recreated. However, all the accounts have empty inboxes (except new emails that arrived since the migration).

7

@mdecimus Is there any way to recover the emails? I think they are still in the database according to database size.