Missing Emails after v1.0.0 migration

Issue Description

After migration from 0.16.27 to 1.0.0 emails after 10/21/26 - Present were missing in all mailboxes.

Expected Behavior

All emails migrated successfully

Actual Behavior

Missing emails without migration tool mentioning any errors other than some spamtrainingsamples missing blobs.

Reproduction Steps

  1. Follow the guide here Quick migration guide | Stalwart
  2. Set stalwart to start from /var/lib/stalwart-v1
  3. Observe missing mail on all accounts

Relevant Log Output

Migration tool output nor stalwart indicated any issues other than warnings about enterprise features being dropped and 3 entries of spamtrainingsample blobs missing.

Discourse will not let me upload the full logs, and it is too long to paste.

Stalwart Version

v1.x

Installation Method

Binary (Linux)

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

Additional Context

I was able to restore a backup from this morning. Please let me know the best way to run the migration to gather any additional logging to diagnose this issue and ensure no mail would be missing on another attempt.

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

Here’s how to run the migration again so it records what you need. I can’t say from the docs why mail after 10/21/26 went missing. That date may be when your last migration run or copy happened, but that’s only a guess.

What to collect

stalwart-migrate already writes the diagnostics you need into its state directory (for example /var/lib/stalwart-migrate):

  • logs/ holds one log file per command, named after the command, start time and PID. These are the full logs you couldn’t paste. You can attach them as a .txt or .zip, or put them in a paste service.
  • state.sqlite holds the progress, the runs, and every report row.
  • final-report.txt is the report of a completed final run.
  • prepare-report.txt is the settings conversion plan.

If the state directory from the failed attempt still exists, keep it and look at it before you start again.

Review the report first

Run these on the old state directory:

stalwart-migrate status --state /var/lib/stalwart-migrate
stalwart-migrate report --state /var/lib/stalwart-migrate --severity error
stalwart-migrate report --state /var/lib/stalwart-migrate --category skipped
stalwart-migrate report --state /var/lib/stalwart-migrate --category failed

status shows any accounts in error and the report counts by kind. Rows of kind skipped and failed mean an object couldn’t be read, converted or written. The report is meant to record every difference, so a skipped or failed row for the missing emails would explain the loss. The docs say that error rows block run --final unless you pass --allow-errors. If you used that flag, the errors would have been accepted silently, so check whether you did.

On the next attempt

  1. Run the first copy while 0.16 is still running. Use --log-level debug if you want more verbosity on the console.
  2. Stop every 0.16 node, then run run --final.
  3. Before starting 1.0, run a full comparison. It checks every account and every change log row, and the final run’s own check only samples accounts:
stalwart-migrate verify --full \
  --source /etc/stalwart/config.json \
  --target /etc/stalwart-v1/config.json \
  --state /var/lib/stalwart-migrate
  1. Run report --severity error and report --severity verify. Only start 1.0 if there are no unexplained mismatches. verify exits with status 1 when there are any.

The verify step compares the set of objects in every collection, including emails, so it should flag missing messages. Verification only gives a meaningful result when 0.16 is stopped and the final run has completed.

The three missing spam training sample blobs are probably unrelated, but they should appear in the report. Their rows will show which accounts they belong to.

If the old state directory or the report output shows skipped, failed or mismatch rows, please post those rows here. They will help a human narrow this down.

Docs: Verifying the migration | Stalwart

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

stalwart-migrate status --state /var/lib/stalwart-migrate
State database: /var/lib/stalwart-migrate/state.sqlite
tool_version           1.0.1
edition                community
source_def_hash        6262a0a7fea88a0cad21985e47d9e21e
target_def_hash        81a023434b1f436a83ae7694b8f90b9f
target_schema_version  2147483655
categories             settings,principals,mail,calendars,contacts,files,sieve,identities,push_subscriptions,spam_samples,reports,archive,spam_model,queue,tasks,telemetry
created_at             2026-10-27T15:27:58Z
source_store           directory /var/lib/stalwart
target_claim           f4cfa93558dfd72c199a2fb22086e8e2
claim_generation       7918ab08883be2099d09c86fef019233
claim_generation_pending 7918ab08883be2099d09c86fef019233
final_done             not done

Accounts: 4
  done                  4

Recent runs:
  #1     incremental  failed       started 2026-10-27T15:27:58Z  took 20s
         accounts_full=4, blob_bytes=3479097652, blobs_copied=58129, bytes_written=3498896105, emails_written=57918, errors=3, issues=203, log_rows=60917, objects_written=57965, registry_dropped=14, registry_written=3657

Reports in the latest run:
  error    skipped               3
  info     changed              11
  warning  changed               1
  warning  dropped             188

Reports in all runs:
  error    skipped               3
  info     changed              11
  warning  changed               1
  warning  dropped             188
stalwart-migrate report --state /var/lib/stalwart-migrate --severity error
run    severity category   account    object               id                   message
1      error    skipped    -          x:SpamTrainingSample 331107191746234368   blob 4f918cee2cb6ca7d561aa36c042eb60bfef375a9e58206fc22da3e316cf55d1f is missing in the source; the object was not copied
1      error    skipped    -          x:SpamTrainingSample 332034255798870016   blob 45a2735cdd97bf57a5751ae1ba15c0d55f24450702d5d467d55f7c295512d0ae is missing in the source; the object was not copied
1      error    skipped    -          x:SpamTrainingSample 332210858109800960   blob 52657d6970888a922f03ffb284efe12a0bd4aac0b94ac4c8a47db4dd5615e7a7 is missing in the source; the object was not copied
3 rows
stalwart-migrate report --state /var/lib/stalwart-migrate --category skipped
run    severity category   account    object               id                   message
1      error    skipped    -          x:SpamTrainingSample 331107191746234368   blob 4f918cee2cb6ca7d561aa36c042eb60bfef375a9e58206fc22da3e316cf55d1f is missing in the source; the object was not copied
1      error    skipped    -          x:SpamTrainingSample 332034255798870016   blob 45a2735cdd97bf57a5751ae1ba15c0d55f24450702d5d467d55f7c295512d0ae is missing in the source; the object was not copied
1      error    skipped    -          x:SpamTrainingSample 332210858109800960   blob 52657d6970888a922f03ffb284efe12a0bd4aac0b94ac4c8a47db4dd5615e7a7 is missing in the source; the object was not copied
3 rows
 stalwart-migrate report --state /var/lib/stalwart-migrate --category failed
run    severity category   account    object               id                   message
0 rows

I do not see anything that would indicate dropped/skipped/errored mail if I continued with --allow-errors since the only ones listed are about SpamTrainingSamples.

This appears to have been caused by accidentally launching 0.16.27 while pointed at /var/lib/stalwart-v1, it corrupts the datastore.

I’ve done the migration again, ensuring to only launch after updating to v1.0.0 and I appear to have all mail.