Stalwart-migrate 1.0.0 (0.16 → 1.0): span mismatches fail run --final verification and persist across reruns

Issues
1

Issue Description

When migrating from 0.16.27 to 1.0.0 with stalwart-migrate, run --final copies and finalizes with 0 errors, but verification then reports 13 unexplained mismatches, all of type span. Because of this the migration is never marked done. Rerunning run --final with the source stopped and unchanged reports the same 13 mismatches. --allow-errors doesn’t help because these are verification mismatches, not errors.

There are no mail, account or settings mismatches; only spans are affected. This may share a root cause with #2040, where the v1 boot during migration cleans up objects that verify then reports as missing. I haven’t confirmed that.

Environment:

  • Source: Stalwart 0.16.27 (Docker stalwartlabs/stalwart:v0.16, enterprise build), RocksDB store
  • Target: new, empty RocksDB store for 1.0.0
  • Tool: stalwartlabs/stalwart-migrate:1.0.0, run as a Docker Compose service as described in the installation docs
  • Single node, 8 accounts, ~2 GB of data

Expected Behavior

One of these:

  • the incremental final run recopies the differing spans so verification passes, or
  • trace mismatches count as explained by the documented trace limitation, so they don’t block finalization.

Actual Behavior

run --final finishes the copy and finalize with status=ok and 0 errors, but verification fails with 13 span mismatches and the final run isn’t marked done. Rerunning gives the same result every time. The only way forward seems to be starting over with --exclude telemetry.

Reproduction Steps

  1. Run prepare and accept every default (all categories, recommended default policy, accept the OAuth registration change, grant new permissions). Result: 0 errors.
  2. Run run while 0.16 is still running. Result: status=ok, 0 errors, 6,634 emails, 8 accounts.
  3. Stop 0.16 and run run --final. The 30-second source watch passes, and copy/finalize returns status=ok with 0 errors. Verification then fails.
  4. Rerun run --final --yes with 0.16 still stopped. Same 13 mismatches.

Relevant Log Output

INFO migrate::orchestrator: run finished run_id=2 status=ok stats={“accounts_unchanged”:8,“errors”:0,“global_records”:16,“issues”:38,…}
INFO migrate::orchestrator::verify: verify finished run_id=3 status=failed accounts_checked=8 mismatches=13 unexplained=13
ERROR migrate::orchestrator: finalize completed but verification failed: the final run is not marked done; fix the reported mismatches and repeat the command (run --final, or rollback with the same --state)

INFO migrate::orchestrator: run finished run_id=4 status=ok stats={“accounts_unchanged”:8,“errors”:0,“global_records”:16,“issues”:38,…}
INFO migrate::orchestrator::verify: verify finished run_id=5 status=failed accounts_checked=8 mismatches=13 unexplained=13

stalwart-migrate report --severity verify:
run severity category account object id message
3 verify mismatch - span 333905986825891328 differs in the target
3 verify mismatch - span 333921088201438208 differs in the target
3 verify mismatch - span 333936189004462080 differs in the target
3 verify mismatch - span 333951287894884864 differs in the target
3 verify mismatch - span 333966388794378752 differs in the target
3 verify mismatch - span 333981490178316288 differs in the target
3 verify mismatch - span 333996588661894144 differs in the target
3 verify mismatch - span 334011689781591552 differs in the target
3 verify mismatch - span 334026791274583040 differs in the target
3 verify mismatch - span 334041892272651264 differs in the target
3 verify mismatch - span 334056991104354304 differs in the target
3 verify mismatch - span 334072091934657536 differs in the target
3 verify mismatch - span 334081053583152128 differs in the target

Stalwart Version

v1.x

Installation Method

Docker

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Your 13 mismatches are all span objects, which are stored traces and belong to the telemetry category. The docs don’t describe this exact failure, so I can’t say why those spans differ.

What the docs say

  • telemetry (stored traces and metrics) is Off by default in prepare. Accepting every category turned it on for your run. See prepare.
  • The only documented trace limitation is different from yours. 0.16 stores a trace when it ends, under an identifier taken when it started. Each run re-reads traces from six hours before the newest one it has seen. A trace stored more than six hours after it started is not copied, and verification reports it as missing. Your report says differs in the target, not missing, so that limitation probably doesn’t explain it. See troubleshooting.
  • The docs don’t say that trace mismatches are treated as explained or excluded from the finalization check.

Options

  • --include and --exclude override the chosen categories for a single run. Running stalwart-migrate run --final --exclude telemetry ... should leave traces and metrics out of that run. See migrate. I couldn’t confirm that verification then skips spans, so check the verify output after the run.
  • Excluded categories are no longer updated, and what was already copied stays in the target. Your existing copied spans would therefore remain in 1.0.
  • If you need the traces to be copied correctly, the docs don’t cover it.

I haven’t confirmed a link to #2040. A human will need to check whether the 1.0 boot cleaning up objects during migration is involved.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

This has been fixed. Please try again with stalwart-migrate 1.0.1 which is being compiled now.