Upgrade to 0.16

Questions
1

Your question

Accidental :latest upgrade, database in mixed state between v0.13 and v0.16
Running Stalwart via Docker on Coolify using :latest tag. Was running v0.13 (confirmed via docker image history). At some point Coolify pulled a newer image — possibly v0.15 or v0.16 — and ran it briefly. Server was working fine receiving emails. Later noticed some features missing and started the v0.16 migration procedure.
Current state:

v0.16 panics on start: “Migration aborted: You must first upgrade to version 0.15”
v0.15 fails with: “Data corruption detected” on key AAsEWI1RPqAAAQ==
Database is RocksDB at /var/lib/stalwart-all/data
v0.16 recovery mode works and I have successfully applied export.json (domains, accounts, DKIM created)
All email data appears intact
No backup available (60GB on dedicated server with no spare disk space)

Is there a way to force v0.16 to skip the version check and proceed with migration from v0.13 data directly? Or another path forward without a backup?

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Here is what the source code reveals about your situation and the available path forward.

What’s happening

The v0.16 migration code checks the stored database schema version number and enforces a strict upgrade path. Schema versions 0-4 (v0.13/v0.14) cannot be migrated directly by v0.16 – it hard-aborts and demands you go through v0.15 first. Schema version 5 (v0.15) requires the multi-step process described in the upgrade guide. The None (missing version key) case also hits the same abort unless the database looks like a fresh install.

The problem is you’re caught in between: v0.15 sees “data corruption” on that key, and v0.16 refuses to proceed.

The one escape hatch in the code

The source (crates/migration/src/lib.rs, line 30) contains an undocumented environment variable bypass:

if !std::env::var("DANGER_FORCE_MIGRATE").is_ok_and(|v| v == "1") {
    return Ok(());
}

This variable is only checked when the schema version already equals DATABASE_SCHEMA_VERSION (i.e., it forces a re-run of migration even if already at the current version). It does not bypass the Some(0..=4) or None abort paths that you are hitting. So there is no env-var way to skip the version check for a v0.13 database directly.

Your realistic options

  1. Find out what schema version is actually stored. The corruption error from v0.15 is on a specific key (AAsEWI1RPqAAAQ==). The schema version is stored on a different key (subspace SUBSPACE_PROPERTY, key [0u8]). It’s possible the schema version key itself is missing or corrupted. If you can use a RocksDB tool (e.g. ldb or sst_dump) to inspect that key directly, you’d know exactly which branch of the code you’re in.

  2. Use v0.16 recovery mode + re-import path – which you’ve already partially done. Since you confirmed that export.json was applied successfully in recovery mode (domains, accounts, DKIM), the question is whether the email data (blobs) can be recovered. The migration docs cover export/import for backend migration; if the blobs are intact on disk, a staff member may know whether they can be re-associated.

  3. Wait for a human follow-up. This is an edge case that requires someone who knows the exact RocksDB key layout and whether the v0.15 “data corruption” on that one key can be worked around (e.g. by deleting or resetting that specific corrupted key to unblock v0.15, which then unblocks v0.16).

The official upgrade guide explicitly warns against using :latest in Docker for exactly this reason, and recommends pinning to a major version tag: Upgrading docs.

A human staff member should follow up – this situation needs someone who can look at the raw RocksDB state with you.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

Update with new findings:

After further investigation, we decoded the value of the problematic key AAsEWI1RPqAAAQ== and it contains:

Stalwart Web Interface
https://github.com/stalwartlabs/webui/releases/latest/download/webui.zip
/admin
/account

This is a v0.16 WebUI bundle record that was written when v0.16 briefly ran in bootstrap mode. It is not real data corruption — this is a v0.16 format record that does not belong in a v0.13 database.

The database is otherwise a clean v0.13 schema. The correct upgrade path is v0.13 → v0.15 → v0.16. However v0.15 is blocked from opening the database because of this single v0.16 format key.

Question: Is it safe to delete this one key AAsEWI1RPqAAAQ== from RocksDB so that the proper migration path can proceed? If yes, what is the recommended way to do this — the system ldb tool is too old to read the database format, so we would likely need a Rust program using the same rust-rocksdb crate.

4

Update with detailed findings:

After carefully tracing through our actions, here is the exact sequence of what happened:

Background:

  • Was running v0.13 binary but database had very likely been auto-migrated to v0.15 schema through multiple restarts/redeploys over months. The upgrade docs for v0.13->v0.14 and v0.14->v0.15 both state that schema migration happens automatically on startup, and the migration script confirmed this by finding 3595 v0.15 settings keys in the database.

What went wrong during migration attempt:

  1. Migration script ran successfully against live server — generated config.json and export.json :white_check_mark:

  2. Stopped the server :white_check_mark:

  3. First recovery container attempt — used old cached v0.13 image by mistake, did nothing to database :white_check_mark:

  4. Pulled fresh v0.16 image :white_check_mark:

  5. Started recovery container without config.json mounted — ran in bootstrap mode instead of recovery mode, did NOT detect old data :cross_mark:

  6. While in bootstrap mode, attempted to complete bootstrap by pointing it at the real RocksDB database via stalwart-cli update Bootstrap --field dataStore...

  7. This caused v0.16 bootstrap mode to open the real database and write a WebUI bundle record into it :cross_mark:

Result: The database has what should be a clean v0.15 schema + one stray v0.16 format WebUI record (key AAsEWI1RPqAAAQ==), which decodes to:

Stalwart Web Interface
https://github.com/stalwartlabs/webui/releases/latest/download/webui.zip
/admin
/account

Current state:

  • v0.16 normal startup panics: “Migration aborted: You must first upgrade to version 0.15”

  • v0.16 recovery mode also panics with same error before port 8080 actually binds — recovery mode never gets to do its wipe/migration step

  • The underlying v0.15 data appears intact — only this one stray v0.16 key is causing the schema version check to fail

  • System ldb tool is too old to read Stalwart’s RocksDB format

Question: How can we safely delete key AAsEWI1RPqAAAQ== from RocksDB to unblock the migration? Would a small Rust program using the same rust-rocksdb crate be the right approach?

Sorry for ai generated steps, but this should be easier to explain than i personally ever could

5

Ok, Ai wrote a script that tries to launch a 0.15 version. sees if it complains about any db values, then deletes them using rust. This deleted a few values and and now i can run 0.15 stalwart. all other data seems to be persisted correctly. Thanks for now.