Frequent database corruptions (rocksdb)

Issues
1

Issue Description

After upgrading to 0.16 I’ve been hit with regular database corruptions. Theres a seemingly 50/50 chance that after stopping/starting stalwart in systemd it won’t start back up again and produce the following error:

Startup failed: Failed to write node id to store: RocksDB error (store.rocksdb-error): reason = Corruption: Merge operator failed

The rocksdb-ldb restore command suggested as a way to fix this doesn’t seem to do anything, it claims to see and resolve issues but if you run it again it claims to have found and resolved exactly the same issues, and no matter how many times it runs stalwart continues to report data corruption.

Expected Behavior

The database to not corrupt whenever you stop stalwart

Actual Behavior

The rocksdb store frequently (but not consistently) corrupting

Reproduction Steps

Run stalwart with rocksdb
Restart it a few times

Relevant Log Output

2026-06-29T10:44:55Z ERROR RocksDB error (store.rocksdb-error) listenerId = "imap", localPort = 143, remoteIp = 2403:5813:2408:1:6395:6b33:8de:f7dc, remotePort = 42816, reason = "Corruption: Merge operator failed", causedBy = "crates/store/src/dispatch/store.rs:133", causedBy = "crates/common/src/storage/quota.rs:31", accountId = 1, id = "96", details = "Internal Server Error", code = "CONTACTADMIN", causedBy = "crates/imap/src/op/quota.rs:176"
2026-06-29T10:49:54Z INFO Task scheduled for future execution (task-manager.task-scheduled) due = 2026-06-29T10:54:53Z, id = "calculateMetrics"
2026-06-29T10:49:54Z INFO Metrics collected (telemetry.metrics-collected) elapsed = 4ms
2026-06-29T10:54:39Z ERROR RocksDB error (store.rocksdb-error) listenerId = "imaps", localPort = 993, remoteIp = 2403:5813:2408:1:6395:6b33:8de:f7dc, remotePort = 46538, reason = "Corruption: Merge operator failed", causedBy = "crates/store/src/dispatch/store.rs:133", causedBy = "crates/common/src/storage/quota.rs:31", accountId = 1, id = "31", details = "Internal Server Error", code = "CONTACTADMIN", causedBy = "crates/imap/src/op/quota.rs:176"
2026-06-29T10:54:54Z INFO Task scheduled for future execution (task-manager.task-scheduled) due = 2026-06-29T10:59:53Z, id = "calculateMetrics"
2026-06-29T10:54:54Z INFO Metrics collected (telemetry.metrics-collected) elapsed = 4ms
2026-06-29T10:55:06Z ERROR RocksDB error (store.rocksdb-error) listenerId = "imap", localPort = 143, remoteIp = 2403:5813:2408:1:6395:6b33:8de:f7dc, remotePort = 42816, reason = "Corruption: Merge operator failed", causedBy = "crates/store/src/dispatch/store.rs:133", causedBy = "crates/common/src/storage/quota.rs:31", accountId = 1, id = "100", details = "Internal Server Error", code = "CONTACTADMIN", causedBy = "crates/imap/src/op/quota.rs:176"
2026-06-29T10:58:40Z INFO Shutting down Stalwart Server (server.shutdown) causedBy = "SIGINT"

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

The docs don’t cover a repair procedure. Let me look for any repair/recovery tooling in the codebase.

3

I think it may have been related to Cannot recalculate disk quotas
I’ve completed the steps outlined there and it seems to be okay so far :crossed_fingers:

4

Just an update for anyone in the future reading this, I just had another database corruption, this one far more insidious as stalwart reported no errors and continued to sync email however would not send or receive any, quietly dropping them. The webadmin would also return code 500 when attempting to sign in.

I had to do a complete reset and configure a new instance from scratch because rocksdb was seemingly unrecoverable in this state, and it had seemingly gone on long enough that all my backups presented the same issue. I also didn’t do anything abnormal to trigger this, besides possibly attempting to sync my obsidian vault which contained one single markdown file into webdav.

If anyone else runs into corruptions like this I would appreciate hearing what you did so I can make sure it doesn’t happen again.