Hi — while capacity-testing Stalwart v0.16.5 (RocksDB backend, default settings, single node) we found that deleting mail effectively never returns disk space on a mostly-idle server, and traced it to the RocksDB BlobDB configuration.
What we measured
- Seeded a mailbox with 40,000 messages (~8.7 GB of blob data, realistic text+attachment mix) over IMAP.
- Expunged all of them (
STORE +FLAGS \Deleted+EXPUNGE, verifiedEXISTS 0). - Sampled the data directory every 15 minutes for 11.5 hours (idle otherwise).
Result: the blob store stayed byte-identical the whole time — 138 .blob files, 8,705 MB, zero bytes reclaimed. The WAL was also static, i.e. the periodic purge had not deleted the blob keys in that window either.
Why
crates/store/src/backend/rocksdb/main.rs enables BlobDB on the blobs CF (set_enable_blob_files(true), set_min_blob_size(...)) but does not enable blob garbage collection. With GC off, a .blob file is deleted only when no SST references it — and since blob keys are content-hash distributed, real-world files mix live and deleted blobs and are never rewritten. There’s no fallback either: purge_store issues per-key deletes only, and the backend never calls manual compaction, so the only idle-store backstop is RocksDB’s ~30-day stale-file horizon — which still can’t shrink partially-live blob files.
Operationally this means mailbox deletions (tenant offboarding, retention enforcement) look like a silent disk leak.
This refines the answer in GH discussion #335 (“it is up to the database when this data will actually be removed … usually occurs when the database performs a compaction”): for the blobs CF that only holds for blob files whose every message was deleted — compaction never rewrites partially-live blob files while GC is off, and as noted there, manual compaction isn’t triggerable on RocksDB either. So for mixed live/deleted data the space is effectively never returned.
Proposed fix (1 line)
In crates/store/src/backend/rocksdb/main.rs, blobs CF setup:
// Blobs
let mut cf_opts = Options::default();
cf_opts.set_enable_blob_files(true);
cf_opts.set_min_blob_size(config.blob_size);
// Enable BlobDB garbage collection: without it, a .blob file is only
// deleted once *no* SST references it, so blob files holding a mix of
// live and deleted messages are never reclaimed — deleting mail frees
// no disk space on mostly-idle stores. With GC enabled, compactions
// rewrite live values out of the oldest blob files (default age
// cutoff 0.25), allowing space from deleted messages to be returned.
cf_opts.set_enable_blob_gc(true);
cfs.push(ColumnFamilyDescriptor::new(CF_BLOBS, cf_opts));
RocksDB’s defaults (age_cutoff 0.25, force_threshold 1.0) then let normal compactions rewrite live values out of the oldest blob files and return the space, with write amplification bounded by the age cutoff (only the oldest quarter of blob files are GC candidates per compaction). This is the standard recommended configuration when running BlobDB with deletions. The set_enable_blob_gc API is available in the rocksdb = "0.24" crate you already pin.
Ready-to-cherry-pick commit (GitHub PRs are collaborator-restricted, so posting here instead):
Happy to sign the FLA if you want to take the commit as-is, and to test a build against our measurement setup — or to add a config knob next to blobSize if you’d prefer it tunable.
Stalwart Version
v0.16.x
Installation Method
Docker
Database Backend
RocksDB
Blob Storage
RocksDB
Search Engine
Internal
Directory Backend
Internal
I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.
on
I have searched this support forum (open and closed topics) and confirm this is not a duplicate.
on
I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.
on