Issue Description
Follow-up to t/570 (our report: deleted mail never freed disk space). v0.16.10 fixed that by enabling BlobDB garbage collection on the blobs column family. Thank you! But it was enabled with blob_gc_age_cutoff = 1.0 (and force_threshold = 0.5), not RocksDB’s default 0.25 (crates/store/src/backend/rocksdb/main.rs, unchanged on main and in v0.16.25):
cf_opts.set_enable_blob_gc(true);
cf_opts.set_blob_gc_age_cutoff(1.0);
cf_opts.set_blob_gc_force_threshold(0.5);
An age cutoff of 1.0 marks every blob file as old. Each compaction therefore rewrites every live blob it meets into new blob files, whether or not that file holds any garbage.
The blobs CF (t) makes that a whole-store rewrite:
- Its SST data is tiny next to its blob data (keys plus blob indexes; we see about 0.6 % on a real store). With
level_compaction_dynamic_level_bytesthe whole CF sits in one level (L6) until the SSTs passmax_bytes_for_level_base(256 MiB). - Keys are content hashes, so every L0 file spans the whole key range. Each L0→L6 compaction therefore merges all of L6.
- The CF’s write buffer is 32 MiB (default
bufferSize128 MiB / 4) withlevel0_file_num_compaction_trigger = 4. So that compaction runs after roughly every 128 MiB of new message bodies.
Net effect: about every 128 MiB of new mail, the entire message store is rewritten. Write amplification therefore grows with store size, not with the amount of garbage.
Seen in production (one deployment, 40 GB store): the RocksDB LOG shows compaction_reason: LevelL0FilesNum on CF t (4 L0 files + all 4 L6 files, about 261 MB of SST input). That job wrote num_blob_output_files: 149, total_blob_output_size: 40373387141 in 7 minutes. The same full rewrite recurs every few days, growing with the store since v0.16.10: 15 GB, 20, 24, 28.5, 33.5, 38.6, 39.7, 40.1, 40.4 GB.
Consequences for operators:
- Disk writes: the whole store is written again on every cycle (SSD wear; I/O stalls on shared or HDD storage).
- Transient disk space: old blob files are deleted only after the job finishes, so peak usage approaches 2× the store. A server sized for its mail can fill its disk mid-compaction.
- File-level replication and backup (rsync standbys, restic/borg, snapshots) see every blob file as new each cycle. Our 40 GB rewrite became about 49 GB of node-to-node traffic.
- Imports and migrations: loading N GB means about N/128 MiB compactions, each rewriting everything loaded so far, so cost grows quadratically. Loading 2.75 GiB cost 15.9 GiB of relocation writes.
- t/941 describes the same effect (“compactions rewrite ~270 MB blob files continuously”).
Expected Behavior
Garbage collection should reclaim deleted-mail space (the t/570 fix) without rewriting live, garbage-free blobs on every compaction. Write cost should stay roughly proportional to new mail plus garbage, not to the size of the store.
Actual Behavior
Every L0→L6 compaction of CF t (about every 128 MiB of new mail) relocates the entire live blob set, so steady-state write amplification scales with store size. Measurements are below.
Reproduction Steps
Standard/Default Stalwart Setup
Stalwart Version
v0.16.x
Installation Method
Docker
Database Backend
RocksDB
Blob Storage
RocksDB
Search Engine
Internal
Directory Backend
Internal
I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.
on
I have searched this support forum (open and closed topics) and confirm this is not a duplicate.
on
I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.
on