The RocksDB keeps growing larger and larger and overflowing the entire server every week. Users have already deleted multiple gigabytes of mail last week but the database size keeps growing. The “purge data store” and “purge bloob store” tasks have absolutely no effect.
Expected Behavior
The “purge data store” and “purge bloob store” tasks should remove deleted data from the database and the database should compact itself regularily.
Actual Behavior
The database keeps growing around 1 GB per week even when there is just less than 200 MB of new mails per week. When deleting 2 GB of mails, the database does not shrink even after a week.
When the disk is full, the CPU usage goes to 99%.
Tenant defined storage quotas have no effect at all.
Stalwart Version
v0.16.x
Installation Method
Docker
Database Backend
RocksDB
Blob Storage
RocksDB
Search Engine
Internal
Directory Backend
Internal
I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.
on
I have searched this support forum (open and closed topics) and confirm this is not a duplicate.
on
I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.
The scheduled “purge data store” and “purge blob store” tasks (and the automatic dataCleanupSchedule / blobCleanupSchedule jobs, both daily at 02:00 by default, configured on the DataRetention object) only mark and remove logically-expired/orphaned records. They don’t force a full RocksDB compaction, so the underlying .sst/.blob files aren’t necessarily shrunk afterward, RocksDB reclaims space as part of its own background compaction cycle.
For the blob column family specifically, the server only triggers a forced compaction (with blob garbage collection) when accumulated blob garbage exceeds 256 MB and is at least 25% of the live blob data size, this is an internal threshold in the RocksDB backend, not something exposed as a setting. If your garbage never crosses that ratio (e.g. because total blob volume is large), disk space from deleted blobs won’t be reclaimed by that path, and the main data column families rely entirely on RocksDB’s normal compaction behavior, which can lag well behind deletes under sustained write load, explaing why deleting several GB of mail doesn’t immediately shrink the store.
That said, continual growth well beyond new mail volume, “purge” tasks having no visible effect at all, and tenant storage quotas not being enforced are not expected/documented behaviors, this looks like it needs a deeper look at your specific setup (store layout, column family sizes, RocksDB properties) rather than something addressed in current docs.
This isn’t covered in the current docs. A community member or staff member will follow up.
This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.
This shows the disk space. It keeps growing and every time the disk usage is about 93% then RocksDB turns crazy and fills up the disk to 100% and the CPU usage goes to 100% as well. Only solution is to add 1 GB of virtual disk. Those events have been 4 times in the last 2 weeks.
Deleting mail doesn’t free space right away. Deleted messages stay in Trash until they’re expunged (30 days by default), the blob purge only removes blobs nothing references anymore, and RocksDB gives disk space back when it compacts, not at delete time. A week after deleting 2 GB I wouldn’t expect the files to have shrunk yet.
1 GB a week against 200 MB of new mail is still more than that explains. With blobs in RocksDB as well, the usual extra growth comes from tracing, metrics and report history kept in the same store, archived deleted items, or a large queue. To narrow it down I’d need your data retention settings (DataRetention | Stalwart), whether the tracing and metrics stores point at RocksDB, and the output of du -sh on the RocksDB directory taken a few days apart. For the tenant quotas: which limit did you set, and what usage does the tenant show against it?