RocksDB error after 0.16.2 update

Issues
1

Issue Description

Suddenly after update from 0.16.11 to 0.16.12 I got systematic errors with RocksDB. I can no longer send or receive emails. In logs, I can see : RocksDb Error with are more or less
reason = “IO error: While appending to file: /var/lib/stalwart//002267.blob: Disk quota exceeded”, details = “Failed to lock event.”
I tried to launch tasks to reset quotas, but it refuse to create it in webui.
To me seems there is an error with path to blob file with double slash //

Expected Behavior

The server should continue to function as before

Actual Behavior

Impossible to receive or send new emails or even create new tasks

Reproduction Steps

Update to 0.16.12 and wait

Relevant Log Output

reason = “IO error: While appending to file: /var/lib/stalwart//002267.blob: Disk quota exceeded”, details = “Failed to lock event.”, causedBy = “crates/store/src/dispatch/lookup.rs:382”, details = “Failed to lock event.”, causedBy = “crates/smtp/src/queue/spool.rs:232”
reason = “IO error: While appending to file: /var/lib/stalwart//002236.blob: Disk quota exceeded”, causedBy = “crates/smtp/src/reporting/dmarc.rs:651”, details = “Failed to write DMARC report”
listenerId = “https”, localPort = 443, remoteIp = 192.168.1.109, remotePort = 55551, reason = “IO error: While appending to file: /var/lib/stalwart//002267.blob: Disk quota exceeded”, causedBy = “crates/jmap/src/registry/mapping/task.rs:348”, accountId = 1, causedBy = “x:Task/set”

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

SQL

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

This looks like a file storage issue unrelated to Stalwart’s quotas. The error is coming from RocksDB.

3

I don’t understand this answer. RocksDB is only used by stalwart executable and error appeared 30 min after update.

So option 1, update is involved in some way

Option 2 : rocksDb should’nt be used because it’s unreliable. I loose receiving email during 2 days on prod server.

How can I recover ? How to prevent it happens again ?

4

Try delete ur LOG.old.* Files. U dont need them, if its a OS Quota error u should get rid of it. After Updates they usually grow a lot.

5

Hi, thanks for advice. It wasn’t enough. I had to oversized storage to get more free space than the size of data in /var/lib/stalwart