Stalwart only removes a blob from the store once it has no remaining internal references, and it tracks those references in the data store (your Postgres), never by listing the MinIO bucket. The purge task frees a blob only when its commit record has zero link records left.
I am not aware of any current or past bugs related to orphaned blobs in the blob store caused by they garbage collection process (the purge task). Stalwart will not remove a blob link from Postgres unless the blob store returns success on the deletion request. So my guess is that this could be an issue in the migration process and not Stalwart itself.
To diagnose the issue please download list_active_blobs.py and decompress_blob.py from the Stalwart repository.
I asked Claude to write a small guide (see below) explaining how to install and use these scripts. If you have any questions or issues, let me know.
Auditing the blob store for orphans
Stalwart reference-counts blobs. The raw blob bytes live in the blob store
(an S3/MinIO bucket, the filesystem, or the data store), while the references
that keep each blob alive live in the data store under the SUBSPACE_BLOB_LINK
subspace (the PostgreSQL table named k). A blob becomes garbage-collectable
once it has no surviving reference; the housekeeper then deletes it from the blob
store.
If blobs accumulate in the bucket that the database no longer references (for
example after an interrupted purge, a restore, or a backend migration), you can
find and inspect them with the two scripts in this directory:
list_active_blobs.py prints the S3 object key of
every blob that is still referenced, read straight from PostgreSQL.decompress_blob.py decodes a raw blob object back to
its original bytes (Stalwart stores blobs with a one-byte compression marker,
optionally LZ4-compressed).
Diffing the bucket listing against list_active_blobs.py yields the orphans.
What counts as an “active” blob
list_active_blobs.py reads the k table and classifies each row by length
(the subspace byte is not stored; it only selects the table). All integers are
big-endian:
| Bytes |
Meaning |
Active? |
| 32 |
Commit marker (<hash:32>) |
No, it only records that the blob exists |
| 40 |
Id link (<hash:32><id:8>) |
Yes |
| 41 |
Document link (<hash:32><account:4><collection:1><document:4>) |
Yes |
| 44 |
Temporary link (<hash:32><account:4><until:8>) |
Only while until (unix seconds) is in the future |
A blob is reported as active when it has at least one Id/Document link, or a
Temporary reservation that has not expired. The S3 object key is the 32-byte
BLAKE3 hash encoded with Stalwart’s custom base32 alphabet, optionally preceded
by the configured key_prefix.
Temporary links cover blobs that were just uploaded but not yet committed to a
message. They are intentionally treated as active so an in-flight upload is
never flagged as an orphan.
Requirements
- Python 3.8+
psycopg2 (or psycopg v3) for
the PostgreSQL connectionlz4 for decompress_blob.py- A way to talk to your bucket: the MinIO client
mc
or the AWS CLI
python3 -m venv venv
source venv/bin/activate
pip install psycopg2-binary lz4
The scripts have no Stalwart dependency; run them from anywhere that can reach
your PostgreSQL server.
Listing referenced blobs
python3 list_active_blobs.py \
--host localhost --port 5432 \
--user stalwart --password stalwart --dbname stalwart \
> active.txt
Connection settings can also come from the standard PGHOST, PGPORT,
PGUSER, PGPASSWORD, PGDATABASE environment variables, so the above is
equivalent to:
PGPASSWORD=stalwart python3 list_active_blobs.py > active.txt
Useful flags:
| Flag |
Purpose |
--prefix <str> |
The key_prefix configured on the S3 store, so the printed keys match the real object names. Omit it if no prefix is set. |
--table <name> |
The blob-link table name (default k). |
--now <unix_seconds> |
Override the clock used to expire temporary links. |
--include-expired-temporary |
Treat expired reservations as active too (rarely needed). |
Each line of output is one S3 object key, for example:
fxwmjktuburqu0qgzj3xeyr91z9pkqx0chk9gwdo79rrmjolazga
ooz7zxgwekjv3v7x7cwr2xetzw0bqskravgurchpdoxatne0frba
Finding blobs that are in MinIO but not in Stalwart
The orphans are the objects present in the bucket but absent from active.txt.
-
List everything in the bucket. With mc:
mc alias set myminio http://localhost:9000 minioadmin minioadmin
mc ls --recursive myminio/stalwart | awk '{print $NF}' | sort > bucket.txt
or with the AWS CLI:
aws s3 ls --recursive s3://stalwart/ | awk '{print $NF}' | sort > bucket.txt
If a key_prefix is configured, mc/aws print it as part of each key, so
pass the same --prefix to list_active_blobs.py to keep both sides aligned.
-
Sort the referenced set and compute the difference:
sort -o active.txt active.txt
comm -23 bucket.txt active.txt > orphans.txt
orphans.txt now lists every object in the bucket that Stalwart no longer
references.
Take a consistent snapshot. The listing and the database query are not
atomic. A blob uploaded between the two steps would look like an orphan. List
the bucket first, run the query second, and ideally point the script at a
quiesced primary or a replica. When in doubt, re-run the diff and only act on
objects that appear as orphans in two consecutive runs comfortably apart in
time.
Fetching a blob to inspect its contents
Download a single object (object keys come from orphans.txt):
mc cp myminio/stalwart/<object-key> ./blob.bin
# or
aws s3 cp s3://stalwart/<object-key> ./blob.bin
Stalwart stores blobs with a trailing one-byte compression marker, and message
blobs are usually LZ4-compressed with a little-endian u32 uncompressed-size
prefix. decompress_blob.py strips the marker and decompresses as needed:
python3 decompress_blob.py blob.bin -o blob.eml
It also reads from stdin, so you can inspect an object without writing the raw
form to disk:
mc cat myminio/stalwart/<object-key> | python3 decompress_blob.py - | less
The marker byte determines the handling:
| Last byte |
Encoding |
Action |
0xa1 |
LZ4 (u32 size prefix + lz4_flex block) |
strip the marker, then LZ4-decompress |
0x00 |
uncompressed |
strip the marker |
| anything else |
legacy blob without a marker |
emitted unchanged (a warning is printed) |
A decoded message blob is a standard RFC 5322 message: open blob.eml in any
mail client or pipe it through tools like formail/reformime to examine it.
Removing orphans
These scripts are read-only and never delete anything. Prefer Stalwart’s own
garbage collection (it runs automatically and can be triggered from the admin
tooling) over deleting objects from the bucket by hand. If you do delete
directly, keep a backup of orphans.txt and the objects until you have
confirmed the server is healthy.