TRACE Blob read operation (store.blob-read) key = base64:c/m2LB7kK3VGdfd5h2EgTK7P WARN Blob missing marker (store.blob-missing-marker) key = base64:c/m2LB7kK3VGdfd5h2EgTK7PfiJq8z1+K6cjjrYEmfQ=fiJq8z1+K6cjjrYEmfQ=, elapsed = 20ms, size = 3545

Questions
1

Your question

After upgrading from v0.15 to v0.16 (same blob store, no data migration), every read of a
v0.15-era uncompressed blob fires a store.blob-missing-marker WARN. The warning is a
false positive: get_blob still returns the blob data (Ok(Some(data))), so reindex /
Email/get / IMAP FETCH keep working — there is no data loss. But the WARN floods the logs
(most visible during a post-upgrade reindexAccounts) and is actively misleading: it looks
exactly like “blobs are missing from S3”, when in fact the objects are present and valid.

2

SO hit the same “problem”, after digging into the 0.16 code and checking diff with 0.15 it seems that the put blob method has changed, and essentially writes an extra 0 byte in 0.16 vs 0.15. In theory this means that if any of the blobs have a 0 byte (NO CLUE IF THIS IS POSSIBLE), it will be eaten.

0.15 put blob: https://github.com/stalwartlabs/stalwart/blob/0398a9d28e13a39514483542c4ba92767ec4bf5b/crates/store/src/dispatch/blob.rs#L101-L108

        let data: Cow<[u8]> = match self.compression {
            CompressionAlgo::None => data.into(),
            CompressionAlgo::Lz4 => {
                let mut compressed = lz4_flex::compress_prepend_size(data);
                compressed.push(CompressionAlgo::Lz4.marker());
                compressed.into()
            }
        };

0.16 put blob: stalwart/crates/store/src/dispatch/blob.rs at d2d16a0186decd8687bf0c238c7f143846ae2f0c · stalwartlabs/stalwart · GitHub

      let data = match compression {
            CompressionAlgo::None => {
                let mut uncompressed = Vec::with_capacity(data.len() + 1);
                uncompressed.extend_from_slice(data);
                uncompressed.push(NONE_MARKER);
                uncompressed
            }
            CompressionAlgo::Lz4 => {
                let mut compressed =
                    vec![
                        LZ4_MARKER;
                        lz4_flex::block::get_maximum_output_size(data.len()) + U32_LEN + 1
                    ];

                // Compress the data
                let compressed_len =
                    lz4_flex::compress_into(data, &mut compressed[U32_LEN..]).unwrap();

                // Prepend the length of the uncompressed data
                compressed[..U32_LEN].copy_from_slice(&(data.len() as u32).to_le_bytes());

                // Truncate to the actual size
                compressed.truncate(compressed_len + U32_LEN + 1);
                compressed
            }
        };

@stalwart, I would argue this is either a bug, or a missing migration step. What should be done when you have many many many blobs pre 0.16 that are uncompressed and are missing this additional marker null byte?

Edit: in theory I think we could do a Database Migration | Stalwart export/import to “fix” this, but it feels rather heavy…

3

You’re right; indeed, the blob in version 0.16 has extra 0x00 bytes. My blob is stored in S3.

image
image799×170 19.7 KB

4

The warning itself is harmless, that branch returns the data exactly as stored. In 0.16 we append a single marker byte to each blob to record how it was encoded, 0.15 didn’t write one for uncompressed blobs, and the reader notices it’s missing and carries on. Only stores with compression disabled are affected; LZ4 blobs have the same layout in both versions.

The warning is the good case. We decide what to do by looking at the last byte, so a legacy blob whose real final byte happens to be 0x00 gets treated as marked and comes back one byte short, and one that happens to end in 0xa1 gets sent to the LZ4 decoder and fails the read. Neither of those logs anything. It works out to roughly one in 256 blobs for each case, and it’s mostly harmless for stored messages, which end in text, but not for arbitrary uploads or file storage.

To quiet the log now, exclude the store.blob-missing-marker event on that tracer by adding it to the events map with the policy set to exclude. There’s no migration today; an export and re-import would rewrite the blobs correctly but that’s a heavy way to fix a warning. The real fix is on our side: either re-mark the legacy blobs once or record the format per store instead of inferring it from the last byte.