IMAP FETCH of a small body section reads and decodes the whole message (potential optimization opportunity)

Issues
1

Issue Description

Not a malfunction, rather a potential optimization, noticed with Roundcube 1.7.4 as the webmail client. In my setup a small message opens in about 50 ms, while a 46 MB message with 48 photos takes about 5 s to open, almost all of it Stalwart CPU.

On the Stalwart side, every IMAP FETCH of a body section costs about the same CPU on a large message, driven by the size of the whole message rather than by what is returned. Fetching a part’s MIME header (121 bytes) or the first 2 KB of a part costs as much as fetching a whole part, and Stalwart reads the whole message from storage each time. Only BODY.PEEK[HEADER] and HEADER.FIELDS are cheap.

To open that message, Roundcube sends dozens of such FETCH commands, as it fetches the MIME headers of some image parts one at a time, so the cost per command adds up to the 5 s.

Expected Behavior

A suggestion, though I’m not sure how feasible it is: when a FETCH asks only for a small section, such as a part’s MIME header or a partial range, Stalwart might avoid reading and decoding the whole message, as it already does for BODY.PEEK[HEADER].

Actual Behavior

Measured on 0.16.24, one command at a time over loopback IMAP, 10 repetitions each, on a 46 MB message with 48 JPEG attachments. CPU is the Stalwart process’s CPU time during the command (median, /proc, 10 ms resolution); “Read” is the bytes the process read (rchar).

Command                   Returned    Stalwart CPU   Read
BODY.PEEK[HEADER]         383 bytes   < 10 ms        0
BODY.PEEK[1] (text part)  40 bytes    ~100 ms        ~46 MB
BODY.PEEK[2.MIME]         121 bytes   ~100 ms        ~46 MB
BODY.PEEK[2]<0.2048>      2 KB        ~100 ms        ~46 MB
BODY.PEEK[2]              844 KB      ~110 ms        ~46 MB
BINARY.PEEK[2]            616 KB      ~100 ms        ~46 MB
BODYSTRUCTURE             -           ~180 ms        ~46 MB

The cost follows the size of the message, not the size of the returned section.

Reproduction Steps

  1. Put a large message with many attachments in a mailbox (ours: 46 MB, 48 JPEG attachments).
  2. Open an IMAP session, SELECT the mailbox and run these commands one at a time:
    UID FETCH (BODY.PEEK[HEADER])
    UID FETCH (BODY.PEEK[2.MIME])
    UID FETCH (BODY.PEEK[2]<0.2048>)
    UID FETCH (BODYSTRUCTURE)
  3. Around each command, read the Stalwart process’s CPU time (utime + stime in /proc//stat) and rchar (/proc//io).
  4. The HEADER fetch stays under 10 ms and reads nothing; each of the others takes 100-200 ms and reads the whole message.

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

Additional Context

  • Server: AlmaLinux 10.2
  • Client: Roundcube 1.7.4 webmail on the same host, connecting over loopback IMAPS, with no proxy in between.
  • From a quick look at the 0.16.24 source, crates/imap/src/op/fetch.rs seems to load the full blob and decode the whole message for any body section other than HEADER / HEADER.FIELDS, and for BODY and BODYSTRUCTURE. I may be misreading it, and there may be good reasons for it; I mention it only in case it helps.

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Your read of the source is accurate. In crates/imap/src/op/fetch.rs, the needs_blobs check only skips loading the raw message body when the first requested section is HEADER or HEADER.FIELDS, or a plain BodySection/Binary peek is present with BODY/BODYSTRUCTURE/BINARY.SIZE. Any other body-section request (a MIME sub-part header, a partial range, a specific part number) sets needs_blobs = true, which triggers:

raw_body = self.server.blob_store()
    .get_blob(metadata.blob_hash.0.as_slice(), 0..usize::MAX)
    .await ...

That always fetches the entire blob (0..usize::MAX), regardless of how small the requested section or byte range is, and then metadata.decode_contents(...) parses the whole MIME structure again. So your measurements match the code: cost is driven by full-message size and full re-decode, not by the size of the returned section. This isn’t something covered as configurable behavior in the docs, since it’s an internal engine characteristic rather than a setting.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

Try with the v1.0.0 branch.

4

Thanks! :slight_smile: From a short look at the v1.0.0 branch, the FETCH path seems to be reworked and to cover this. We only run releases in production, so we’ll measure again once 1.0 is out.