Message-cache warming slows down auth for large mailboxes

Issues
1

Issue Description

We migrated mailboxes that have 10-100k emails in them. I connected via openssl and logged in and it just hangs. After a few minutes, I finally get the response.

openssl s_client -connect mail.example.com:993 -crlf

…
OK [CAPABILITY IMAP4rev2 IMAP4rev1 ENABLE SASL-IR LITERAL+ ID UTF8=ACCEPT AUTH=PLAIN AUTH=OAUTHBEARER AUTH=XOAUTH2] Stalwart IMAP4rev2 at your service.
a CAPABILITY
CAPABILITY IMAP4rev2 IMAP4rev1 ENABLE SASL-IR LITERAL+ ID UTF8=ACCEPT AUTH=PLAIN AUTH=OAUTHBEARER AUTH=XOAUTH2
a OK CAPABILITY completed
b LOGIN [email protected] password
(hangs for minutes)
b OK [CAPABILITY IMAP4rev2 IMAP4rev1 ENABLE SASL-IR LITERAL+ ID UTF8=ACCEPT JMAPACCESS IDLE NAMESPACE CHILDREN MULTIAPPEND BINARY UNSELECT ACL UIDPLUS ESEARCH WITHIN SEARCHRES SORT THREAD=REFERENCES LIST-EXTENDED LIST-STATUS ESORT SORT=DISPLAY SPECIAL-USE CREATE-SPECIAL-USE MOVE CONDSTORE QRESYNC UNAUTHENTICATE STATUS=SIZE OBJECTID+ PREVIEW RIGHTS=texk QUOTA QUOTA=RES-STORAGE] Authentication successful

Expected Behavior

The login should take less than 3 seconds.

Actual Behavior

The login takes minutes.

Reproduction Steps

  1. Have an inbox with 100k emails in it.
  2. Try logging in.

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

PostgreSQL

Blob Storage

S3-compatible

Search Engine

Elasticsearch

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Please set the log level to trace and check the logs to see what is taking all that time, data store and blob operations include the number of milliseconds they take to fetch.
Also check your PostgreSQL for any slow query reports.

3

I believe I found a likely cause for this, and it does not seem to be PostgreSQL or S3 latency itself.

We are running Stalwart v0.15.5 with PostgreSQL metadata storage and S3 blob storage. I also checked the current v0.16.21 source, where
the relevant logic appears unchanged.

During every successful IMAP login, SessionData::new() calls fetch_account_mailboxes(). That function builds the state for every mailbox
and calculates:

  • total_messages
  • total_unseen
  • total_deleted
  • uid_next
  • whether the mailbox has children

For each mailbox, the three message counters call in_mailbox*().count():

The in_mailbox*() implementations iterate over the complete emails.items cache and filter it for the requested mailbox:

This means that even with a warm message cache, opening a new IMAP session appears to perform roughly O(number_of_mailboxes *
number_of_messages) work. There are three full message scans per mailbox, plus mailbox child scans and one uid_next store lookup per
mailbox.

I was able to reproduce a strong correlation using the same Stalwart instance, endpoint, and authentication method:

  • account with 5 folders: LOGIN 27-46 ms
  • account with 6 folders: LOGIN 34-39 ms
  • account with 38 folders: LOGIN 318-327 ms
  • large account with 83 folders: LOGIN 589-692 ms

All measurements were repeated three times. The larger account was already warm, so this is not just the initial cache population.

For comparison, on the same system:

  • fetching a 107 KB message body over IMAP took 39-56 ms
  • directly reading a 1.46 MB object from S3 took about 93-100 ms total
  • PostgreSQL showed no blocked or unusually slow queries during the test

This is especially visible with webmail clients such as Nextcloud Mail because they create multiple short-lived IMAP connections while
opening one HTML message. Each connection pays the mailbox-state construction cost again.

Could this repeated per-mailbox scan explain the slow authentication reported here?

4

Try using the v1.0.0 dev branch, but not in production and using a separate database. It fixes the caching issues. Release is expected in about 2 months.

5

Adding a data point to this thread, because we hit what looks like the same mechanism from the write side, and it is much more expensive than the LOGIN case.

Setup: Stalwart 0.16.20, Ubuntu 24.04 VM (4 vCPU, 7.9 GB RAM), RocksDB data store, blobs on filesystem, FTS in the data store, internal directory. Migrating 39 accounts from CommuniGate Pro with imapsync over IMAPS. One account is an audit archive: 1.2 M messages, ~93 folders.

Symptom: APPEND cost grows with the number of messages already in the account. Same 30 KB message, same open TLS session, appended into a freshly created empty folder:

  • account with 1 message: 45 ms per APPEND, constant
  • same account at ~458k messages: ~1,100 ms per APPEND (872–1,863 ms)

The bulk copy went from 15–19 msg/s on day one (empty account) to 1.05 msg/s at 458k and 0.65 msg/s now at ~547k, and keeps degrading. Every APPEND also emits a Mailbox state change: a JMAP client (Bulwark) connected to that account refetches its whole folder list after each appended message, which is what first made us look at mailbox-state construction.

Excluded by measurement: the source server (fetches in 2–24 ms); contention (37 other accounts imported concurrently at 5–27 msg/s on the same server, and no speed-up once they finished, load ~1.0); RocksDB (per-event tracing over a 17 s window: store.data-write 31 ms / 299 calls, store.data-iterate 110 ms / 197 calls, message-ingest.imap-append 2–30 ms per message, iowait ≈ 0); full-text indexing (indexEmail=false + reload, verified in effect: 1,153 vs 1,120 ms); Cache.messages 50 MB → 512 MB (1,025 / 966 ms); rate limits; parallelism (1/2/4/8 concurrent APPEND streams into separate folders: aggregate 0.55/0.41/0.59/0.59 msg/s, latency 1,127/2,989/4,888/9,538 ms — a queue on one per-account resource, load average falling to 0.23 at 8 streams). The time is user CPU in the stalwart process, roughly one CPU-second per APPEND.

This fits in_mailbox*().count() scanning the whole emails.items cache, as @tonobo describes above — but on the insert path it runs on every message, not once per session, so a large migration pays it a million times.

Two questions:

  1. Is there anything on 0.16.x that avoids the rebuild during a bulk import — a setting, or something we could switch off while importing and rebuild afterwards, the way Search.indexEmail + AccountMaintenance/reindex works for FTS?
  2. Since 1.0 fixes it: could the fix, or a narrower one for the counters, be backported to an upcoming 0.16.x release? At the current rate the remaining ~650k messages will take about two more weeks, and we would rather not run a development branch in the middle of a migration.

Happy to share the tracing output or run any further measurement.

6

No backport is possible. 1.0 is expected in 1 to 1.5 months.