Outlook Investigation - Stalwart DB Connectivity

[itsupport]

I just swapped to Stalwart two (2) weeks ago from another provider. I have what I believe everything configured correctly but we are still seeing delays in send/receive and sync progression with some end users on Outlook itself. Personally using betterbird and I am finding that the send/receive and sync times are working correct.

I am wondering if there is anyway we can speed up the send/receive and sync progression when it talks to the Stalwart DB. We are noticing the delay with it happening now compared to our old provider.

Relevant info:
Outlook IMAP sync very slow and load-dependent (150–770 msgs/s) — per-message DB round trips, PostgreSQL backend, 0.16.21

SUMMARY
Since migrating 554 accounts from VPOP3 to Stalwart (cutover Oct 6–7, 2026), Outlook (IMAP) users see slow or never-completing Send/Receive and folder sync. A flag-only fetch (UID FETCH 1:* (FLAGS)) runs
at 650–770 msgs/s when the system is quiet but drops to 150–210 msgs/s at busier times, on the same folders. PostgreSQL is fast and lightly loaded (~0.06–0.09 ms per query); the time appears to be spent
in sequential Stalwart ↔ PostgreSQL round trips (several per message) and in waiting for a pooled connection while each request crosses the network. We would like confirmation of the cause and guidance on the best fix (configuration, deployment layout, storage backend, or upgrade path).

ENVIRONMENT
Stalwart

• Version 0.16.21 (Enterprise), Ubuntu 24.04.4 LTS, kernel 6.8.0-142-generic
• Proxmox VM: 8 vCPU, 15 GB RAM. CPU 140–210% during business hours; RSS 2.6–3.1 GB
• Stores: DataStore = PostgreSQL (TLS on, poolMaxConnections 10 (raising to 30 once we can restart the service), timeout 15000 ms, poolRecyclingMethod
  fast, no read replicas). BlobStore, SearchStore, InMemoryStore = Default (all on the same PostgreSQL
  data store). Full-text search is therefore in PostgreSQL (s_email).
• Cache: messages 4 GB (raised from 1 GB; active since restart Oct 21 00:28 UTC), accessTokens 64 MB,
  accounts 64 MB. No measurable effect; RSS stays ~3 GB.

PostgreSQL (dedicated physical server)

• PostgreSQL 16.15, 251 GB RAM, ZFS (ARC c_max 64 GB) on 2 x 14 TB KIOXIA enterprise SSDs
• shared_buffers 32 GB, effective_cache_size 160 GB, work_mem 64 MB, maintenance_work_mem 2 GB,
  max_connections 300, random_page_cost 1.1
• Network: 1 GbE (DB server NIC speed 1000 Mb/s). The Stalwart VM shares its Proxmox host’s 1 GbE
  uplink with other VMs. DB server sends ~690 Mbps to Stalwart sustained (~70% of link), 24/7.
• RTT Stalwart → PostgreSQL: quiet 0.20 ms avg / 0.49 ms max; busy 0.31 ms avg / 0.89 ms max;
  loopback on the DB server 0.04 ms

Database (total ~2.8 TB)

• t 2662 GB | s_email 101 GB | p 75 GB (incl. TOAST) | k 5.7 GB | l 1 GB | others < 1 GB
• Largest indexes: gin_s_email_body 14 GB, p_pkey 3.5 GB, k_pkey 3.0 GB, t_pkey 1.1 GB
• NOTE: we added one custom index: idx_s_email_accid_rcvd (accid, rcvd) INCLUDE (docid), which cut
  the date-window query “SELECT docid FROM s_email WHERE accid = $1 AND rcvd >= $2” from ~260 ms to
  ~12 ms. Please advise whether this is safe to keep, including for future schema migrations.

Workload

• 554 accounts, 5 domains; 25 mailboxes between 46 and 117 GB; large accounts hold 40k–124k messages across 49–76 folders, folders up to ~60k messages
• Clients: mostly Outlook in IMAP mode (no CONDSTORE/QRESYNC), some Thunderbird/Betterbird,
  Nextcloud Mail polling ~130 accounts, mobile devices
• Previous server (VPOP3) also used an external PostgreSQL database and handled the same clients without this issue

MEASUREMENTS
Flag-only sync (SELECT folder, then UID FETCH 1:* (FLAGS)), all folders of an account:

• Account A, 49 folders / 123,862 msgs: 11:54 ADT 162.8 s (761 msgs/s) | 14:10 ADT 596.6 s (208 msgs/s)
• Account B, 76 folders / 41,574 msgs: 11:56 ADT 60.5 s (687 msgs/s)
• Account B, single folder ~14k msgs: 11:56 ADT 18.1 s (772 msgs/s) | 14:34 ADT 96.4 s (150 msgs/s)
• Earlier (Oct 20): 1,929-msg folder 12.5–14.4 s; 27,579-msg folder 215 s (6–8 ms/msg)
• During a VACUUM of s_email: Account B 212.9 s (195 msgs/s)
  Performance is not tied to mailbox size; it varies ~5x with overall load on the same folders.

PostgreSQL side (same periods)

• Transactions/s from Stalwart: 14,555 (quiet) / 16,116 (busy)
• Non-idle Stalwart backends (100–200 samples): avg 1.3 / 1.0, max 6 / 4
  => PostgreSQL service time ~0.06–0.09 ms per query; DB is not saturated
• Active-query sampling: 195 of 200 samples = “SELECT v FROM p WHERE k = $1”
• Cache: ~92% of p reads served from PostgreSQL memory; ZFS ARC hit rate 90.6%; disks lightly
  loaded (13–23 MB/s, 600–1,600 reads/s)
• Per-table read rate, before → after vacuum: s_email ~295 → ~160 MB/s; p ~246 → ~217 MB/s

OUR INTERPRETATION (please confirm or correct)

1. Each message in a FLAGS fetch costs several sequential round trips to PostgreSQL. Community thread support.stalw.art #840 describes per-message round trips in 0.16.x being fixed in 1.0 without a
   backport. At ~5–6 trips x (0.2–0.3 ms network + ~0.07 ms DB), this matches the ~1.3–1.5 ms per message we see when the system is quiet.
2. A pooled connection is held for the whole round trip, not just DB execution time. At 16,000
   queries/s x ~0.37 ms per round trip, ~6 of the 10 pool connections are in use on average, so bursts from Outlook syncs likely exhaust the pool and requests queue inside Stalwart. That would
   explain the sharp, load-dependent slowdown while PostgreSQL looks idle.
3. Network latency also rises under load (0.20 → 0.31 ms avg, max 0.89 ms) on a ~70%-utilized
   shared 1 GbE link, lengthening every round trip.

ACTIONS TAKEN

• Stalwart cache raised (messages 4 GB, accessTokens/accounts 64 MB): no effect
• Expunge timers set (Trash/Junk 30 days, Email Submissions 3 days)
• PostgreSQL maintenance Oct 21: throttled VACUUM ANALYZE of l, p (incl. TOAST) and s_email (incl.
  TOAST), removing ~9 M dead tuples from p and ~3 M from s_email; per-table autovacuum scale factor
  0.02; autovacuum cost limit raised. Note: throughput dropped sharply for hours afterwards, likely cache eviction from the vacuum reading ~175 GB through a 64 GB ZFS ARC.
• Custom index idx_s_email_accid_rcvd added (see above)
• Stopped unattended upgrades from restarting PostgreSQL (it had interrupted syncs at night)
• Outlook server timeout raised to 10 min for a client stuck in a reconnect loop (logins dropped
  from ~53/h to ~10/h)
• poolMaxConnections raised 10 → 30 (takes effect at next restart). Result: [pending]

OPTIONS WE ARE CONSIDERING
a) Run Stalwart on the PostgreSQL server (loopback, no TLS on the DB connection, 64 cores)
b) Raise shared_buffers to ~96 GB so p and its TOAST stay resident
c) Upgrade to 1.0 when released

QUESTIONS

1. Can you confirm the per-message round-trip behaviour for FETCH FLAGS in 0.16.21 (thread #840)?
   Is there a patch, backport, setting or workaround for 0.16.x, and an expected timeline for 1.0?
2. Is our reading of the connection pool right (connections held for the full round trip)? What
   poolMaxConnections do you recommend for 554 accounts / ~16k queries/s, and is there a metric or log event that shows time spent waiting for a pool connection?
3. Of options a–c above, which do you recommend? For (a), any concerns running Stalwart on the
   database host?
4. Is TLS on the DB connection a significant cost at ~16k round trips/s, and is disabling it over a
   private network or loopback a supported, recommended setup?
5. Are there settings that reduce work for clients without CONDSTORE/QRESYNC (Outlook) doing repeated full-flag syncs?
6. Is the custom index on s_email safe to keep, and will 1.0’s schema migration handle it?
7. Should full-text search (s_email, ~100 GB, 17.6 M rows) move to a dedicated SearchStore at this size?
8. What tracing or data would you like captured during a slow period, and how should it be enabled safely in production? (We operate 24/7.)

[itsupport]

Since yesterday morning I have upped “MaxPoolConnections” to 30 and noticed a spike in network traffic. All of those connections immediately filled up and stayed that way. I reduced MaxPoolConnections down to 15 to reduce the bandwidth usage.

For reference, this is what traffic looks like constantly between Stalwart on my Proxmox VM and my PostgreSQL DB looks like on a physical server.

I plan to set up a separate connection between Stalwart and the DB on the back end so they can communicate with one another on a 10G NIC to see if that’ll help.

Biggest unknown/concern is the constant chatter between the two, is this expected behavior? What can I possibly do to help my end users with their syncing/send & receive times.

[stalwart]

In 0.16 the IMAP FETCH loop reads the message metadata from the store once per message, one sequential SELECT v FROM p WHERE k = $1 each, and it does that even for a FLAGS-only fetch where FLAGS, UID and RFC822.SIZE are already in the in-memory message cache. That’s one round trip per message, not five or six, but each one ships the stored metadata including the raw header block, which is probably a good share of the ~690 Mbps you see on that link. It also explains why raising the message cache to 4 GB did nothing: that read bypasses it. Thread #840 is a different problem (the cache being built at login for large cold accounts), although your bigger migrated accounts will hit that one too after a restart.

0.16 has no setting or backport for this, and 0.16.27 behaves the same. In 1.0, FLAGS, UID, RFC822.SIZE, MODSEQ and INTERNALDATE are answered from the cache with no store reads, and fetches that do need headers or structure load metadata in batches. 1.0 should be out at the end of this month or early November. It uses a new database layout, so moving to it means running the migration tool; plan for that.

On the pool: yes, a connection is held from checkout until the result comes back, network time included. If poolMaxConnections is unset, 0.16 uses 2 per CPU (16 on your VM), so 10 was below the default and 30 is a sensible value against max_connections 300. 0.16 has no wait timeout on the pool, so exhaustion shows up as queueing, not as errors, and nothing measures pool wait directly. The closest proxy is store.data-read-time, which covers range reads and includes the wait for a connection.

Of your three options, I’d do (a) now. It removes the RTT and takes Stalwart off a shared 1 GbE link that’s already ~70% busy and also carries every blob download from the 2.6 TB t table. Just give the Stalwart process its own cores and memory so it doesn’t fight the ARC and shared_buffers. (b) won’t buy much when Postgres is already answering in ~0.07 ms. (c) fixes the cause. TLS on the DB connection only costs a handshake when a pooled connection is created plus symmetric crypto per query, so it’s small, but turning it off over loopback or a trusted private link is fine and supported.

Outlook without CONDSTORE/QRESYNC will keep doing full flag syncs, and no 0.16 setting makes them cheaper; in 1.0 they become cache-only. Your idx_s_email_accid_rcvd index is safe on 0.16, since Stalwart only runs CREATE ... IF NOT EXISTS and never drops indexes it doesn’t know. It won’t carry over to 1.0, and it won’t be needed there: s_email no longer has rcvd. I’d leave moving full-text search to Elasticsearch or Meilisearch for later; it needs a reindex, and the network and pool issues come first.

For data during a slow period, enable Prometheus metrics and compare store.data-read-time, store.blob-read-time and imap.request-time between quiet and busy hours (https://stalw.art/docs/performance/measuring), with pg_stat_statements and link throughput on the database side. If you add a log tracer for imap.fetch, keep it short or limit it to a couple of accounts: each event lists every fetched document id, so on a 60k-message folder the lines get huge. I’d also try to find out what keeps the ~16k queries/s going around the clock; my guess is Nextcloud Mail polling 130 accounts, but the metrics will tell you. Post the numbers once the pool change and restart are in.

[itsupport]

I have these graphs so far, hope that helps. I forgot to mention as well, we have updated plans for tonight to try. I am going to connect a direct cable to Stalwart and the Stalwart DB as they both have a 40G NIC in them so they have a direct connection to one another to help alleviate some of the bandwidth usage that is going on.

As for Nextcloud, we only have a handful of people experimenting with Nextcloud Mail at the moment but we do have calendar syncs from Outlook/Thunderbird/Betterbird that sync to Outlook. I am going to investigate that more.