Fixed IMAP IDLE not sending EXISTS when it should, can't issue PR on github

Issues
1

Issue Description

IMAP IDLE clients do not receive real-time new-message notifications when mail is delivered into the currently selected mailbox on Stalwart. The TCP/TLS connection stays open and IDLE appears active, but new mail is only discovered later by periodic polling/sync.

Expected Behavior

When an IMAP client has selected INBOX and entered IDLE, then a new message is delivered to INBOX, Stalwart should promptly send untagged selected-mailbox updates such as:

  • 1 EXISTS
  • 1 FETCH (FLAGS () UID 1)

The client should wake immediately without waiting for its periodic sync interval.

Actual Behavior

Stalwart accepts the IDLE connection, keeps it open, and periodically refreshes correctly, but does not push EXISTS / FETCH notifications for new mail delivered to the selected mailbox.

In our observed production case, all new-message detection lined up with the client’s 15-minute periodic sync rather than the actual delivery time.

Reproduction Steps

  1. Connect to Stalwart over IMAPS.
  2. Authenticate as a user.
  3. Select INBOX.
  4. Enter IDLE.
  5. Deliver a new message to that same account’s INBOX via SMTP/LMTP.
  6. Observe the active IDLE connection.

Expected: immediate untagged EXISTS / FETCH.

Actual: no selected-mailbox new-message notification is sent on the IDLE connection.

Relevant Log Output

From client-side testing/telemetry:

IDLE TCP connections establish successfully and re-issue every ~26 minutes.
Stalwart silently never pushes EXISTS notifications to the IDLE channel.
Every new-mail detection lines up with the 15-minute periodic sync, not with mail arrival times.

TLS and IMAP connectivity were otherwise healthy:

  • OK [CAPABILITY IMAP4rev2 IMAP4rev1 ENABLE SASL-IR LITERAL+ ID UTF8=ACCEPT JMAPACCESS AUTH=PLAIN AUTH=OAUTHBEARER AUTH=XOAUTH2] Stalwart IMAP4rev2 at your service.
    a1 OK CAPABILITY completed

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

Additional Context

he fix changes IMAP IDLE change processing so selected-mailbox email changes are handled before general mailbox STATUS changes.

Previously, write_changes processed mailbox status updates first, then selected mailbox email changes. In the selected-mailbox delivery case, that ordering could cause the client to receive mailbox-level status activity
without the expected selected-mailbox EXISTS / FETCH notifications.

The patch:

  • Processes selected mailbox changes first.
  • Emits EXISTS / FETCH for changed messages in the selected mailbox.
  • Then processes general mailbox STATUS updates.
  • Avoids returning early after selected mailbox handling, so unrelated mailbox status updates are still sent.

A regression test was added for SMTP/LMTP delivery into a selected INBOX while the client is in IDLE.

Tested with:

cargo fmt --package imap --check
STORE=Sqlite cargo test -p tests imap – --nocapture

Patch branch:

GitHub - brocktice/stalwart at fix-imap-idle-smtp-exists · GitHub

Compare link:

Comparing stalwartlabs:main...brocktice:fix-imap-idle-smtp-exists · stalwartlabs/stalwart · GitHub

Upstream PR could not be opened because repository owners have restricted PR creation to collaborators.

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Let me look at the relevant IMAP IDLE source code to understand the current implementation before responding.

Thank you for this detailed report and patch. Looking at the current upstream source in crates/imap/src/op/idle.rs, the write_changes function does indeed process mailbox-level STATUS changes first (lines ~135-175) before handling selected-mailbox email changes (lines ~178+). There is also an early-return at the end of the selected-mailbox block (return self.fetch(...)), which means if only email changes are present the flow is correct, but the ordering you identified is real: if both check_mailboxes and check_emails are true, the general STATUS updates are written first, which matches the race condition you observed.

Your diagnosis is accurate and the fix you’ve described – processing selected-mailbox changes before general mailbox STATUS updates – is logically sound given the code.

A few notes for the maintainers reviewing this:

  • The return Ok(()) at line ~206 (when new_state == modseq) and the return self.fetch(...) inside the check_emails block both look like they could silently skip mailbox STATUS writes in certain orderings, which aligns with your report.
  • The upstream repository has PRs restricted to collaborators, so the patch branch at https://github.com/brocktice/stalwart/tree/fix-imap-idle-smtp-exists and the compare link are the right places for maintainers to pick this up.
  • The regression test you added (STORE=Sqlite cargo test -p tests imap -- --nocapture) covers exactly the SMTP-delivery-into-selected-INBOX-during-IDLE scenario, which is not currently tested in tests/src/imap/idle.rs in the upstream tree.

Since you cannot open a PR directly, this forum post with the branch link is the correct path. A Stalwart maintainer will need to review and merge the patch. No configuration workaround exists on the user side for this – it is a server-side notification ordering bug.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong. If you found it useful, mention @helpbot in any reply to ask follow-up questions and the bot will respond again.

3

I added an assertion to the existing IDLE integration test that does exactly what you described: SELECT INBOX, IDLE, LMTP-deliver to that INBOX, then require both an EXISTS and a FETCH to come back on the IDLE connection. With no code changes on main, that assertion passes. EXISTS and FETCH are being emitted. Tracing it through the code matches: EmailPush sets both the email and mailbox flags, write_mailbox_changes runs synchronize_messages, sees uid_max increase and serializes EXISTS, and the changelog walk produces the FETCH. The early return you removed only fires when there are no message changes, so it can’t be swallowing an EXISTS that should have been sent.

What I think you’re actually hitting is the order. The current code emits an untagged STATUS for the currently selected mailbox first, then EXISTS, then FETCH. RFC 9051 §6.3.11 is pretty clear that STATUS shouldn’t really be used on the selected mailbox at all, and at least a couple of older clients (some mutt and fetchmail builds in particular) get tripped up by a server spontaneously sending one during IDLE. Reordering hides the symptom for those clients, which is probably why your patch made your 15-minute polling go away in practice.

So the patch as written isn’t quite the fix I’d land, because it keeps emitting that STATUS for the selected mailbox, just later in the sequence.