IMAP SEARCH silently omits recently delivered messages when using an external SearchStore (eventual-consistency race)

Kees_Smit · July 24, 2026, 10:17am

Issue Description

With an external SearchStore, IMAP SEARCH criteria (SINCE, FROM, SUBJECT, TEXT, …) are answered exclusively from the external index (crates/store/src/dispatch/search.rs::query_account). The index is populated asynchronously by the task queue, typically a few seconds after ingest.

A SEARCH executed inside that window silently omits messages that already exist in the mailbox (SELECT/STATUS/FETCH all see them).

This race is not theoretical: clients such as Outlook enumerate new mail by issuing SEARCH immediately upon an IDLE/EXISTS notification, milliseconds after delivery, so they lose the race on effectively every delivery.

Worse, the client then records the mailbox as synced at the current UIDNEXT/HIGHESTMODSEQ, so the missed message is never fetched again. It stays permanently invisible in the client until unrelated mailbox churn forces a re-enumeration.

Restarting the client does not help.

The external-comparator sort path already handles this (“Add any remaining results not yet in the index” in dispatch/search.rs); the filter path has no equivalent.

Expected Behavior

SEARCH results reflect the actual mailbox contents (RFC 3501 §6.4.4, the search is over messages in the mailbox). A message visible to SELECT/FETCH must not be absent from a SEARCH that its content satisfies, regardless of index flush state. Suggested fix: compute the mask documents not yet present in the external index (a doc-id-range sub-query, as the sort path already does) and include them conservatively, or evaluate the pending few inline against the criteria.

Actual Behavior

SELECT INBOX returns EXISTS n, UIDNEXT u including the new message; UID SEARCH SINCE issued immediately afterwards does not return its UID. Roughly 4s later (index flushed), the same SEARCH returns it.

Reproduction Steps

Configure an Elasticsearch SearchStore.
Deliver a message to a mailbox via SMTP.
Within ~2s, in an IMAP session: SELECT INBOX, then UID SEARCH SINCE .
The new UID is missing from the SEARCH response while STATUS/SELECT counters include it. Repeat after ~5s: the UID appears.

Relevant Log Output

09:45:59 INFO Message ingested (message-ingest.ham) accountId=1 documentId=71 mailboxId=[0]
09:45:59 DEBUG IMAP SELECT INBOX total=24 uidNext=39 ← message visible
09:45:59 DEBUG IMAP SEARCH mailboxId=0 total=23 ← SEARCH result set omits it
09:45:59 DEBUG IMAP FETCH documentId=[71] [“Flags”,“Uid”] ← client fetched flags only,
never enumerated it as new;
message permanently missing
in client until later churn

Stalwart Version

v0.16.x

Installation Method

Built from source

Database Backend

FoundationDB

Blob Storage

S3-compatible

Search Engine

Elasticsearch

Directory Backend

Internal

Additional Context

Suggested fix:

Before evaluating external criteria, figure out which documents in the mask are not in the external index yet, using the same doc-id-range sub-query the sort path already uses, and treat those as matching any positive external criterion. Internal filters (flags, sequence sets) still apply to them, under NOT they stay excluded, and in sorted results they go last, same as the sort path does today. The pending set is empty in steady state, so nothing changes outside the indexing window. The stricter option would be to evaluate the pending few (usually 0 to 2 documents) against the actual criteria inline. We run the patch below in production against v0.16.4;

diff --git a/crates/store/src/dispatch/search.rs b/crates/store/src/dispatch/search.rs
— a/crates/store/src/dispatch/search.rs
+++ b/crates/store/src/dispatch/search.rs
@@ -61,16 +61,57 @@ impl SearchStore {
return Ok(query.mask.iter().collect());
}

   // The external index is eventually consistent: documents queue for

   // indexing after ingest, so treating the index as authoritative

   // silently drops freshly delivered messages from results. Compute

   // the mask documents not yet present in the index and treat them as

   // matching any positive external criteria; internal filters (flags,

   // sequence sets) still apply to them. Under NOT they stay excluded.

   let unindexed = if has_external_filters

       || (!has_local_filters && !query.comparators.is_empty())

```
   {
```
```
       let range_filters = vec![
```
```
           SearchFilter::Operator {
```

               field: SearchField::AccountId,

               op: SearchOperator::Equal,

               value: SearchValue::Uint(account_id as u64),

```
           },
```
```
           SearchFilter::Operator {
```

               field: SearchField::DocumentId,

               op: SearchOperator::GreaterEqualThan,

               value: SearchValue::Uint(query.mask.min().unwrap() as u64),

```
           },
```
```
           SearchFilter::Operator {
```

               field: SearchField::DocumentId,

               op: SearchOperator::LowerEqualThan,

               value: SearchValue::Uint(query.mask.max().unwrap() as u64),

```
           },
```
```
       ];
```

       let mut unindexed = query.mask.clone();

       for id in self.sub_query(query.index, &range_filters, &[]).await? {

```
           unindexed.remove(id);
```
```
       }
```
```
       unindexed
```
```
   } else {
```
```
       roaring::RoaringBitmap::new()
```
```
   };
```

   if !has_local_filters && query.comparators.iter().all(|c| c.is_external()) {
       return self
           .sub_query(query.index, &query.filters, &query.comparators)
           .await
           .map(|results| {
               if !results.is_empty() || has_external_filters {

```
                   results
```

                   let mut results = results
                       .into_iter()
                       .filter(|id| query.mask.contains(*id))

```
                       .collect()
```

                       .collect::<Vec<_>>();

                   // Pending-index documents sort last; the IMAP layer

                   // re-sorts plain SEARCH results anyway.

                   results.extend(&unindexed);

                   results
               } else {
                   // Database sort is broken, return masked results
                   query.mask.iter().collect()

@@ -92,12 +133,15 @@ impl SearchStore {
match split_filter {
SplitFilter::External(external) => {
// Execute sub-query

                   filters.push(SearchFilter::DocumentSet(

                       self.sub_query(query.index, &external, &[])

```
                           .await?
```

                           .into_iter()

```
                           .collect(),
```
```
                   ));
```

```
                   let mut set = self
```

                       .sub_query(query.index, &external, &[])

```
                       .await?
```
```
                       .into_iter()
```

                       .collect::<roaring::RoaringBitmap>();

                   // Pending-index documents match positive external

                   // criteria conservatively.

```
                   set |= &unindexed;
```

                   filters.push(SearchFilter::DocumentSet(set));
               }
               SplitFilter::Internal(filter) => {
                   filters.push(filter);

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

helpbot · July 24, 2026, 10:17am

Confirmed against the current source: crates/store/src/dispatch/search.rs::query_account does exactly what you describe. The external-comparator sort path (the !has_local_filters && query.comparators.iter().all(|c| c.is_external()) branch and the “Add any remaining results not yet in the index” fallback in the sorted-merge branch) already reconciles the mask against pending documents, but the external-filter path (the has_external_filters branch that builds SearchFilter::DocumentSet via sub_query) has no equivalent reconciliation step — it just returns whatever the external index currently has, intersected with the mask, with no fallback for documents not yet indexed. So a message that exists in the mask but hasn’t been flushed to the external index yet is dropped from filtered SEARCH results, exactly as your logs show.

This is a code-level bug report with a working patch, not something answered by docs, so I can’t give more than confirmation that the behavior and code paths you cite are accurate as of the current source. A staff/community member with commit context will need to review your proposed fix for correctness (e.g. interaction with NOT and with the total_results short-circuit above it).

github.com/stalwartlabs/stalwart

crates/store/src/dispatch/search.rs

main

/*
 * SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
 *
 * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
 */

use crate::{
    SearchStore, Store,
    search::{
        IndexDocument, SearchComparator, SearchField, SearchFilter, SearchOperator, SearchQuery,
        SearchValue,
        split::{SplitFilter, split_filters},
    },
    write::SearchIndex,
};
use std::cmp::Ordering;
use trc::AddContext;

impl SearchStore {
    pub async fn query_account(&self, query: SearchQuery) -> trc::Result<Vec<u32>> {

This file has been truncated. show original

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

stalwart · July 26, 2026, 10:25am

This has been already fixed in the v1.0.0 branch, the received date will now live in the cache and won’t require the search store to be up to date.

As a side note, the fix your LLM proposed is inefficient and the reason why LLM contributions are not accepted: they do the job but at a great cost.

Topic		Replies	Views
JMAP search: can we know when the external index is caught up? Questions no-ai	2	29	August 2, 2026
Outlook draft autosave still races IMAP SEARCH on v0.16 Issues	3	13	August 5, 2026
IMAP: Fetch does not list moved emails until I list the origin folder Issues no-ai	2	34	May 14, 2026
Fixed IMAP IDLE not sending EXISTS when it should, can't issue PR on github Issues	2	82	May 26, 2026
Keep relevance order when Email/query mixes text and local filters Feature Ideas	1	14	August 8, 2026