JMAP search: can we know when the external index is caught up?

Questions
1

Hi, we use Stalwart 0.16.14 with Meilisearch and we are building a search API on top of JMAP. Search quality and relevance are working quite well.

The problem for us is knowing if an empty result is really empty, or if the account is still indexing/reindexing.

Is there currently any account-level state or watermark that says something like “all emails up to this Email state are indexed”? Or is something like that planned for v1?

I found this thread about newly delivered messages missing from IMAP SEARCH:

I saw the received-date part is fixed in the v1 branch. I’m not sure if that also solves FROM, SUBJECT, TEXT and body searches while the external index is catching up.

Two other questions:

  1. Can SearchSnippet/get expose which field matched? For example sender, recipient, subject, body or attachment. A sender-only match currently gives us no useful snippet.
  2. Is the order from Email/query without an explicit sort intended to be relevance order, and can clients rely on it being deterministic?

We don’t want direct Meilisearch access and we don’t want to maintain a Stalwart fork. Mostly trying to understand if these things already exist, or if they are on the roadmap.

Thanks

2

Taking these in order. There is no account-level watermark saying the index has caught up, and nothing on the session or on Email state that a client can read for it. What does exist is the task queue: indexing is a per-document task, and pending tasks are queryable objects, so x:Task/query filtered by account (or stalwart-cli query task) tells you whether an account still has indexing work outstanding. That is a management API rather than a per-user one, so it works for a backend sitting in front of JMAP, which is your case, but it is not something you can hand to end users directly.

On the snippets: partly, and more than you may be getting today. SearchSnippet is RFC 8621’s shape, three properties, but subject and preview are HTML with the matching terms wrapped in <mark> tags, and each is null when nothing matched in that field. Stalwart implements that as specified. So there are two signals you can use: the <mark> positions tell you which terms hit and where, and the null pattern distinguishes a subject match from a body match.
What you cannot get is anything about from, to or attachments. The preview is only built from text and HTML body parts (including inside forwarded messages), so a sender-only match legitimately comes back with both fields null, which is the case you ran into. That is the object’s shape in the RFC rather than something left out on our side, so a per-field matched indicator would mean going beyond the spec. Worth knowing when you design around it: snippets are also capped at 255 characters, and only the first matching body part is used.

On ordering: do not rely on it. It depends on which filters are present. If every filter is handled by the external engine and there is no sort, you get the engine’s relevance order. If a local filter (mailbox, keyword, ACL) is mixed with a text filter, results are merged locally and you end up in document id order. With no filters at all it is always document id order. JMAP does not define a default sort either, so pass an explicit sort for anything that needs to be deterministic. Happy to take “relevance order for mixed-filter queries” as a feature request if that is the one that matters to you.

3

Thanks, this was very helpful.

We tested it on 0.16.14. x:Task/query and x:Task/get work with a scoped API key, and the IndexDocument tasks include accountId.

One small difference: filtering x:Task/query directly by accountId returned unsupportedFilter for us. So for now we have to query the task set and filter by account on our side.

We also created the mixed-filter relevance request here:

Thanks again.