Stalwart breaks the thread on every outgoing message and fails to append the messages to the existing thread

Issues
1

Issue Description

Stalwart can not assign correct threadId to outgoing mails and breaks the chain although the headers correctly align.

Expected Behavior

Stalwart should connect messages in-reply-to each other with the correct threadId so that the responses go into the correct thread.

Actual Behavior

Although the Message-ID and the response’s In-Reply-To labels match identically; Stalwart creates a new thread for the responses I make.

Reproduction Steps

  1. Have a mail from outside ([email protected]) sent to your Inbox on Stalwart ([email protected]) (This is thread 1)
  2. Reply to this mail (from [email protected]). Your response will create a new thread although it should not. (Now we have thread 2)
  3. Then respond again to the mail received in [email protected]. This response will correctly append to thread 2, which is created on step 2.
  4. Then respond from [email protected] again; now we have thread 3, a new thread created again although it should not have been.

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

PostgreSQL

Blob Storage

PostgreSQL

Search Engine

PostgreSQL

Directory Backend

Internal

Additional Context

Example threadId “baa”:

The first message is received:

Message-ID: VI1PR01MB7117F571D7BB27FD41B3945CBEBB2@VI1PR01MB7117.eurprd01.prod.exchangelabs.com

Then I reply to this message and a new threadId is assigned as “bbp”. Here is my response:

Message-ID: [email protected]
In-Reply-To: VI1PR01MB7117F571D7BB27FD41B3945CBEBB2@VI1PR01MB7117.eurprd01.prod.exchangelabs.com
References: VI1PR01MB7117F571D7BB27FD41B3945CBEBB2@VI1PR01MB7117.eurprd01.prod.exchangelabs.com

As you see, In-Reply-To matches identically to the first received message which created “baa” thread. Now why wouldn’t Stalwart append the message to “baa” thread and create a new one (bbp) instead?

Then the peer responds to my reply and this time their response is correctly appended to the thread bbp:

Message-ID: PR3PR01MB7115F3965C11A5ECAE88FCB5BEBA2@PR3PR01MB7115.eurprd01.prod.exchangelabs.com
In-Reply-To: [email protected]
References: VI1PR01MB7117F571D7BB27FD41B3945CBEBB2@VI1PR01MB7117.eurprd01.prod.exchangelabs.com, [email protected]

And when I reply to this new response; my mail creates yet another thread although the headers match again:

Message-ID: [email protected]
In-Reply-To: PR3PR01MB7115F3965C11A5ECAE88FCB5BEBA2@PR3PR01MB7115.eurprd01.prod.exchangelabs.com
References: VI1PR01MB7117F571D7BB27FD41B3945CBEBB2@VI1PR01MB7117.eurprd01.prod.exchangelabs.com, [email protected], PR3PR01MB7115F3965C11A5ECAE88FCB5BEBA2@PR3PR01MB7115.eurprd01.prod.exchangelabs.com

Now we have thread “bpf”, yet another one created with this response.

(Note: I redacted only the domain name to be example.com)

As you see, reference chain is correctly adding up. And as you see, all Message-IDs are identical to the In-Reply-To ids. Yet Stalwart creates a new thread in every response and doesn’t append it to the thread which Stalwart already has all information of.

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Threading in Stalwart is keyed on two things together: a hash of the normalized subject (Stalwart’s “thread name”, which strips Re:/Fwd: prefixes, encoded words, whitespace differences, etc.) and the set of Message-ID/In-Reply-To/References header values. When a new message arrives, Stalwart first computes the thread-name hash and only scans the index bucket for messages that hash to the same normalized subject; within that bucket it then checks whether any stored reference matches one of the incoming message’s Message-ID/References/In-Reply-To values. If it finds an overlap, the message is attached to that thread’s thread_id (and if it matches more than one existing thread, those threads are merged).

This means a matching In-Reply-To/References chain alone is not sufficient to guarantee thread continuity — if the Subject header on your outgoing reply normalizes to a different value than the subject on the original message (for example due to encoding differences, added/altered prefixes, extra whitespace, or a mail client rewriting the subject), the lookup will search the wrong hash bucket and never see the matching reference, so a brand-new thread is created even though the reference headers line up correctly.

Given your reproduction (Stalwart’s own outgoing replies break the chain while the remote peer’s replies correctly re-attach), it’s worth comparing the exact raw Subject: header bytes on the original inbound message versus the outgoing reply Stalwart generated — a subtle subject mismatch there is the most likely explanation based on how this lookup is implemented (find_thread_id in crates/email/src/message/ingest.rs).

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

Then there’s a problem with this subject logic too.

Because not everyone is sending mails only in English which would have “Re:” or “Fwd:” prepended to the subject. In my case, it is kind of “Ynt:” and some clients prefer “Yn:” and “İlt” for forward or something similar.

Why wouldn’t Message-ID/In-Reply-To match along with References match be enough? If the subject is completely anew, maybe this logic would make sense. But taking a “hash” of the subject will eliminate the threading logic completely because almost every time it will create a new thread (which is exactly my case here); so what is the point of threading in that case?

4

This has been fixed in the v1.0.0 branch as it could not be fixed in v0.16 branch without an index migration. v1.0 is expected at the end of October or early November.