Hi everyone,
I would like to share some practical experience from our current Stalwart setup, because we found something interesting while testing the built-in LLM spam classification.
Our goal was not to replace Stalwart’s classical spam filtering, but to add semantic classification as another independent signal.
We deliberately tested two different deployment approaches:
-
a commercially hosted LLM connected via API;
-
a self-hosted Mistral-family model running locally through LocalAI.
We do not see these approaches as competitors. They represent two different operational models.
The hosted API approach is much easier to deploy. There is no local inference hardware to maintain, no model installation and very little infrastructure work.
The local approach requires considerably more effort. The hardware has to be suitable, the model has to be selected and installed, LocalAI has to be operated and updated, and CPU/RAM resources have to be managed.
On the other hand, local inference keeps the complete classification process under local control and avoids per-request API costs.
Our current local setup uses LocalAI with a Ministral/Mistral-family 14B model.
What we discovered
During testing we found an interesting false negative.
We had a phishing-style email claiming to be from Commerzbank.
The visible sender was:
Commerzbank Support <[email protected]>
The subject was:
Ihre neue Commerzbank Card – Jetzt aktivieren
The HTML part contained a fake Commerzbank card activation request.
However, the plain-text part contained an unrelated and harmless Schneider Invest test message.
Stalwart’s classical spam analysis already detected suspicious characteristics of the message, including differences between MIME parts.
The LLM nevertheless classified the message as:
Legitimate,High,NORMAL_MESSAGE
This initially looked like a model failure.
It was not.
Inspecting the actual LLM request
We captured the HTTP request from Stalwart to LocalAI on localhost.
That showed us that the built-in LLM integration was essentially sending:
Subject
Body
The model did not receive the visible From: address.
For multipart/alternative messages, the text body selected by Stalwart can also be different from what the user primarily sees in HTML.
So in our Commerzbank example, the model effectively saw a banking-related subject followed by an innocent plain-text message — but it did not see:
From: Commerzbank Support <[email protected]>
From that perspective, the Legitimate classification suddenly made much more sense.
Small source experiment: adding From
We are running a licensed Stalwart Enterprise installation and tested a small source modification in our own deployment.
The relevant code is in:
crates/spam-filter/src/analysis/llm.rs
We added the parsed visible sender to the prompt before the subject.
Conceptually, the LLM input changed from:
Subject
Body
to:
From
Subject
Body
Nothing else was changed:
-
same email;
-
same model;
-
same classifier prompt;
-
temperature 0.0.
The result changed from:
Legitimate,High,NORMAL_MESSAGE
to:
Harmful,Medium,UNEXPECTED_ACTION
That was our first indication that the quality of the information supplied to the LLM may be at least as important as changing or enlarging the model itself.
Second experiment: To and Envelope-To
During normal mail operation we then observed several spam campaigns where the visible To: address had nothing to do with the mailbox that actually received the message.
For example:
From: Beach Security <[email protected]>
To: [email protected]
while the actual SMTP recipient was:
[email protected]
Stalwart already has this information internally and can detect forged-recipient patterns using its classical rules.
But the LLM did not receive the difference between the visible recipient and the SMTP envelope recipient.
We therefore made a second small experiment and added:
To:
Envelope-To:
The LLM input now looks conceptually like this:
From:
To:
Envelope-To:
Subject:
Body:
We then created a controlled test message with:
To: [email protected]
Envelope-To: [email protected]
and verified the exact request sent to LocalAI.
A comparable Beach Security message had previously been classified semantically as:
Commercial,High,COMMERCIAL_OFFER
With the recipient mismatch available to the model, the controlled test produced:
Harmful,High,UNEXPECTED_ACTION
Again, we did not change the model or the classifier prompt.
We only supplied additional message context that Stalwart already had available.
Why we find this interesting
The main lesson for us so far is not that one particular LLM is better than another.
The more interesting point is that semantic spam classification depends heavily on what the model is actually allowed to see.
A relatively small local model can make a substantially different decision when it receives information such as:
From
To
Envelope-To
Subject
Body
instead of only subject and body.
At the same time, we still regard the LLM as only one part of the overall spam decision.
Our setup combines signals from:
-
Stalwart’s classical spam rules;
-
SPF/DKIM/DMARC;
-
reputation and DNS-based signals;
-
statistical filtering;
-
semantic LLM classification.
This is important because LLMs can also generate false positives.
For example, we have seen a legitimate Cloudflare security notification classified by the LLM as harmful, while Stalwart’s classical authentication and reputation signals correctly pushed the overall result back into ham territory.
So our intention is not to let the AI replace the existing filtering logic.
It is another independent source of evidence.
Current status
We are now running the From + To + Envelope-To modification in production observation and are collecting real-world examples before changing anything else.
For the moment we are deliberately keeping the following unchanged:
classifier prompt
temperature
LLM scores
spam thresholds
This gives us a cleaner way to see what effect the additional input fields actually have.
Possible later experiments could include carefully selected context such as:
Reply-To
SPF result
DKIM result
DMARC result
URL domains
campaign/history information
but we have not implemented those yet.
One particularly interesting future direction may be campaign recognition: detecting repeated subjects/body fingerprints across changing sender domains and recipients, and providing only a compact campaign summary to the LLM.
For now, however, we want to observe the current setup before adding more variables.
I thought this might be useful to others experimenting with Stalwart’s LLM integration.
The most surprising result for us was how much difference a few additional pieces of already available mail metadata made without changing the model itself.