Feature request: Make the message context sent to the LLM configurable in the WebUI
Following our recent testing of Stalwart’s built-in LLM spam classifier, I would like to suggest making the message context supplied to the LLM configurable in the WebUI.
I described our practical tests in a separate General post. The short version is that we found significant classification improvements simply by giving the model additional message metadata that Stalwart already has available internally.
Currently, according to the documentation and our inspection of the LLM request, the classifier primarily supplies the subject and selected message body to the AI model.
For our licensed Stalwart Enterprise installation, we experimentally modified the LLM integration to additionally provide:
From:
To:
Envelope-To:
This produced two interesting A/B results with the same model, prompt and temperature.
Example 1: sender context
A phishing-style message claimed to be from Commerzbank:
From: Commerzbank Support <[email protected]>
Subject: Ihre neue Commerzbank Card – Jetzt aktivieren
Without From: available to the LLM:
Legitimate,High,NORMAL_MESSAGE
After adding From::
Harmful,Medium,UNEXPECTED_ACTION
Example 2: recipient mismatch
A promotional/spam message contained:
To: [email protected]
but was actually delivered through SMTP to:
Envelope-To: [email protected]
A comparable message had previously been classified as:
Commercial,High,COMMERCIAL_OFFER
With both To: and Envelope-To: available to the LLM, our controlled test produced:
Harmful,High,UNEXPECTED_ACTION
Suggested WebUI option
It would be useful if administrators could select which message information is supplied to the LLM under:
Settings > Spam Filter > LLM Classifier
For example:
LLM message context
Content
☑ Subject
☑ Body
Addresses
☑ From
☑ To
☑ Envelope-To
☑ Reply-To
Authentication
☐ SPF result
☐ DKIM result
☐ DMARC result
The selected values could be appended to the LLM input using stable, explicit labels such as:
From: ...
To: ...
Envelope-To: ...
Reply-To: ...
SPF: pass
DKIM: pass
DMARC: pass
Subject: ...
Body:
...
This would have several advantages:
-
administrators could decide how much context the model receives;
-
token usage could remain under their control;
-
privacy-sensitive installations could enable only the fields they consider appropriate;
-
different local or hosted models could use the same structured context;
-
useful information already available inside Stalwart could be exposed to the semantic classifier without requiring local source modifications.
I would especially suggest supporting Reply-To in addition to the three fields we tested, as a discrepancy between From and Reply-To can be highly relevant for phishing detection.
SPF, DKIM and DMARC results could also be useful as optional context. They should of course be treated as evidence rather than proof that a message is legitimate: correctly authenticated domains can still send spam or malicious messages.
For a conservative default, I would suggest enabling:
Subject
Body
From
To
Envelope-To
Reply-To
while leaving authentication context optional.
Our main observation from the tests was that improving the information available to the LLM can materially improve classification without requiring a larger model or a different prompt.
A configurable message-context section in the existing LLM Classifier WebUI would also remove the need for local source patches when administrators want to experiment with these signals.