Issue Description
Rules of type Url cannot distinguish a bare domain-like string found in body text from a real clickable link, because URL candidates are scheme-normalised before any rule is evaluated.
crates/spam-filter/src/analysis/init.rs rewrites every schemeless URL the tokenizer extracts:
TokenType::UrlNoScheme(s) => {
TokenType::UrlNoScheme(UrlParts::new(format!("https://{}", s.trim())))
}
UrlParts keeps the pre-normalisation string in url_original, but SPAM_URL_VARIABLE does not expose it. A rule can read url, path_query, path, query, scheme, authority, host, sld, port — all derived from the rewritten value.
This matters because ZERO_WIDTH_SPACE_URL and SUSPICIOUS_URL come from the per-character scan in analysis/url.rs, whose candidate set includes bare domain-like strings taken from plain body text, not only href/src values. Many ESP templates deliberately insert U+200C into plain-text brand mentions so clients do not auto-linkify them — we see it from ConvertKit, SendGrid, Salesforce Marketing Cloud and Crypto.com. None are inside a clickable URL, yet at stock weights (7.0 and 5.0, either alone above a 5.0 threshold) every such message is classified spam on that basis alone.
Request: expose url_original (or equivalent) in SPAM_URL_VARIABLE, resolving to
UrlParts::url_original. The field already exists and is retained; this is exposure only, with no behaviour change and no cost to anyone not using it.
Expected Behavior
A Url rule should be able to identify that the flagged candidate was a bare mention rather than a functional link, so a site can exempt that case without weakening the signal generally. The natural expression is:
!contains(url, '://') && is_empty(path_query)
Actual Behavior
That condition can never be true. The https:// prefix is applied during normalisation, before rule evaluation, so contains(url, '://') is always true for a tokenizer-derived candidate.
No other variable distinguishes the two cases either: when a zero-width character breaks URI parsing, url_parsed is None, so host, authority and path_query are all empty — exactly as they are for a genuinely obfuscated href. The two are indistinguishable to the DSL, while url_original separates them trivially.
A related surprise: the same rule does fire for HTML attribute values, which analysis/url.rs builds with UrlParts::new(value.trim().to_string()) and does not prefix. So an identical rule matches or not depending on which candidate class the string came from, which is not documented.
Reproduction Steps
- Add a
Urlrule with condition!contains(url, '://') && is_empty(path_query)and a tag, e.g.SCHEMELESS_BARE_URL. - Add an
Anyrule combining it with the native tag:$ZERO_WIDTH_SPACE_URL && $SCHEMELESS_BARE_URL. - Classify a message whose
text/plainbody contains a bare domain mention broken by U+200C, e.g.Expe<U+200C>dia<U+200C>.<U+200C>com, and no other URLs. ZERO_WIDTH_SPACE_URLandSUSPICIOUS_URLfire;SCHEMELESS_BARE_URLdoes not, so the exemption never applies.- Move the identical string into an HTML
<a href="...">and reclassify — nowSCHEMELESS_BARE_URLfires and the exemption applies.
Both were confirmed against the running server using x:Action/set with ClassifySpam.
Relevant Log Output
(no error is logged — the rule evaluates correctly and simply never matches; the observable evidence is the tag list from x:Action/set ClassifySpam)
plain-text bare mention:
ZERO_WIDTH_SPACE_URL, SUSPICIOUS_URL, UNPARSABLE_URL score 22.90
(no SCHEMELESS_BARE_URL, so no exemption)
same string as an HTML href:
SCHEMELESS_BARE_URL, ZWSP_SCHEMELESS_EXEMPT,
SUSPICIOUS_URL_SCHEMELESS_EXEMPT, ZERO_WIDTH_SPACE_URL,
SUSPICIOUS_URL score 11.60
UNPARSABLE_URL on the first case is corroborating: analysis/url.rs skips every candidate without :// before reaching that tag, so its presence proves the plain-text candidate was carrying a scheme.
Stalwart Version
v0.16.x
Installation Method
Built from source
Database Backend
PostgreSQL
Blob Storage
PostgreSQL
Search Engine
PostgreSQL
Directory Backend
OIDC
Additional Context
Two earlier attempts to fix this locally, both unsuccessful, which is what motivates the request:
- Scoping the exemption to a single verified sender identity. Correct but does not scale — every new ESP exhibiting the same template behaviour needs its own rule.
- The
!contains(url, '://')rule above, which turned out to be unsatisfiable for the case it was written for, and was only discovered to be inert by checking whether the tag ever appeared in a realX-Spam-Result.
Current workaround is to down-weight ZERO_WIDTH_SPACE_URL (7.0 → 1.5) and SUSPICIOUS_URL (5.0 → 1.0) via x:SpamTag. That works but loses the distinction between genuine obfuscation and a brand mention — precisely the distinction a rule would have preserved. If url_original were exposed, the down-weight could be reverted in favour of a targeted rule.
Also worth noting the asymmetry in the second reproduction step: identical text, identical rule, different outcome depending on whether it came from body text or an href. Even without the feature, documenting that difference would help.
Environment: FreeBSD 15 jail, in-memory store is Redis (Valkey), single node.
On the build, in case it matters for triage: this is a locally-compiled FreeBSD port from a private ports overlay, and the ports checkout was unclean at build time, so it is not reproducible from a stock tree. I do not believe it affects this report — the behaviour is in analysis/init.rs normalisation that no build flag would alter, and I confirmed it against the v0.16.12 tag of the upstream repository, which matches the installed version exactly. But I cannot currently reproduce on an official binary without a separate install, so say the word if that is needed.
I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.
on
I have searched this support forum (open and closed topics) and confirm this is not a duplicate.
on
I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.
on