OpenPhish/PhishTank lookups miss mixed-case URLs because the URL is lowercased before exact matching

Issues
1

Issue Description

Stalwart 0.16.22 with the official spam-filter ruleset v3.0.1 has a case-normalization mismatch between extracted URLs and HTTP lookup keys.

The official URL rules use:

  • STWT_PHISHED_OPEN: key_exists(‘stwt_openphish’, url)
  • STWT_PHISHED_TANK: key_exists(‘stwt_phishtank’, url)

UrlParts::new() sets url to url_original.trim().to_lowercase(), lowercasing the entire URL, including its path and query.

However, the HTTP lookup loader preserves case in both List entries and CSV keys. HttpStore::contains() then performs an exact contains_key() lookup.

Consequently, a URL containing uppercase characters in its path or query can fail to match the identical URL present in the phishing feed.

The rules and lookup configurations were confirmed enabled on our server. This report is based on source inspection and an isolated reproduction of the lookup mismatch, not an end-to-end SMTP test.

Expected Behavior

An extracted URL that exactly matches a loaded OpenPhish or PhishTank entry should trigger PHISHED_OPENPHISH or PHISHED_PHISHTANK respectively, including when the path or query contains uppercase characters.

URL handling should preserve potentially significant case in the path and query, rather than indiscriminately lowercasing the complete URL.

Actual Behavior

For a feed entry such as:

https://example.invalid/Account/Verify?Token=AbC123

the rule looks up:

https://example.invalid/account/verify?token=abc123

The HTTP lookup contains the original key, so the lowercased lookup fails unless that separate lowercase key also exists.

This causes a missed phishing-list signal. Other spam checks may still detect the message; this report does not claim that the message necessarily reaches the Inbox.

In a freshly downloaded OpenPhish feed, 72 of 300 entries changed when lowercased and had no corresponding lowercase key. This is a snapshot of a changing feed, not a fixed affected percentage.

Reproduction Steps

Source-level reproduction, without contacting any phishing destination:

  1. Use this synthetic feed entry:
    https://example.invalid/Account/Verify?Token=AbC123

  2. Store it unchanged as a List entry, or as the URL key in a CSV lookup.

  3. Apply the transformation used by UrlParts::new():
    trim the URL and lowercase the complete string.

  4. Perform the exact key lookup used by HttpStore::contains().
    The original URL matches; the transformed URL does not.

Minimal Python illustration of the verified data flow:

import csv
import io

url = "https://example.invalid/Account/Verify?Token=AbC123"

list_keys = {url.strip()}
csv_data = "phish_id,url\n1," + url + "\n"
csv_keys = {
    row["url"]
    for row in csv.DictReader(io.StringIO(csv_data))
}

for name, keys in [("List", list_keys), ("CSV", csv_keys)]:
    print(name, url in keys, url.strip().lower() in keys)

Output:
List True False
CSV True False

This illustrates the source-level mismatch; it is not a Stalwart integration test.

Relevant Log Output

No SMTP trace was captured. The evidence is the source-level lookup mismatch described above, rather than a server error or exception.

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

Additional Context

Relevant source in v0.16.22:

URL lowercasing:

Expression variable mapping:

Exact lookup and case-preserving feed loading:

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

2

Thanks for the detailed writeup and source references. I confirmed the code path you cited: UrlParts::new() in crates/spam-filter/src/analysis/url.rs does lowercase the full url_original string (scheme, host, path, and query) via .trim().to_lowercase(), and that lowercased url field is what gets exposed to expression rules like url in key_exists('stwt_openphish', url). Meanwhile the HTTP lookup loader in crates/store/src/backend/http/lookup.rs stores List/CSV keys verbatim (case-preserved) and HttpStore::contains() does an exact contains_key() lookup with no normalization on the lookup side. So a case mismatch between a mixed-case feed entry and the lowercased extracted URL will indeed cause a miss, exactly as you describe.

This is a real behavioral gap between URL extraction normalization and HTTP lookup key matching, not something addressed in current docs, so I can’t point you to an official statement on intended fix behavior.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

This has been fixed in v0.16.24 (yet to be released).