Spam-filter: URL redirector resolution needs per-message cap, shared cache, and client reuse (rspamd/SpamAssassin parity)

Issues
1

Issue Description

(Claude assistance in quoted text)

Summary

The URL-redirector resolution in crates/spam-filter/src/analysis/url.rs issues serial, inline, uncached GETs at the SMTP DATA stage: a fresh reqwest::Client per fetch (http_get_header, url.rs:297 in v0.16.13), hardcoded 5 s timeout, redirect(Policy::none()) with a manual 3-hop walk, gated by the url-redirectors lookup list (~1076 domains). Compared with the two long-standing implementations of the same feature — rspamd url_redirector and SpamAssassin DecodeShortURLs (SA 4.0 core) — four protective layers are missing, and the gap is production-impacting.

Expected Behavior

  1. provide a dedicated proxy endpoint for URL checks (initially that could be a squid type, but I think the idea should be an endpoint that might perhaps do HTTP 4xx/5xx on bad URLs and HTTP3xx on suspect and HTTP200 on “clean” URLs)
  2. for each URL in message
  • quick return url lookup and if not found in cache
  • do a 4xx while background/offline process check remainder URLs found in message that was not not yet checked and cached

Actual Behavior

Production incident (what this costs today)

A Mailchimp campaign (119 KB, ~100 us.list-manage.com links — list-manage.com is on the stock url-redirectors list) arrived via an upstream relay into a deployment whose mail VLAN has no general internet egress. Every link burned the full 5 s timeout, serially, inside end-of-DATA: ~10 minutes between RCPT and queue.message-queued. The upstream relay (Postfix, default 600 s data-done timeout) hung up at minute 10; Stalwart completed ingest at ~minute 11 into a dead socket; the relay retried forever (message-ingest.duplicate every cycle) and its queue backed up with 4.4.2 deferrals. Even with working egress, 100 sequential fetches with per-fetch TLS setup is 30–100 s of DATA-stage latency — an outbound-request amplifier and a slow-loris an attacker can trigger with one link-stuffed message.

(Side observation: a >3-hop chain sets URL_REDIRECTOR_NESTED, and url.rs:180 then skips resolution for all remaining URLs in the message — an undocumented implicit cap with odd semantics.)

Reproduction Steps

  1. Stalwart server setup with inbound from mailrelay (My case ProxMox Mail Gateway)
  2. Stalwart only has HTTP(S) access to cached copies of the various fingerprint files (refreshed by cache servers that does outbound access) - and the S3 storage, anything else denied.
  3. receive file with 100s tracking/shortening URLs to be checked in URL DBs.

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

FoundationDB

Blob Storage

S3-compatible

Search Engine

Internal

Directory Backend

Internal

Additional Context

Proposals (rspamd/SpamAssassin parity)

  1. Per-message fetch cap — spam-filter.url-redirect.max-urls (rspamd: max_urls = 5; SA: max_short_urls = 10). When exceeded: tag (e.g. URL_REDIRECTOR_MANY) and stop fetching; REDIRECTOR_URL tagging and original-URL DNSBL checks still apply.
  2. Shared cache with TTL — no new dependency needed: the in-memory KV store already backs greylisting with exactly the required primitive (in_memory_store().key_set(KeyValue::new(key, ...).expires(ttl)), crates/smtp/src/inbound/rcpt.rs:268-294, KV_GREYLIST prefix pattern). New prefix, key = hashed URL, value = resolved Location or a status-code marker (negative caching, as SA does), TTL default 1 day, per-hop entries (rspamd ≥4.1 semantics). On shared backends (Redis/FDB) the cache is cluster-wide for free — one fetch per campaign link per day instead of one per recipient. Both prior implementations converged on exactly this (rspamd: Redis rdr:*, expire = 86400; SA: DBI short_url_cache, TTL 86400, negative results cached as bare status codes).
  3. Best-effort single-flight — rspamd writes a processing sentinel with SET NX EX so one worker cluster-wide resolves a given URL during a campaign burst. A best-effort sentinel-before-fetch with short TTL collapses duplicate concurrent fetches; a lost race merely duplicates one fetch.
  4. Configurable timeout + proxy + shared client — the 5 s timeout is hardcoded, and no egress proxy is possible: reqwest is pulled in with default-features = false, features = ["rustls","http2","stream"] (crates/common/Cargo.toml, crates/spam-filter/Cargo.toml), which drops reqwest’s default system-proxy feature — HTTP_PROXY/HTTPS_PROXY env vars are silently dead, and no ClientBuilder::proxy() is exposed. Network-restricted deployments (spam filter behind a strict egress firewall) currently cannot route these fetches through an egress proxy at all. Proposals: spam-filter.url-redirect.timeout, optional spam-filter.url-redirect.proxy wired to ClientBuilder::proxy(), build the client once at spam-filter init instead of per fetch (also restores connection/TLS-session reuse), optional per-message wall-clock budget (rspamd: 8 s). Consider making danger_accept_invalid_certs(true) (url.rs:301) configurable, keeping the current default (rspamd’s check_ssl analog).

All of this is fail-open, mirrors two independent implementations that converged on the same design, and reuses existing Stalwart infrastructure (lookup lists, in-memory TTL keys, trc events).

References

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

This confirms the report’s technical claims accurately reflect the current code: per-fetch client construction, hardcoded 5s timeout, no proxy support, no shared cache, no per-message cap. No settings currently exist to configure this behavior, and there’s no doc page addressing it.

I don’t see anything in the docs to point to that would change or refute this report, so I’ll respond acknowledging it as a valid, documented-gap issue rather than answer a question.

I reviewed the current url.rs implementation and the report’s technical details check out: a new reqwest::Client is built per fetch with a hardcoded 5s timeout, no proxy support (reqwest is pulled in without the system-proxy feature), no shared cache across recipients/nodes, and no per-message fetch cap beyond the undocumented “3-hop nested” skip-all-remaining behavior. There is no existing configuration setting (spam-filter.url-redirect.* or similar) to control this, and the docs don’t cover a cap, cache, or proxy option for the URL redirector resolver.

This is a legitimate feature/hardening request rather than something answerable from current docs, so I won’t fabricate a workaround. A staff or community member with visibility into the roadmap should follow up on whether/when these four proposals (cap, cache, single-flight, configurable timeout/proxy/shared client) will be implemented.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

This will be considered for v1.0.0.