Full-text search: index text content of document attachments (PDF, etc.)

Feature Ideas
1

Currently the FTS indexer only extracts text from text/plain and text/html MIME parts (plus text parts of nested message/rfc822); all binary parts are skipped
(crates/email/src/message/index/search.rs, the catch-all arm in the part-type match). As a result, searching for text contained in a PDF attachment returns nothing, even with an
Elasticsearch search store configured.

Feature request: extract and index text from common document attachments (PDF at minimum) into the existing attachment search field, ideally opt-in with a size cap.

Two possible approaches:

  1. Extract in Stalwart at index time (e.g. a Rust PDF-text crate, sandboxed against malformed input).
  2. For the Elasticsearch backend: optionally include attachment bytes in the indexed document so an ES ingest-attachment (Tika) pipeline can extract text server-side.This would
    cover many formats (PDF, DOCX, …) with no parsing in Stalwart.
2

Confirming this with measurements from a Zimbra migration, plus a few things that may help whoever picks it up.

Measured on 0.16.16 (Meilisearch backend), one migrated account, 710 mailboxes / 40,223 messages:

  • 5,233 messages carry attachments; only 479 (9 %) have any text in the attach field —
    all plain-text formats (log files, .ics).
  • Controlled test: a purpose-built PDF with an uncompressed text stream and two unique
    markers, delivered over SMTP.
    • pdftotext extracts every marker → the PDF itself is fine
    • the message’s subject keyword is in the index → indexing ran
    • hasAttachment = true → the attachment is recognised
    • attach field: empty, and both markers return 0 hits on exact search

So the attachment is seen and its content is dropped.

Where it would fit — the synchronous path does not need to change:

  • index_document() (crates/email/src/message/index/search.rs) drops binary parts in _ => {}
  • IndexDocument::index_text() takes &mut self, so the async caller
    build_email_document() (crates/services/src/task_manager/index.rs) can append to the
    finished document
  • reqwest is already a dependency of crates/services; infer already does type detection
  • reindex_account() already exists, so existing mail can be caught up afterwards

Three operational points, from running a comparable Rust + Meilisearch pipeline elsewhere:

  1. Extraction costs more CPU than the entire rest of the ingest combined. It belongs in the
    background task manager with a hard timeout and a size cap, and must degrade to “index without attachment text” rather than stalling the queue.

  2. Caching is mandatory — signature images and boilerplate PDFs repeat thousands of times —
    and the key already exists. Stalwart addresses blobs by hash and deduplicates them:
    38,819 blobs back 40,223 messages in our account. An extraction cache keyed on the
    attachment blob hash reuses machinery that is already there.

  3. The switch should be per-account or per-tenant, not just global. In a hosting setup only a
    subset of mailboxes needs attachment search, and given the cost profile that granularity is
    what decides whether the feature is affordable at all.

On the two approaches in the first post: the Elasticsearch ingest-attachment route is
elegant, but it only helps ES deployments — Meilisearch, OpenSearch and the built-in backend
would stay blind, while the FTS backend is otherwise interchangeable. And for clustered
deployments, embedded Rust extraction couples extraction capacity to the number of mail nodes
and duplicates the work on each, whereas a shared external extractor scales independently. It
also keeps format parsers — historically a rich source of CVEs — out of the mail server process.

Glad to implement whichever direction you would accept. Is there a preference?