I was using Postgres for search in 0.16. What changes can I expect in v1 for the built in search? Does built in and/or Postgres support features like sub string search and document search? Is switching between them a migration that can be done at any time in the future? Thank you.
I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.
In v1.0 the built-in search engine was rewritten, and you can switch between search backends later. The catch is that the index doesn’t move with you, so you have to rebuild it.
What changed. The v1.0 announcement says the old design stored one bitmap per word and had no word positions. The new engine keeps word positions for every message, which makes real phrase searches possible. It indexes word stems and prefixes, compresses its data with zstd, and indexes new mail through a background queue with retries and rebuilds. The post says the PostgreSQL and MySQL search backends were reworked as well.
Substring search. The docs I found describe stem and prefix indexing and phrase search. They don’t say that arbitrary substring matching (for example, matching the middle of a word) is supported. I can’t confirm it for either backend.
Document search. Text extracted from attachments is indexed through the attachment email field, which is on by default. For files, the file name is always searchable. The file contents are indexed only if indexFiles is enabled. Calendar events and contacts can also be indexed. The built-in index covers about 1 MiB per document, and attachment text is cut first. On PostgreSQL, each field is limited to 250,000 characters, so text past that point isn’t searchable. PostgreSQL stemming only works for languages whose text search configuration is installed.
Built-in or Postgres. With the Default search store type, a PostgreSQL data store uses PostgreSQL’s own full-text search (tsvector and GIN). On RocksDB, SQLite and FoundationDB it uses the built-in index. You can also use a separate PostgreSQL database as the search store, or ElasticSearch or Meilisearch.
Switching. You can change the search store at any time. The existing index is not copied, so the new store starts empty. Restart the server after the change, then rebuild the index. Text searches return incomplete results until the rebuild finishes.
With PostgreSQL as your data store, the Default search store in v1 keeps using PostgreSQL’s own full-text search, so after the upgrade you’re on the same kind of index you had in 0.16. Stalwart’s built-in index is what Default means on RocksDB, SQLite and FoundationDB, so it needs one of those as the data store. It does stemming for 18 languages and caps each document’s index at about 1 MiB; PostgreSQL caps each field at 250,000 characters and stems only for the text search configurations installed in the database. The comparison table is here: Search store | Stalwart
Neither backend does arbitrary substring search. Both match whole words (with stemming where the language allows), a quoted term matches as a phrase, and a term ending in * matches as a prefix, so invoi* finds “invoice” and “invoicing”. Searching for “voice” will not find “invoice”.
Document search works with both. Text is extracted from PDF, Office (docx, xlsx, pptx), OpenDocument, EPUB and RTF attachments when they’re indexed, and the JMAP text filter and IMAP TEXT search cover it. Details and limits: Attachment text extraction | Stalwart
You can switch search stores whenever you like. The new store starts empty, so after changing it and restarting you rebuild the index, and text searches are incomplete until that finishes: Search indexing | Stalwart