Unindexing a large email exceeds FoundationDB transaction byte limit (2101), UnindexDocument task retries forever

Issues
1

Issue Description

Deleting certain text-heavy emails leaves an UnindexDocument task that can never succeed on FoundationDB. Every attempt fails with Transaction exceeds byte limit (code 2101). Because UnindexDocument uses perpetual_retry_time, the task is retried forever (every 6 h at most), on every node of the cluster. We had six such tasks for one account, with 72 to 110 attempts since mid-September.

SearchStore::unindex (crates/store/src/search/index.rs, around line 160–182 in v0.16.20) loads the document’s TermIndex and writes all of its deletions in a single batch:

let mut batch = BatchBuilder::new();
term_index.delete_index(&mut batch, index, SearchIndexId::Account { account_id, document_id });
self.write(batch.build_all()).await?;

For a text-heavy email the clears add up to more than FoundationDB’s 10 MB transaction limit. #2567 on GitHub fixed the same limit on the indexing path, so such a document can be indexed but not removed again. The code is unchanged in v0.16.26.

A secondary effect: unindex returns on the first failing document, and SearchIndexTask::index then marks every delete in the same batch as failed. Other deletions batched together with the large one are held back too.

Expected Behavior

The term deletions are split into several transactions, e.g. with commit_point() / is_large_batch() the same way the indexing path does after #2567, or the document’s term keys are cleared with a range delete if the key layout allows it. The UnindexDocument task then succeeds.

Actual Behavior

The UnindexDocument task fails with FoundationDB error 2101 on every attempt and stays in Retry indefinitely; attemptNumber keeps increasing (74 after 17 days). Each retry logs a store.foundationdb-error on every node, which in our case fired a “FoundationDB errors” alert four times a day. The orphaned term index of the deleted email stays in the store.

Reproduction Steps

  1. Use FoundationDB as data store with the internal search engine.
  2. Deliver an email with a very large text body (or text attachment) so that its term index exceeds ~10 MB of keys, and let it be indexed.
  3. Delete the email and let it be purged.
  4. The UnindexDocument task fails with code 2101 and stays in Retry indefinitely (x:Task/get shows attemptNumber increasing).

Relevant Log Output

ERROR FoundationDB error (store.foundationdb-error) reason = "Transaction exceeds byte limit", code = 2101, causedBy = "crates/store/src/search/index.rs:182", causedBy = "crates/services/src/task_manager/index.rs:308", details = "Failed to delete documents from index", collection = "email"

Task as returned by x:Task/get:

{
  "@type": "UnindexDocument",
  "documentType": "email",
  "status": {
    "@type": "Retry",
    "attemptNumber": 74,
    "createdAt": "2026-09-25T04:12:55Z",
    "failureReason": "Failed to delete documents from index"
  }
}

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

FoundationDB

Blob Storage

S3-compatible

Search Engine

Internal

Directory Backend

Internal

Additional Context

Running v0.16.20 (official release binary in our own container image) on Kubernetes with 2 Stalwart nodes, FoundationDB 7.4.6. The code path is unchanged in v0.16.26. As a workaround we destroyed the stuck tasks via x:Task/set; the orphaned index entries do not show up in search results because Email/query masks hits with the existing messages.

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

This looks like a confirmed bug in the source, not a config or usage issue. In crates/store/src/search/index.rs, the index() path does split large batches via batch.commit_points() / build_one(), but the unindex() path (used by UnindexDocument) still calls batch.build_all() as a single transaction, as you described. On FoundationDB, a term index large enough to exceed the ~10 MB transaction size limit will cause that single write to fail with code 2101 on every retry, since nothing about the failure is transient, and the UnindexDocument task’s perpetual retry policy means it keeps failing forever with the orphaned index left behind.

I’ve verified this matches the current code, not just the version you cited. This is a legitimate defect report; I don’t have the authority to fix or schedule it, but a human maintainer will follow up on this thread given the detailed reproduction and code reference you’ve provided.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

That’s what happens. The term deletes already carry commit points, but unindex writes the whole batch as one transaction and ignores them, so a large enough document goes over FoundationDB’s 10 MB limit every time and the task never completes. Indexing splits correctly, which is why only deletion hits it.

1.0.0 has a rewritten search index where deletes are split across transactions (and indexed text per document is capped), so this doesn’t happen there. It’s due at the end of this month or early November. Until then, destroying the stuck tasks the way you did is safe: the leftover term entries only point at messages that no longer exist, and queries already filter those out.