Redis task locks can be kept alive indefinitely because failed contenders refresh the TTL

Issues
1

Issue Description

When Redis is used as Stalwart’s in-memory store, a task lock abandoned by a
terminated worker may never expire while other task-manager instances continue
trying to acquire it.

Redis lock acquisition increments the lock key and grants the lock only when the
returned counter is 1. The same atomic Redis pipeline also executes EXPIRE
on every acquisition attempt. Consequently, a failed contender that receives
counter value 2 or greater still resets the lock TTL.

In a multi-node deployment, staggered task-manager retries can therefore keep an
abandoned lock alive indefinitely. The corresponding tasks remain pending even
though no worker can acquire them.

This is reproducible in the current release, v0.16.15

Expected Behavior

  • A failed lock contender must not change the existing owner’s lease expiry.
  • If a worker terminates without releasing its lock, the lock must expire at
    the deadline established by the successful acquisition.
  • After expiry, another task manager must be able to acquire and process the
    still-pending task.

Actual Behavior

  • Every failed Redis acquisition increments the key and resets its TTL.
  • Abandoned task locks can remain alive indefinitely in a multi-replica
    deployment.
  • Restarting task processors can produce more failed acquisitions and renewed
    TTLs rather than recovering the tasks.
  • Pending maintenance and account tasks remain blocked until task processing is
    stopped long enough for expiry or the locks are cleared operationally.

Reproduction Steps

The attached Docker Compose reproducer exercises, see: bootstrap.ndjson · GitHub the real Stalwart task-manager
path. It does not seed or modify a Redis lock directly.

It uses the exact application and Redis versions from the incident:

  • stalwartlabs/stalwart:v0.16.7 at digest
    sha256:6a8ddaa5728a5e78a8611085069f63414cd43c3a669471785dd41aad1ca16e63;
  • bitnami/redis:8.0.1-debian-12-r2 at digest
    sha256:333cd28208f4196c2cf9e7a31c9334abdd7fcfa42438c7f6849a8f0d0805a00b.

The reproducer consists of four files:

  • compose.yaml - defines Redis, Stalwart bootstrap/control/worker roles, the Stalwart CLI, and a small TCP service that deliberately keeps the synthetic rules download
    open;
  • config.json - provides the disposable RocksDB metadata store;
  • bootstrap.ndjson - selects Redis as the in-memory store and separates the API-only control role from the task-processing worker role;
  • reproduce.sh - creates a real SpamFilterMaintenance(updateRules) task, observes its binary KV_LOCK_TASK Redis key, terminates the lock owner, starts a replacement
    worker, and asserts the task remains pending.

Run it on a machine with Bash, curl, Docker, and Docker Compose:

chmod +x reproduce.sh
./reproduce.sh

The script uses the fixed Compose project name stalwart-lock-repro, binds the
temporary Stalwart HTTP listener only to 127.0.0.1:18080, and removes its
containers, network, and volume on exit.

The sequence is:

  1. Bootstrap a disposable Stalwart v0.16.7 instance using Redis as its
    in-memory store.
  2. Create a due SpamFilterMaintenance(updateRules) task through Stalwart’s
    management API.
  3. Start a task-processing Stalwart worker. Its rules request is intentionally
    kept open so that it remains inside task execution after acquiring the lock.
  4. Read the task’s Redis lock and verify counter 1 with a TTL close to the
    one-hour DEFAULT_LOCK_EXPIRY.
  5. Send SIGKILL to that Stalwart worker before it can call
    remove_index_lock.
  6. Start a replacement Stalwart worker before the abandoned lease expires.
  7. Verify that the failed contender changes the counter from 1 to 2 and
    moves the TTL back toward one hour.
  8. Query the task through Stalwart’s management API and verify that it remains
    Pending

Relevant Log Output

Created Task i0cfketiaaab
before_crash=1,3599675
after_owner_crash=1,3595872
after_contender_restart=2,3599426
task={“maintenanceType”:“updateRules”,“status”:{“createdAt”:“2026-07-29T11:29:11Z”,“due”:“2026-07-29T11:29:11Z”,“@type”:“Pending”},“@type”:“SpamFilterMaintenance”,“due”:“2026-07-29T11:29:11Z”,“id”:“i0cfketiaaab”}
REPRODUCED: failed Stalwart contender incremented the abandoned lock and reset its TTL.

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

FoundationDB

Blob Storage

S3-compatible

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Your analysis matches the current source. In crates/store/src/dispatch/lookup.rs, try_lock calls store.key_incr(key, 1, duration) for the Redis backend and treats the lock as acquired only when the returned counter is 1. But key_incr (in crates/store/src/backend/redis/lookup.rs) always executes an atomic INCR + EXPIRE pipeline whenever an expiry is passed, with no conditional check on the returned counter value:

async fn key_incr_(...) -> trc::Result<i64> {
    if let Some(expires) = expires {
        redis::pipe()
            .atomic()
            .incr(key, value)
            .expire(key, expires as i64)
            .ignore()
            .query_async::<Vec<i64>>(conn)
            .await
            .map_err(into_error)
            .map(|v| v.first().copied().unwrap_or(0))
    } else {
        conn.incr(key, value).await.map_err(into_error)
    }
}

So any failed contender (counter returned > 1) still resets the TTL on the shared lock key, exactly as you described, since the EXPIRE is unconditional inside the same atomic pipeline used for the increment. This confirms the bug as reported at the code level; a human maintainer will need to follow up on a fix (e.g. only setting the expiry on first acquisition via a Lua script or SET NX PX pattern instead of INCR + unconditional EXPIRE).

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.