When InMemoryStore is a valkey cluster, any slot migration breaks (failover/resharding) breaks the rate limiter

Issues
1

Issue Description

When x:InMemoryStore is a Redis/Valkey cluster, any slot migration
(failover, resharding) permanently breaks the rate limiter on every already-running
Stalwart node. The rate limiter issues a MULTI/EXEC pipeline, and in cluster
mode the MOVED redirects come back inside the EXEC response array rather than
as a top-level error, so redis-rs never refreshes its slot map. The pooled
connection keeps pointing at the old master indefinitely.

Only a restart clears it. Because the failures are intermittent — only keys in the
migrated slot range are affected — the node stays up and serves most requests,
which makes it easy to miss.

AI-disclaimer: i used an AI to help me figure out why things weren’t working, and what was the cause, and to formulate this issue

Expected Behavior

I expect stalwart to follow the redirect given by the valkey cluster, and not continue hammering the wrong shard.

Actual Behavior

redis-rs handles a top-level MOVED by refreshing the slot map and retrying, but
here both redirects are elements of the EXEC reply (Index 0 and Index 1 are the
incr and the expire), so no refresh is triggered.

This is not specific to MOVED. Any cluster-level condition surfaces the same way
and is equally unrecoverable — ReadOnly: You can't write against a read only replica after a demotion, and ClusterDown: The cluster is down while a cluster
is reorganising, both wrapped in the same Transaction aborted and both from the
same call site.

Nothing else recovers it either: the deadpool manager in
crates/store/src/backend/redis/pool.rs only issues PING in recycle, and a
stale-but-healthy node answers that normally, so the connection is never discarded.

Reproduction Steps

  1. Run Valkey in cluster mode with at least 3 shards and 1 replica per shard.
  2. Point x:InMemoryStore at it as RedisCluster, start Stalwart, confirm no errors.
  3. Trigger a failover on one shard — valkey-cli cluster failover on a replica is
    enough; no node needs to die.
  4. Generate traffic that hits the rate limiter (any authenticated request will do).

Relevant Log Output

ERROR Redis error (store.redis-error) listenerId = “http”, reason = “Transaction aborted:
[(Index 0, error: Moved: 8699 10.244.2.250:6379), (Index 1, error: Moved: 8699 10.244.2.250:6379)]”,
causedBy = “crates/store/src/dispatch/lookup.rs:111”,
causedBy = “crates/store/src/dispatch/lookup.rs:317”,
causedBy = “crates/common/src/auth/rate_limit.rs:82”

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

PostgreSQL

Blob Storage

S3-compatible

Search Engine

Elasticsearch

Directory Backend

SQL

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Thanks for the detailed report and repro steps. This matches what’s in the source: the cluster connection manager’s recycle() in crates/store/src/backend/redis/pool.rs only issues a PING, which succeeds even when a node has moved on from a slot range or gone read-only, so a stale connection is never evicted from the deadpool. The rate limiter in crates/common/src/auth/rate_limit.rs does indeed pipeline INCR/EXPIRE inside a Redis MULTI/EXEC, and redis-rs’s cluster client only auto-refreshes its slot map on a top-level MOVED/ASK response, not on ones nested inside an EXEC reply array, exactly as you describe with Transaction aborted: [(Index 0, error: Moved: ...), (Index 1, error: Moved: ...)].

This is a code-level issue rather than a documentation/configuration question, so I can’t point you to a doc that “fixes” it. A human maintainer will need to follow up on whether the fix is upgrading/patching the redis-rs cluster transaction handling, or having Stalwart’s recycle check surface MOVED/ASK/ClusterDown/ReadOnly errors as reasons to discard the pooled connection.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

Fixed in v0.16.23 (yet to be released).