Cluster nodes showing as stale - why?

Questions
1

Your question

I added one more node to my cluster, and both the main node as the remote one showed green. Then, after a while, everything showed as Stale.

Reading the documentation, it is said that:

Stalwart supports two coordination modes. The default is a lightweight peer-to-peer mode that requires no central coordinator or dedicated server.

However, on other pages, it hints that one needs Zenoh.

So, do I need to use Zenoh, or will the cluster work as is out of the box?

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Here is what the documentation says clearly:

The peer-to-peer (default) mode IS Zenoh – and it is not included in the standard binary. That is almost certainly why your nodes are going stale: without an active coordination mechanism, nodes cannot exchange heartbeats or state, and the cluster UI marks them as stale.

Key points:

  1. “Peer-to-peer” = Zenoh. The docs describe both as the same thing. The cluster overview diagram even shows “Zenoh / Kafka” as the coordination layer. There is no coordination mechanism that works with zero dependencies.

  2. Zenoh is not compiled into the default Stalwart binary. The docs state explicitly:

    Peer-to-peer coordination is not included in the default Stalwart binary. To enable it, build Stalwart with the zenoh feature flag.

  3. Your options: Either build Stalwart yourself with the zenoh feature flag for peer-to-peer mode, or configure one of the other supported backends – Apache Kafka, NATS, or Redis – which do not require a custom build.

Without a working coordination backend in place, nodes will appear green on first contact (they can still see the shared storage) but will drift to “stale” because there is no live inter-node messaging to keep them synchronised.

Relevant docs:

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

The node might show as stale if your hostname changes constantly or if there is no coordinator configured.

Note that a stock binary does not include Zenoh support. To get coordination working, configure a coordinator backend the binary supports (Redis, NATS, or Kafka) under the cluster settings, and make sure every node uses the same one. If you specifically want peer-to-peer, you would need a build with the zenoh feature enabled.

4

@stalwart I just added `zenoh` to the list of features on the Dockerfile, and it broke building the docker container. I’m trying on 0.16.8

5

Anything I could do to fix this one?

6

Your Docker build breaking when you added zenoh is a build-features problem (zenoh pulls extra system dependencies); unless you specifically need P2P I’d take the Redis/NATS/Kafka route rather than a custom zenoh build. If you do want zenoh, send the exact build error. It’s also worth checking each node has a stable hostname/node-id, since hostname churn shows up as Stale too.