I added one more node to my cluster, and both the main node as the remote one showed green. Then, after a while, everything showed as Stale.
Reading the documentation, it is said that:
Stalwart supports two coordination modes. The default is a lightweight peer-to-peer mode that requires no central coordinator or dedicated server.
However, on other pages, it hints that one needs Zenoh.
So, do I need to use Zenoh, or will the cluster work as is out of the box?
I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.
The peer-to-peer (default) mode IS Zenoh – and it is not included in the standard binary. That is almost certainly why your nodes are going stale: without an active coordination mechanism, nodes cannot exchange heartbeats or state, and the cluster UI marks them as stale.
Key points:
“Peer-to-peer” = Zenoh. The docs describe both as the same thing. The cluster overview diagram even shows “Zenoh / Kafka” as the coordination layer. There is no coordination mechanism that works with zero dependencies.
Zenoh is not compiled into the default Stalwart binary. The docs state explicitly:
Peer-to-peer coordination is not included in the default Stalwart binary. To enable it, build Stalwart with the zenoh feature flag.
Your options: Either build Stalwart yourself with the zenoh feature flag for peer-to-peer mode, or configure one of the other supported backends – Apache Kafka, NATS, or Redis – which do not require a custom build.
Without a working coordination backend in place, nodes will appear green on first contact (they can still see the shared storage) but will drift to “stale” because there is no live inter-node messaging to keep them synchronised.
The node might show as stale if your hostname changes constantly or if there is no coordinator configured.
Note that a stock binary does not include Zenoh support. To get coordination working, configure a coordinator backend the binary supports (Redis, NATS, or Kafka) under the cluster settings, and make sure every node uses the same one. If you specifically want peer-to-peer, you would need a build with the zenoh feature enabled.
Your Docker build breaking when you added zenoh is a build-features problem (zenoh pulls extra system dependencies); unless you specifically need P2P I’d take the Redis/NATS/Kafka route rather than a custom zenoh build. If you do want zenoh, send the exact build error. It’s also worth checking each node has a stable hostname/node-id, since hostname churn shows up as Stale too.
Ok, I’ll send the building error.
The reason why I want to go this route is that I want to have as few components as possible that are cluster-dependent. TBH, I didn’t want to have this as a cluster at all - I want redundancy so that one server could take over if the other host fails or is down.
In order to make Stalwart to work, I had to have a cluster-based database (Foundation), and a cluster storage (Garage), and that required three-nodes, when I only wanted two nodes just for the case one server is down.
Fortunately for me, it seems that Stalwart is smart enough to handle downtime of a nearby component (db or storage) so that the mail ends up being redirected to a lower priority MX. But I still have to maintain a three nodes of db and storage.
I’m very grateful for all the work that was done to Stalwart, and I’m aware that it is my problem that it doesn’t implement things the way I’m used to, and also I’m aware it is way more modern for current infrastructure patterns.
But it would be so cool to be able to have geographically independent servers which are not dependent on shared infrastructure so they could still replicate among themselves.
But this was a huge digression. I will provide the logs.
It is a bit odd to declare zenoh the default even though it’s not available in the default binaries.
The documentation would be a bit less confusing if it stated that by default no clustering is available and your options are to build the binary with zenoh for built-in clustering or to use an external service.
@zeeZ the docs need to be corrected, saying that Zenoh is the default is wrong. Zenoh is not included by default as it considerably increases the binary size and, based on user feedback, most large deployments prefer on Redis, NATS or Kafka.
@oculos, on your wider point: two geographically separate nodes replicating between themselves with no shared infrastructure is not something the current architecture does, so you are not missing a setting. Coordination decides which node runs which task; it does not replicate data, which is why you ended up with FoundationDB and Garage underneath.