Stalwart Cluster installation with FoundationDB & Issues

Issues
1

Issue Description

Hi,
I decommissioned the old single node Stalwart mail server and deployed a five nodes HA cluster Stalwart with foundationdb. Added all five nodes and configured the cluster cordinatior using Zenoh peer-to-peer with the following configuration string successfully. “{
“@type”: “Zenoh”,
“config”: “{\n mode: peer,\n scouting: {\n multicast: {\n enabled: true,\n address: "224.0.0.224:7446",\n interface: "auto",\n autoconnect: { router: , peer: ["router", "peer"] },\n listen: true,\n },\n },\n}”
}”
Copied as it is from the documentation. I also enable {
“@type”: “Zenoh”,
“config”: “{\n mode: peer,\n scouting: {\n multicast: {\n enabled: true,\n address: "224.0.0.224:7446",\n interface: "auto",\n autoconnect: { router: , peer: ["router", "peer"] },\n listen: true,\n },\n },\n}”
} "
I also, edited the stalwart.env and enabled "STALWART_ROLE=primary on one of the node as a test.
However, I am experiencing the following three issues:-

  1. The nodes kept going in “Stale” state. Restarting the Stalwart put it back to active and few minutes later it goes into the stale state.
  2. I have an 150 Enterprise Licenses and active. However, addting the prevous unused license fails to retain in Stalwart and activate the Enterprise features though it said it is installed successfully.
    3.I also am having difficulty getting SSL certificate. The ACME provider is configured to use RFC2136 TCP and TLS is set to use DNS-01-Persistent

Any help or guide is highly appreciated. Thank you.

Expected Behavior

Stalwart should be able to retain the Enterprise license and activate the features. Second, the nodes should not be able to go into stale state unless I forgot to do something else that I am not aware of. Third, Stalwart should be able to obtain SSL certs without any issue using RFC2136 protocol. I have seen issuses with Certbot and RFC2136 protocol to obtain SSL certificate in the past with Nginx Proxy Manager. However, I am not sure what Stalwart uses to request certs.

Actual Behavior

  1. The nodes kept going in “Stale” state. Restarting the Stalwart put it back to active and few minutes later it goes into the stale state.
  2. I have an 150 Enterprise Licenses and active. However, addting the prevous unused license fails to retain in Stalwart and activate the Enterprise features though it said it is installed successfully.
    3.I also am having difficulty getting SSL certificate. The ACME provider is configured to use RFC2136 TCP and TLS is set to use DNS-01-Persistent

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

FoundationDB

Blob Storage

FoundationDB

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

How difficult was it to setup?

Once you have been running for a while I would love to hear how it’s been working out for you. I will also purchase 150 licenses and plan on buying the enterprise lic. I host email for a few clients mainly so that it just works and when it breaks I’m the guy fixing it. Dealing with Godaddy support is a royal pain. Currently can’t get them to remove a BLOCK on outbound mail for a client that got a trojan and spammed a few of the customers the client had. Low volume.

So far 10 calls to support, many periods of it’s fixed just wait 24 hours. Support not understanding the problem.

I have some really nice servers in my colocation so running 3 or 5 virtual machines won’t be a problem. The thought of having the redundancy just sounds nice as I’m a solo IT guy and often out of the office. I just want something that works. Currently running postfix and dovecot and they have been great, no issues with it but it’s time to upgrade the server and Stalwart looks pretty nice. It’s either stalwart or another dovecot/postfix server by the end of the year.

3

Stalwart is very stable mail solution with a lot of protection buildin from ground up. FoundationDB is an amazing solution as the back-end storage for Stalwart Cluster. It is very stable and simple. However, getting documentation was not easy. Spend almost three weeks learning and gather information. I still yet to organize my notes. However, I am still experiencing three issues related to the Stalwart HA Clustering, Enterprise license installation and obtaining SSL certificate. The SSL certs issue has to do with the front end proxy and do not thing it has anything to do with Stalwart. “X-Forwarded-For), while Stalwart expects the layer-4 Proxy Protocol prefix. Because the incoming request starts with GET / ... instead of the Proxy Protocol signature (PROXY ...), Stalwart drops the connection and logs invalid proxy header.”

4

Thanks for the quick response. It’s looking like it’s pretty stable. I think it’ll work fine for my use case and the cost is very reasonable. Hope you get it all sorted out. What proxy are you using? You mentioned NGINX above so I assume that.

Besides Stalwart I’ve got to pick up a few more skills it seems like proxies but I think in the end it’ll be worth it.

Once you sort it all out if you get a chance please drop a quick howto/watch out for post if you can. I understand if you don’t have the time.
Thanks again.

5

No worries, I will be posting it because I know there are many who can use that knowledge but at present time I am under water and need to get this issue resolved asap.

Troubleshooting the Stale state on the cluster nodes I observed the following message form the log. " shutting down Stalwart Server + causedBy = “SIGINT”". Something is sending that a command to terminate. I hope someone in here has the answer.

6

The stale nodes and the coordinator are the same problem. Neither published Linux binary is compiled with the zenoh feature, so a Zenoh coordinator is rejected at startup and the node runs without one. Check your config-errors panel for “Binary was not compiled with the selected coordinator backend”. Switch the coordinator to NATS or Redis, both of which are in the FoundationDB build, or compile with --features zenoh.

Stale follows from that on its own: the node-id lease renewal is only scheduled when a coordinator is enabled, so lastRenewal never advances after startup and the node flips to stale an hour later, then inactive after a day. Restarting rewrites the timestamp, which is the cycle you are seeing. Fix the coordinator and this goes away.

Also unset STALWART_ROLE, or create the role it names. It has to match a ClusterRole object that already exists in the registry; there is no built-in “primary”, and primary/secondary is not a concept here. You should see “Cluster role “primary” not found in registry” in the log.

On the licence: it is validated against the registered domain of your defaultHostname setting, not the machine hostname, and a mismatch is only logged as a warning, which is why it looked like it installed. Make sure defaultHostname is under the licensed domain, reload the config, then log out and back in.

For item 3 as you restated it, that is the listener’s PROXY protocol setting, not ACME. proxyTrustedNetworks has to list your proxy’s network for the PROXY prefix to be expected; otherwise terminate proxy-protocol at the proxy instead. If you did select DNS-PERSIST-01, that is an unratified draft and needs CA support that Let’s Encrypt does not have, so use DNS-01 with a Tsig DnsServer for RFC 2136.

The SIGINT is not coming from Stalwart; we only log which signal arrived. systemd sends SIGTERM by default, so something else is stopping the process. Post the unit file and journalctl -u stalwart -o short-precise around one of the shutdowns, and confirm which artifact you installed, since a self-compiled build would change my first answer.

7

Very valid points. Let me try to adjust my settings and re-test. I will update here with the result. I wish there is a paid consultant service that I would love to hire and get this thing going. lol! If you know anyone who is a consultant please share my contact or give me there contact. Thank you.

8

One more thing about the licence that I forgot to mention. The hostname mismatch is not the only way it goes quiet: if the account count is over the licensed seats you get a warning in the same place, “License key is valid but only allows N accounts, found M.”, and the outcome is identical, Enterprise features just stay off. At 150 seats, check that alongside defaultHostname. Either way the key is only ever validated at startup and on reload, never when you save it, which is why the UI told you it installed.