Working Zenoh Settings for P2P Cluster with 2 or more nodes

General
1

Sharing my working settings for Zenoh with a workaround for it to work with two nodes that are not on same LAN (no multicast or broadcast).

{
  mode: "peer",
  connect: {
    endpoints: ["tcp/zenoh-endpoint:7447", "udp/zenoh-endpoint:7447"],
  },
  listen: {
     endpoints: ["tcp/[::]:7447", "udp/[::]:7447"]
  },
  scouting: {
    gossip: {
      enabled: true,
      multihop: false,
      autoconnect: { router: [], peer: ["router", "peer"] }, 
    },
  },
}

The workaround: as Stalwart has just one shared configuration for Zenoh and as it can’t have its own address on the connect endpoints because it crashes Zenoh, I added a zenoh-endpoint to the /etc/hosts point to the another host, after it connects it should grab extra nodes using gossip protocol (I didn’t tested it).

Please note that zenoh is not included in the default binary, you must compile it yourself with: cargo build --release --features=zenoh,other-features, where other-features is the features you need (eg.: s3, mysql, postgres, enterprise, etc)

I am looking forward regarding critique to this solution and possible improvements.

2

Hi,

Would you mind sharing /etc/hosts configuration? I did not quite understand how you added zenoh-endpoint to /etc/hosts. Thank you.

3

I also installed zenoh on ubuntu using apt install but not with Stalwart binary

4

One more question. Is the other machine not part of the Stalwart cluster but a separate machine where Zenoh is running? Thank you.

5

IP of Host A: 172.17.30.5
IP of Host B: 172.17.30.6

/ect/hosts for host A:

172.17.30.6 zenoh-endpoint

/ect/hosts for host B:

172.17.30.5 zenoh-endpoint
6

I appreciate your reply. Are the Stalwart servers installed on a bare-metal machines using different host names and IP’s or all machines have the same host name (zenoh-endpoint) but different IP addresses for each server? In my case I have five Stalwart servers with different names and IP’s on each bare-metal servers (192.168.0.1 m1, 192.168.0.2 m2, 192.168.0.3 m3, 192.168.0.4 m4, 192.168.0.5 m5) also set in /etc/hosts. I also installed zenoh on the same Stalwart bare-metal servers manually using “apt install zenoh” in ubuntu. Thank you.

7

FYI: If you installed Stalwart with FoundationDB Redis comes with the binary and easy to setup that up.

8

zenoh-endpoint is just a made up name and not the hostnames of the machines, it just a way to have a name so I can set it on configuration and use different IPs for each machine.

I need to do this because I am not on same VPC (I am using Scaleway).

It just need to be a connection point between the first two machines of the cluster, according to zenoh docs, other machines should be discovered using gossip (where machines tell each other about nodes they know).

On your case, you can set for /etc/hosts:

On m1:

192.168.0.2 zenoh-endpoint

On m2:

192.168.0.1 zenoh-endpoint

For the other machines you can use either of two.

If the Gossip protocol doesn’t work as expected, you can set in each machine the name of the other, eg:

On m1:

192.168.0.2 zenoh-endpoint-1
192.168.0.3 zenoh-endpoint-2
192.168.0.4 zenoh-endpoint-3
192.168.0.5 zenoh-endpoint-4

On m2:

192.168.0.1 zenoh-endpoint-1
192.168.0.3 zenoh-endpoint-2
192.168.0.4 zenoh-endpoint-3
192.168.0.5 zenoh-endpoint-4

And use a list as this for the connection:

endpoints: [
  "tcp/zenoh-endpoint-1:7447", "udp/zenoh-endpoint-1:7447",
  "tcp/zenoh-endpoint-2:7447", "udp/zenoh-endpoint-2:7447",
  "tcp/zenoh-endpoint-3:7447", "udp/zenoh-endpoint-3:7447",
  "tcp/zenoh-endpoint-4:7447", "udp/zenoh-endpoint-4:7447",
],

But I think this it starts to became too much complicated and a Redis may be more suitable if you have some spare memory.

9

I run it with MySQL Galera on Debian with 2GB VPS, I build the binaries with zenoh support, my setup is using about 512MB of RAM when idle.

Please not that for running it zenoh doesn’t need to be installed. Zenoh is a library that is incorporated when you build it with zenoh support and no servers are need.

You may use the zenoh binary to test your configuration.