After months of trialing, I’m finally building a new server that will soon go into production. Nothing huge, only 20-30 users, roughly 300-500 GB of data. No clustering necessary for now, just a single machine.
I’m wondering, should I start upfront with storage via S3 on a local Garage service, or should I just store locally for now, wait 'till I have to enable clustering then migrate from local to S3?
Same goes with the database, should I start with Rocks or PGSQL, wait to grow then migrate, or should I start directlywith Foundation (I have no experience with it)?
Thing is, I have no idea if I’ll scale this project or not. It might work very well and I’ll migrate more users, or it might remain at the current planned size for one more years. It mostly depends if the “beta users” will enjoy using the service or not.
What’s your take on this? Introduce a bit of complexity and be prepared, or use the most simple scenario and scale if/when needed?
I can only speak from my experience, but I think you might as well start with a three node cluster.
You’re correct that a single Stalwart instance can handle your current requirements. But there are two downsides. If you skip clustering now, you’re less prepared for doing so if you scale up operations. Secondly, downtime becomes a bigger risk. This may be acceptable, you mention these are beta users. But users won’t enjoy losing access to their mail. You’ll receive negative feedback for an entirely preventable issue.
I currently run a 3 node cluster and a 5 node cluster. Each node is runs an instance of Stalwart, FoundationDB, and NATs on Rocky Linux. I wouldn’t describe the setup as easy. But everything is documented, so it isn’t too hard either. The 3 node cluster can handle 1 failure, the five node cluster can handle two failures.
So, if the cost of running 3 VMs vs. 1 is negligible, I would go with the 3.
Interesting approach. I don’t have a “resource problem”, în the sense that I can use as many resources as necessary, on all fronts.
But I’m careful with administrative efforts, as this will be just “a gig” for the moment, but can become more.
Sure, until 0.16 there was almost no maintenance required, in the sense that updates/upgrades were a breeze.
So I’m curious. If using “clustering ready” components (PGSQL, S3, etc), would a migration be necessary when/if deciding to scale from one host to multiple? Or the mentioned downtime comes from configuring/adding extra hosts?
If you already have redis, pgsql and s3 in place, adding cluster nodes is zero effort. You tell the cluster settings to use the in-memory redis store, copy your master node config.json to the next slave node, and restart the service.
If you don’t need the redundancy of extra nodes up front, then there’s no benefit to setting them up beforehand as they can be added in so fast and easy.
*load balancer / reverse proxy configuration notwithstanding of course, but this is just focusing on the actual server/cluster question.
I totally agree with @Disrupt, clustering Stalwart itself is straightforward. I was partially conflating my experiences clustering FoundationDB and NATs with Stalwart’s setup.
I didn’t mean to suggest clustering would cause downtime, it doesn’t. But VMs can go down for any number of reasons, which is why I suggested putting up a 3 node cluster now since resources are not an issue. But that’s just my philosophy.
Yeah, switching from Rocks to PSQL later would be a right total pain, so start with PSQL and S3-compatible now and you’ll find it goes a long way.
When you want to start clustering, add in redis, or you can compile in the other cluster method, but redis/valkey helps performance too.
Search performance is slated as a goal to be improved using built-in PSQL for 1.0 so I wouldn’t personally even bother with elasticsearch at a small deployment level.
FoundationDB is super overkill for a small deployment.