Another disaster following an update to a new version

Hello dear Stalwart developers,

I don’t have an urgent problem anymore, but I feel the need to share my thoughts. After the major disaster during the update from Stalwart 0.15.x to 0.16.x – which left everything in a non-functional state – I had hoped that such situations wouldn’t happen again during updates. On Monday evening, however, I learned otherwise.

An update from v0.16.27 to the latest version (image: latest) caused my Stalwart mail server (running in Docker) to stop working – yet again. The container failed to start because Stalwart reported that the version of RocksDB being used (1.0.0) was no longer supported. Everything worked fine again once I reverted to v0.16.27.

My attempts to export the data from the old database and import it into a new one were all unsuccessful. None of the suggestions I found online – or those recommended by AI – worked. In the end, my old database became corrupted, forcing me to set up Stalwart Mail from scratch with an empty, new database. Losing all my stored emails is one thing, but having to reconfigure every single setting is quite another. There is a clear need for a simple, version-agnostic export/import function within the WebGUI.

To make matters worse, the WebGUI became inaccessible a day later, returning only an HTTP 503 error. This led to another long troubleshooting session. I eventually discovered that Stalwart had added the IP address of my Caddy reverse proxy – which I use to expose the service – to its blocklist. This is another behavior I never encountered in v0.16.27. Ultimately, I had to add the IP range of my home network – where Caddy is running – to Stalwart’s whitelist.

Everything seems to be working again now, but this is already the second Stalwart Mail update that has ended in chaos. I don’t know why, but I only ever encounter such dramatic issues with Stalwart Mail. You really need to improve your update strategy. It is unacceptable for updates to repeatedly cause such problems, requiring extensive manual adjustments just to get things running again.

I am a private user, not a paying premium customer, so I don’t want to complain too much – but if I were relying on a mail server for stable, reliable business use, I would have reached the point where I’d switch to another provider.

Thank you for your understanding, and I hope you will consider how things could be handled better in the future.

Best regards,
Mic.

1 Like

Sorry, to break it to you, but this is not really a Stalwart problem. It is commonly known that running (important) containers on :latest is not a good idea due to possible breaking changes. Also, should you nonetheless choose to run with :latest, you should at least have backups in place so you can revert to the last working version without data loss.

The same goes for running the migration tool from v0.16.27 to v1.0.0, there you should also have a backup in place so failed migration attempts do not lead to data loss (which is quite curious anyways, given that the migration tool should open the “old” database as read-only).

There is a migration guide from 0.16.27 to 1.0.0, ur post showes u havent read it. Nor the releasenotes of the 1.0 release.

If u run somekind of automated updates on a :latest container, thats a bad idea on any containered projects.

Hello,

Thanks for your feedback, but no, I don’t agree. Using “latest” is a good idea to always be up-to-date without having to constantly check for new releases, and it works fantastically well for 99% of my Docker containers. Occasionally, new features need to be manually re-enabled, but the only time I’ve ever experienced a container completely ceasing to function was with Stalwart-Mail.

Only Stalwart-Mail keeps causing problems.

Migrating from an old to a new version shouldn’t require some questionable tools or guides that only absolute experts can use safely.

So no, I don’t accept this criticism. You should adapt your release strategy to the real world. As I said, it works for others.

Sorry for my words, but that’s my opinion. After the problem occurred twice in quick succession with Stalwart-Mail, I simply had to speak up. Thank you for your understanding.

Migrating from an old to a new version shouldn’t require some questionable tools that only absolute experts can use safely. Stalwart-Mail is actually a really great email server.

Regards,
Mic.

it works fantastically well for 99% of my Docker containers

Just luck. You should never do this.

Best practice, especially for stateful systems, is to use tagged images alongside a tested backup solution. It really forces you to think about: What happens if the server dies right now?

If you want to stay up to date with your software automatically, you can use Renovate or Dependabot to update your compose files. This can run as a nightly job on GitHub or GitLab (which is what I do for my own setups), and it sends you an email notification that includes the changelog so you can see what changed before merging.

That said, I can completely understand the frustration. I was also a bit unhappy to see that Stalwart itself doesn’t perform the migration out of the box. But i see that this would be hard for a distributed system. That`s a tough engineering problem. And the documentation is written good, so everyone can follow the steps.

As the previous commenter already said, using the :latest tag can be dangerous and issues are pre-programmed.

There are projects that use :latest tag for nightlies and dev builds even and many (if not most) projects discourage you from using the :latest tag as exactly the issues you are describing could happen.

I personally would not risk using the latest tag for something that critical as my contacts and mails.

So no, I don’t accept this criticism.

We are just giving you some advice on how to prevent it in the future. Before v1 was released the project also said it’s a pre-stable release and you should use it with caution. I just looked in the docs and there is written you should use stalwartlabs/stalwart:v1 and not the :latest tag.

You should adapt your release strategy to the real world. As I said, it works for others.

The release strategy is following SemVer and that is used by most projects. It’s standard and a good fit. Each project has a different use-case, complexity and co, so you can put all of them in one pod.

I am a private user, not a paying premium customer, so I don’t want to complain too much –

Sorry with all the respect but in my opinion, you are requesting that an amazing project, which is kindly to give out a free community version to change to your specific use case. Nobody is forcing you to use stalwart, there are other solutions available (even free ones). Also if you don’t want to bother about upgrades maybe consider a managed service.

Anyways for your usecase, I think switching to the v1 tag is the prefect fit. It will upgrade to all non-breaking releases. If in the future a v2 is coming, your setup will not automatically update and you need to manually update it (after doing the migration).

PS: Thanks to the Stalwart team for this amazing software and offering it as open source. I myself are working on open source projects and understand what this means.

1 Like

Hello everyone,

I can understand your frustration, and I was well aware that my post here would meet with a fair amount of pushback. Nevertheless, it might be worth considering how to handle future updates more effectively. Such a migration could be broken down into smaller steps across several updates, with the necessary adjustments automated. You might recall the major backlash you faced back when updating from version 0.15.x to 0.16.0 (back when discussions took place on the GitHub forum) – there was a lot of discontent then, too. I had assumed that something like that would never happen again.

Especially with standardized components like a database, an update should never require manual intervention. The update process itself should handle any necessary migrations.

A good example of how automatic updates can be implemented safely is NextCloud. They frequently update their underlying database (MariaDB) without requiring users to follow migration guides or perform manual steps; it all happens automatically in the background. Occasionally, you might need to tweak or add an entry to a configuration file, or install an additional component, but Nextcloud has never reached a state where it stops working entirely after an update.

I realize that others may disagree with me, but perhaps for future updates, you could consider ways to make the process easier for users. Not everyone has a sandbox environment where they can test things out beforehand.

Additionally, it might be helpful to include an export/import function in the web GUI for backing up all settings – one that remains intact even after updates, without requiring further manual adjustments.

That said, I want to reiterate: I still think the Stalwart mail server itself is fantastic.

Thank you for your understanding.

Best regards,
Mic.

No, NextCloud has a complicated 17-step upgrade procedure.

May be… but it is not needed to be used for docker environments. :slight_smile: