I’m considering changing the Object storage provider from S3 to Filesystem, which leaves me with a few questions.
Is it even possible to migrate between two different Blob storage providers? Downtime would be acceptable.
The documentation says the Filesystem store is “suitable for single-node installations”, but is it possible to use for multiple nodes provided that you have a shared filesystem mount?
My plan is to access the Blob store through the Filesystem provider, using an S3 Mountpoint on each node.
“That sounds insane, why would you possibly want to do that instead of the native S3 support?”, you may ask.
The answer is that Stalwart is currently basing it’s S3 support on rust-s3, which does not appear to be maintained anymore. It has not have any releases for almost an entire year, have several unaddressed issues (even small dependency issues to solve security vulnerabilities in dependencies), and does not work with any S3 implementations that strictly follows the AWS SigV4 spec (which ironically doesn’t appear to include AWS themselves): Bug Report: S3 'Purge blob store' task fails with 403 SignatureDoesNotMatch on Cloudflare R2
At this point I don’t really expect Stalwart to completely replace the S3 package, and I understand this would likely be a significant change, so I am hoping it’s possible to work around it.
Yes, and you don’t need export and import for it. The S3 backend stores each blob as keyPrefix followed by the base32 of its hash, and the filesystem backend uses the same file name, just with depth levels of hex subdirectories in front. Set depth to 0 and the names match the S3 keys exactly. So: copy the bucket into a local directory with rclone copy or aws s3 sync, stop the server, run a final sync, switch the blob store to FileSystem with path pointing at that directory and depth 0, and start it again. If you set a keyPrefix, point path at the matching subdirectory instead; that only lines up when the prefix ends in /. Open a few old messages before deleting anything from the bucket. Field reference: BlobStore | Stalwart
Several nodes on one filesystem store is fine as long as the mount behaves like POSIX. Blobs are content-addressed and nothing is locked, and the single-node remark in the docs is about local disks. With mountpoint-s3 you’ll need --allow-delete or the blob purge fails, possibly --allow-overwrite too, and every miss costs a round trip to the bucket. It also signs its own SigV4 requests, so test it against R2 before counting on it to avoid the rust-s3 problem.
Thank you for the response, that sounds promising!
However, I might not need to try it as within 24 hours of making this post the rust-s3 maintainer has acknowledged the issue, merge a PR that should fix it, and released a new version so it also seems promising that this issue will be resolved anyway.