JMAP for upgrade to 0.16

Your question

I’m preparing for an upgrade from 0.15.x to 0.16. Stalwart is running as docker container. As far as I understand, I should start a 0.16 instance in a fresh container and use another docker image to migrate the settings and accounts to the new container.

I guess, that this migration uses JMAP. But I wasn’t able to get JMAP running in my instance.

Now, that I’m able to run the Web Admin interface (see: WebAdmin UI is not reachable from outside the container - #14 by stalwart) I hope, that I can run JMAP also. As far as I understand, both the UI and Jmap share the same listener.

If I run
curl 10.8.1.1:8181
I get the login dialog of the admin UI. Port 8080 of the container is mapped to port 8181 on the host.

If I run curl 10.8.1.1:8181/jmap I get the following output:

{“type”:“about:blank”,“status”:404,“title”:“Not Found”,“detail”:“The requested resource does not exist on this server.”}

To me, the response in JSON format seems to indicate that the request is reaching the listener. What is missing from these requests to get a meaningful response from JMAP?

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

Good news: you don’t need to test JMAP manually on your current 0.15.x server at all. The whole flow described in the official upgrade guide works like this:

  1. On your current 0.15.x server, you run the migrate_v016.py helper script (downloaded from the Stalwart repo). It talks to your live 0.15.x server’s existing management API using your admin credentials (dump subcommand) and produces settings.json/principals.json, then convert turns those into config.json and export.json. You never need to hit /jmap on the 0.15 server yourself; the script handles the correct endpoints internally.
  2. JMAP only becomes relevant on the new v0.16 instance, and only once it’s started in recovery mode (STALWART_RECOVERY_MODE=1), where it exposes the management API on port 8080. That’s when you run stalwart-cli apply --file export.json against http://<host>:8080 using the temporary STALWART_RECOVERY_ADMIN credentials — the CLI talks JMAP under the hood, you don’t need to curl it directly.

So the 404 you got from curl 10.8.1.1:8181/jmap isn’t something you need to fix on the old container — that endpoint isn’t part of what the 0.15→0.16 migration process expects you to query manually at all.

Full step-by-step instructions (including Docker-specific volume/path handling) are here: stalwart/UPGRADING/v0_16.md at main · stalwartlabs/stalwart · GitHub

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

Thanks for the reply.

If I understand correctly, I start a shell inside the container using docker exec -it stalwart-mail /bin/bash, then download the migrate_v016.py script inside the container and run it.

But how do I migrate the accounts, including the emails stored there? Are these part of export.json?

As far as I understand, you need at least valid credentials, either a bearer token, or user/password using basic auth.

For /jmap more may be required. I was just wrong about something like this myself, as I assumed /.well-known/jmap would respond to unauthenticated GET requests. The JMAP RFC mentions authenticated GET requests for that one.

@mixis Thanks for your reply. I tried the call with -u username:passwd without success. I also tried some sample calls I found in the net with the capabilities query or echo. The answer is always the same.

Another pitfall for me was, that curl was requesting HEAD

curl -u user:pass -X GET https://example.org/.well-known/jmap -I 

should show you a 307 redirect to /jmap/session, where you will then find the information for the endpoint.

/jmap accepts POST only, so a GET returns 404 whatever credentials you send; -u could never have helped there. The session resource is GET /jmap/session with basic auth or a bearer token, and /.well-known/jmap 307-redirects to it. mixis is right on both points, including that a plain -I sends HEAD and does not match the route.

On what the script moves: migrate_v016.py handles settings and principals only, so accounts, groups, domains, tenants, roles, API keys and OAuth clients. Mail, calendars, contacts and Sieve scripts are not in export.json. config.json becomes the new server’s configuration and export.json is loaded with stalwart-cli apply --file export.json.

It is an HTTP client against the old server’s management API, so run it wherever you have Python; it does not need to be inside the new container. For Docker, --patch-paths /opt/stalwart=/var/lib/stalwart rewrites the old paths for you.

Account data moves with Vandelay, per account: Overview | Stalwart, and the full 0.15 to 0.16 procedure is at Overview | Stalwart. Since you publish 8080 as 8181, set STALWART_PUBLIC_URL as well, or the session resource will advertise URLs on the wrong port and clients will not be able to follow them.

As far as I can see, the export via the migration script worked. The source instance uses RocksDB, but the new instance is supposed to use a database—preferably MariaDB.

Am I correct in thinking that I can install the new instance in a new container and then run the migration script on it, so that the data end up in MariaDB?

Is there a way to use the host system’s MariaDB instance?

Yes to the plan, with one correction to the assumption under it: the migration script does not pick the backend. convert writes your old server’s store into config.json, so a RocksDB source gives you a RocksDB config.json no matter which container you run it against.

The data store is the one setting that lives only in that file and cannot be changed through the API, since the API is itself served out of the datastore. So either edit the DataStore block to the MySql variant (host, port, database, authUsername, authSecret) before the new server first starts, or let the setup wizard run and choose MySQL there, then discard the generated block. If your 0.15 pointed blob, in-memory and search at the same RocksDB store, the script writes those three out as Default, meaning “same as the data store”, so that single edit moves all four. The export files themselves are backend independent: stalwart-cli apply --file export.json lands them wherever the data store happens to point.

Host MariaDB works. It is Docker networking, not a Stalwart setting: bind MariaDB beyond loopback, grant from the container’s subnet, and reach it as host.docker.internal (adding --add-host=host.docker.internal:host-gateway on Linux).

One small warning if you were also planning to use MySQL for search: no stemming, one language per index, and words under three characters are not indexed. MySQL / MariaDB | Stalwart has the details. For mail a dedicated search backend is a better fit.