Missing entries in config.json after migration script run

Issues
1

Issue Description

When running the migration script I get a configuration that doesn’t contain all database entries. When looking at the generated export.json I see entries that map all stores to the Default entry. My current store configuration looks like this:

storage.blob = "blob-store"
storage.data = "data-store"
storage.directory = "internal"
storage.fts = "fulltext-store"
storage.lookup = "redis-in-memory"
store.blob-store.compression = "lz4"
store.blob-store.path = "/opt/stalwart/stores/blob"
store.blob-store.purge.frequency = "0 3 *"
store.blob-store.read-from-replicas = true
store.blob-store.settings.min-blob-size = 16834
store.blob-store.settings.write-buffer-size = 134217728
store.blob-store.type = "rocksdb"
store.data-store.compression = "lz4"
store.data-store.path = "/opt/stalwart/stores/data"
store.data-store.purge.frequency = "0 3 *"
store.data-store.read-from-replicas = true
store.data-store.settings.min-blob-size = 16834
store.data-store.settings.write-buffer-size = 134217728
store.data-store.type = "rocksdb"
store.fulltext-store.compression = "lz4"
store.fulltext-store.path = "/opt/stalwart/stores/fulltext"
store.fulltext-store.purge.frequency = "0 3 *"
store.fulltext-store.read-from-replicas = true
store.fulltext-store.settings.min-blob-size = 16834
store.fulltext-store.settings.write-buffer-size = 134217728
store.fulltext-store.type = "rocksdb"
store.metrics-store.compression = "lz4"
store.metrics-store.path = "/opt/stalwart/stores/metrics"
store.metrics-store.purge.frequency = "0 3 *"
store.metrics-store.read-from-replicas = true
store.metrics-store.settings.min-blob-size = 16834
store.metrics-store.settings.write-buffer-size = 134217728
store.metrics-store.type = "rocksdb"
store.redis-in-memory.read-from-replicas = true
store.redis-in-memory.redis-type = "single"
store.redis-in-memory.timeout = "15s"
store.redis-in-memory.type = "redis"
store.redis-in-memory.urls = "redis://127.0.0.1/1"
store.tracing-store.compression = "lz4"
store.tracing-store.path = "/opt/stalwart/stores/tracing"
store.tracing-store.purge.frequency = "0 3 *"
store.tracing-store.read-from-replicas = true
store.tracing-store.settings.min-blob-size = 16834
store.tracing-store.settings.write-buffer-size = 134217728
store.tracing-store.type = "rocksdb"

Expected Behavior

All databases are in the new configuration.

Actual Behavior

I get the following config.json:

{
  "@type": "RocksDb",
  "path": "/opt/stalwart/stores/data",
  "blobSize": 16834,
  "bufferSize": 134217728
}

Stalwart Version

v0.15.x

Installation Method

Binary (Linux)

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I agree to follow the project’s Code of Conduct.

on

3

The 0.16.x version doesn’t have anything in config.json except database connection information. Everything is stored in database and you’ll import a snapshot to fill it.

Try following the UPGRADE.md guide, you’ll need to apply the next steps.

Note that you’ll need to recreate many things in the new version since the conversion script doesn’t do that correctly, so it’s a good idea to keep the export.json file on one terminal open while you’re going from each settings page in 0.16.x to reproduce your actual configuration.

4

X-Ryl669 has the shape of it: in 0.16 the bootstrap config.json carries only the data-store pointer, and everything else moves into export.json which the server imports on first boot. So the bare RocksDb object you see is the expected content of config.json on its own.

The interesting question is whether your blob-store, redis-in-memory, and fulltext-store sections made it into export.json or got dropped on the way through. Could you share that file? If those sections are present I can point you at the right import step in the upgrade guide; if they’re missing, that’s a real migration-script bug I want to get fixed.

Note that tracing-store and metrics-store are pointed at by tracer.* and metrics.store rather than storage.*, so they ride a separate path in the script. Worth eyeballing those settings in the new UI to confirm they landed.

5

The only store settings that made it to the `export.json` are those:

{
  "@type": "update",
  "object": "BlobStore",
  "value": {
    "@type": "Default"
  }
}
{
  "@type": "update",
  "object": "InMemoryStore",
  "value": {
    "@type": "Redis",
    "url": "redis://127.0.0.1/1",
    "timeout": 15000
  }
}

Metrics, tracing, and fulltext are missing and the `BlobStore` has lost it’s configuration.
And as additional information the tracing and metrics store lines from the old `config.toml`:

metrics.history.store = "metrics-store"
tracing.history.store = "tracing-store"

So these point to the store.* settings I posted above.

EDIT:

What I forgot to mention is this: When I try to configure a new instance of Stalwart, I can’t re-create the settings I had before because the dropdowns on the storage configuration page don’t contain the RocksDB option.

6

There was a bug in the migration script and it has been fixed. The blob and full-text builders were silently collapsing to the default when they saw a rocksdb or sqlite store, and the metrics and tracing stores had no builder at all, so those entries never made it into the export. The updated script handles all four roles properly and will now error out with a clear message instead of producing a config that quietly loses data.

On the RocksDB option not appearing in the storage dropdowns: that part is actually intentional rather than a bug. In v0.16 the embedded backends (rocksdb and sqlite) are only valid for the data store. Running multiple embedded stores in parallel never gave any real performance benefit, it just increased memory usage and made the storage layout more fiddly to operate, so they were dropped as choices for the blob, search, metrics and tracing stores. Those roles now either share the data store (the new “Default” option, which keeps everything in one rocksdb) or point at an external backend like FoundationDB, PostgreSQL or MySQL, plus filesystem and S3 or Azure for blobs.

If your v0.15 setup had each role pointing at its own separate rocksdb directory, the cleanest path forward is to keep the data store as your existing data rocksdb and let the blob, FTS, metrics and tracing data live in the same store going forward. The historic blob and FTS data in the other directories won’t be readable from v0.16 directly; if you need to preserve old mail bodies, the easiest route is to bring v0.15 back up briefly and re-export using the import/export utils available in the Stalwart binary.

7

Ok, thanks for the explanation. I mostly used the separated config, because of the backups. I didn’t include the tracing and metrics stores there.