"db error" when running bootstrap with PostgreSQL

Issues
1

Issue Description

I’m trying to run the Stalwart bootstrap using PostgreSQL, however whenever I attempt to do so, I get a very generic error message that does not really give me any information about what specifically is going wrong:

Failed to initialize data store: PostgreSQL error (store.postgresql-error): reason = Error occurred while creating a new object: db error

I’m able to connect to the database from the same server using the same connection details and I’m able to create tables, etc. However the database is otherwise empty because Stalwart’s bootstrap never appears to actually be able to get to that point.

I cannot find any knob to tweak to get more information out of Stalwart about what exactly is failing.

If it helps at all, this is running on a Digital Ocean Droplet with their managed PostgreSQL database.

Expected Behavior

I would either get a helpful error message that would point to what the issue is, or it would create the needed tables.

Actual Behavior

The bootstrap gives the error

Failed to initialize data store: PostgreSQL error (store.postgresql-error): reason = Error occurred while creating a new object: db error

Reproduction Steps

  1. Setup a DO Droplet and PostgreSQL Instance
  2. Attempt to bootstrap Stalwart using the above.

Relevant Log Output

I don’t know how to get more logging out of Stalwart before bootstrapping.

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

PostgreSQL

Blob Storage

S3-compatible

Search Engine

PostgreSQL

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Let me research this issue to provide accurate guidance.

Based on my research, here is what’s likely happening and how to investigate:

The most probable cause for DigitalOcean managed PostgreSQL is a TLS/SSL mismatch. DO’s managed PostgreSQL requires SSL connections, and Stalwart’s PostgreSQL backend has useTls: false by default. Without TLS enabled in Stalwart’s DataStore config, the connection attempt fails at the driver level with the opaque db error message.

What to check/try:

  1. Enable TLS in your DataStore configuration. In the bootstrap DataStore object (type PostgreSql), set useTls: true. If the managed DB uses a self-signed or private CA certificate, you may also need allowInvalidCerts: true to bypass strict certificate validation (test only – prefer trusting the correct CA in production).

  2. Check the options field. DigitalOcean managed PostgreSQL may require sslmode=require or sslmode=verify-full passed as a connection option. Use the options field on the DataStore object to pass these driver-level parameters.

  3. Get more log detail. There is currently no way to get additional detail before bootstrap completes, but you can try running Stalwart with increased log verbosity (RUST_LOG=debug or similar env var) to see if the underlying tokio-postgres/sqlx driver emits a more specific error before Stalwart wraps it.

  4. Verify the database user has CREATE privileges. Bootstrap needs to create tables, indexes, and sequences. Ensure the connecting user has at minimum CREATE ON DATABASE and is the owner of the target database on the managed instance – DO sometimes restricts superuser and certain DDL operations for managed DB users.

Relevant documentation: PostgreSQL backend configuration – see the TLS section specifically.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

First try the bootstrap GUI with the allowInvalidCerts as true, or edit the option in your config.json

That will confirm it is a TLS issue if it moves forward.

If DO is anything like AWS RDS, then you will need to import the cloud provider certificate bundle into the system.

I Googled DO and TLS and a resource didn’t jump out at me, so I’ll give you my steps for AWS RDS and you can apply to any other provider.

  1. Using SSL/TLS to encrypt a connection to a DB instance or cluster - Amazon Relational Database Service
  2. Grab either the global bundle or the region specific bundle as appropriate
  3. Save to /usr/local/share/ca-certificates/ but change the extension from .pem to .crt or it won’t import
  4. update-ca-certificates and verify 1 cert was added
  5. Proceed with Stalwart bootstrap

Hope that helps.

4

Ah sorry, I forgot to mention I did have the allow invalid certificates set to true already, because they’re not using a trusted certificate. The error message you get with an invalid certificate pretty reasonably indicates it’s a TLS problem, but this happens with the invalid TLS certs set to true.

If it helps at all, this is the bootstrap data that the UI is sending (with some redactions):

{
    "using": [
        "urn:ietf:params:jmap:core",
        "urn:stalwart:jmap",
        "urn:ietf:params:jmap:blob",
        "urn:ietf:params:jmap:mail",
        "urn:ietf:params:jmap:calendars",
        "urn:ietf:params:jmap:contacts",
        "urn:ietf:params:jmap:principals",
        "urn:ietf:params:jmap:sieve",
        "urn:ietf:params:jmap:vacationresponse"
    ],
    "methodCalls": [
        [
            "x:Bootstrap/set",
            {
                "accountId": "d333333",
                "update": {
                    "singleton": {
                        "blobStore": {
                            "@type": "S3",
                            "maxRetries": 3,
                            "timeout": 30000,
                            "verifyAfterWrite": true,
                            "region": {
                                "@type": "DoNyc3"
                            },
                            "secretKey": {
                                "@type": "Value",
                                "secret": "<redacted>"
                            },
                            "securityToken": {
                                "@type": "None"
                            },
                            "sessionToken": {
                                "@type": "None"
                            },
                            "bucket": "<redacted>",
                            "accessKey": "<redacted>"
                        },
                        "dataStore": {
                            "@type": "PostgreSql",
                            "authUsername": "stalwart",
                            "database": "stalwart",
                            "poolMaxConnections": 10,
                            "poolRecyclingMethod": "fast",
                            "port": 25060,
                            "timeout": 15000,
                            "authSecret": {
                                "@type": "Value",
                                "secret": "<redacted>"
                            },
                            "host": "private-<redacted>.j.db.ondigitalocean.com",
                            "useTls": true,
                            "allowInvalidCerts": true,
                            "options": null
                        },
                        "directory": {
                            "@type": "Internal"
                        },
                        "dnsServer": {
                            "@type": "Manual"
                        },
                        "generateDkimKeys": true,
                        "inMemoryStore": {
                            "@type": "Default"
                        },
                        "requestTlsCertificate": true,
                        "searchStore": {
                            "@type": "Default"
                        },
                        "tracer": {
                            "@type": "Log",
                            "ansi": true,
                            "enable": true,
                            "eventsPolicy": "exclude",
                            "level": "info",
                            "prefix": "stalwart",
                            "rotate": "daily",
                            "path": "/var/log/stalwart/",
                            "multiline": false,
                            "lossy": false,
                            "events": {}
                        },
                        "serverHostname": "mail.<redacted>.com",
                        "defaultDomain": "<redacted>.com"
                    }
                }
            },
            "0"
        ]
    ]
}

I’ve verified the credentials work with psql from the same VPS that stalwart is running on, and that user is owner of the database and can create tables, etc.

5

So it’s down to what is different from running it command line vs the Rust library connector.

Is there any indication that any data is being written at all? Or is the database still completely empty.

You’ve checked that you can connect to the DB from the same VPS, and permissions are set, but did you create anything while connected?

Since I’m not familiar with DO and their methods, I’ve done some reading as they use pgBouncer as a cluster proxy. Apparently there are some operations that don’t get along with the proxy and there is a recommendation that you create your schemas using the direct port, then move back to the pooler port after. So 25061 instead of 25060.

It could be worth a shot, then there would be something to report back as an issue.

6

Completely empty.

I created a random table and then dropped it using psql.

pgBouncer is an optional addon and I don’t have it enabled.

I haven’t looked closely at the code, but I’m familiar with Rust, so I’ll try to clone the repo and throw some print statements in there to see if I can get some more information out of it.

I assume it’s getting a tokio_postgres::DbError, which should contain information from the database about what the error is. If so it would be nice if that data didn’t get lost, but I’ll see what I can find here.

7

Well, you’ve eliminated everything I could think to test, which is great, at least all of those variables are removed.

It will be curious to see what the incompatibility is once you get there.

8

Have you checked PostgreSQL to see if there is additional information in the logs?