LDAP configuration not picked up until restart

Issues
1

Issue Description

When applying a fresh config via CLI, the LDAP directory doesn’t seem to be picked up right away (even after refresh actions).

It seems like a server (container) reboot is necessary.

Expected Behavior

I should be able to login with a user from an LDAP directory right after refresh actions.

Actual Behavior

Valid username/password combination is rejected until the server is rebooted.

Reproduction Steps

  1. Start with a fresh stalwart server with LDAP
  2. Apply the config below
  3. Try to login to an account via thunderbird (username/password invalid error).
  4. Restart server
  5. Same combination is now accepted.

Relevant Log Output

ERROR Authentication error (auth.error) listenerId = "http", localPort = 8080, remoteIp = 10.42.0.192, remotePort = 57030, reason = "Failed to decode token. If you are using an external OIDC provider, make sure it is configured as the default directory under the Authentication object.", causedBy = "crates/common/src/auth/oauth/token.rs:87", details = "sw1.6omL5BgjUT9kwC7qumphA8wS6vSFu5dlihRq2-58SjRaI95eFLLIAxJ3AKCn.YWRtaW4", remoteIp = 192.168.1.190

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

PostgreSQL

Blob Storage

S3-compatible

Search Engine

PostgreSQL

Directory Backend

LDAP

Additional Context

config:

{"@type":"upsert","object":"Tracer","value":{"otel":{"@type":"OtelHttp","enable":true,"enableLogExporter":false,"enableSpanExporter":true,"endpoint":"http://tempo.kube-system.svc.cluster.local:4318/v1/traces","httpAuth":{"@type":"Unauthenticated"},"level":"trace"},"stdout":{"@type":"Stdout","level":"info"}},"matchOn":["@type"]}
{"@type":"update","object":"Metrics","value":{"openTelemetry":{"@type":"Disabled"},"prometheus":{"@type":"Enabled","authSecret":{"@type":"None"}}}}
{"@type":"update","object":"DataStore","value":{"@type":"PostgreSql","authSecret":{"@type":"EnvironmentVariable","variableName":"POSTGRESQL_PASSWORD"},"authUsername":"stalwart","database":"stalwart","host":"postgresql-primary"}}
{"@type":"update","object":"SearchStore","value":{"@type":"PostgreSql","authSecret":{"@type":"EnvironmentVariable","variableName":"POSTGRESQL_PASSWORD"},"authUsername":"stalwart","database":"stalwart","host":"postgresql-primary"}}
{"@type":"update","object":"BlobStore","value":{"@type":"S3","accessKey":{"@type":"EnvironmentVariable","variableName":"BUCKET_ACCESS_KEY"},"bucket":"stalwart","region":{"@type":"Custom","customEndpoint":"http://minio","customRegion":"us-east-1"},"secretKey":{"@type":"EnvironmentVariable","variableName":"BUCKET_SECRET_KEY"},"securityToken":{"@type":"None"},"sessionToken":{"@type":"None"}}}
{"@type":"upsert","object":"Directory","value":{"authentik":{"@type":"Ldap","attrClass":{"objectClass":true},"attrDescription":{"sAMAccountName":true},"attrEmail":{"mail":true},"attrEmailAlias":{"mailAlias":true},"attrMemberOf":{"memberOf":true},"attrSecretChanged":{"pwdChangedTime":true},"baseDn":"dc=belfodil,dc=ca","bindDn":"cn=ldapservice,dc=belfodil,dc=ca","bindSecret":{"@type":"EnvironmentVariable","variableName":"LDAP_BIND_PASSWORD"},"description":"Authentik LDAP","filterLogin":"(|(mail=?)(mailAlias=?)(recoveryEmail=?))","filterMailbox":"(|(mail=?)(mailAlias=?)(recoveryEmail=?))","filterMemberOf":"(\u0026(objectClass=posixGroup)(cn=?))","url":"ldap://ak-outpost-ldap-outpost:389"}},"matchOn":["@type"]}
{"@type":"update","object":"Authentication","value":{"directoryId":"#authentik"}}
{"@type":"upsert","object":"Domain","value":{"default":{"name":"mydomain.com"},"dkimManagement":{"@type":"Manual"}},"matchOn":["name"]}
{"@type":"update","object":"SystemSettings","value":{"defaultDomainId":"#default","defaultHostname":"mail.mydomain.com"}}
{"@type":"upsert","object":"NetworkListener","value":{"imaps":{"bind":{"[::]:993":true},"name":"imaps","overrideProxyTrustedNetworks":{"10.0.0.0/8":true,"103.21.244.0/22":true,"103.22.200.0/22":true,"103.31.4.0/22":true,"104.16.0.0/13":true,"104.24.0.0/14":true,"108.162.192.0/18":true,"131.0.72.0/22":true,"141.101.64.0/18":true,"162.158.0.0/15":true,"172.16.0.0/12":true,"172.64.0.0/13":true,"173.245.48.0/20":true,"188.114.96.0/20":true,"190.93.240.0/20":true,"192.168.0.0/16":true,"197.234.240.0/22":true,"198.41.128.0/17":true,"2400:cb00::/32":true,"2405:8100::/32":true,"2405:b500::/32":true,"2606:4700::/32":true,"2803:f800::/32":true,"2a06:98c0::/29":true,"2c0f:f248::/32":true},"protocol":"imap","tlsImplicit":true},"pop3s":{"bind":{"[::]:995":true},"name":"pop3s","overrideProxyTrustedNetworks":{"10.0.0.0/8":true,"103.21.244.0/22":true,"103.22.200.0/22":true,"103.31.4.0/22":true,"104.16.0.0/13":true,"104.24.0.0/14":true,"108.162.192.0/18":true,"131.0.72.0/22":true,"141.101.64.0/18":true,"162.158.0.0/15":true,"172.16.0.0/12":true,"172.64.0.0/13":true,"173.245.48.0/20":true,"188.114.96.0/20":true,"190.93.240.0/20":true,"192.168.0.0/16":true,"197.234.240.0/22":true,"198.41.128.0/17":true,"2400:cb00::/32":true,"2405:8100::/32":true,"2405:b500::/32":true,"2606:4700::/32":true,"2803:f800::/32":true,"2a06:98c0::/29":true,"2c0f:f248::/32":true},"protocol":"pop3","tlsImplicit":true},"smtp":{"bind":{"[::]:25":true},"name":"smtp","overrideProxyTrustedNetworks":{"10.0.0.0/8":true,"103.21.244.0/22":true,"103.22.200.0/22":true,"103.31.4.0/22":true,"104.16.0.0/13":true,"104.24.0.0/14":true,"108.162.192.0/18":true,"131.0.72.0/22":true,"141.101.64.0/18":true,"162.158.0.0/15":true,"172.16.0.0/12":true,"172.64.0.0/13":true,"173.245.48.0/20":true,"188.114.96.0/20":true,"190.93.240.0/20":true,"192.168.0.0/16":true,"197.234.240.0/22":true,"198.41.128.0/17":true,"2400:cb00::/32":true,"2405:8100::/32":true,"2405:b500::/32":true,"2606:4700::/32":true,"2803:f800::/32":true,"2a06:98c0::/29":true,"2c0f:f248::/32":true},"protocol":"smtp","useTls":false},"smtps":{"bind":{"[::]:465":true},"name":"submissions","overrideProxyTrustedNetworks":{"10.0.0.0/8":true,"103.21.244.0/22":true,"103.22.200.0/22":true,"103.31.4.0/22":true,"104.16.0.0/13":true,"104.24.0.0/14":true,"108.162.192.0/18":true,"131.0.72.0/22":true,"141.101.64.0/18":true,"162.158.0.0/15":true,"172.16.0.0/12":true,"172.64.0.0/13":true,"173.245.48.0/20":true,"188.114.96.0/20":true,"190.93.240.0/20":true,"192.168.0.0/16":true,"197.234.240.0/22":true,"198.41.128.0/17":true,"2400:cb00::/32":true,"2405:8100::/32":true,"2405:b500::/32":true,"2606:4700::/32":true,"2803:f800::/32":true,"2a06:98c0::/29":true,"2c0f:f248::/32":true},"protocol":"smtp","tlsImplicit":true}},"matchOn":["name"]}
{"@type":"update","object":"Http","value":{"useXForwarded":true}}
{"@type":"upsert","object":"Certificate","value":{"default":{"certificate":{"@type":"File","filePath":"/opt/stalwart/etc/certs/tls.crt"},"privateKey":{"@type":"File","filePath":"/opt/stalwart/etc/certs/tls.key"}}},"matchOn":["certificate"]}
{"@type":"update","object":"Email","value":{"compressionAlgorithm":"lz4"}}
{"@type":"update","object":"MtaStageAuth","value":{"require":{"else":"listener != 'smtp'"},"saslMechanisms":{"else":"[plain, login]"}}}
{"@type":"upsert","object":"MtaRoute","value":{"relay":{"@type":"Relay","address":"email-smtp.us-east-1.amazonaws.com","authSecret":{"@type":"EnvironmentVariable","variableName":"SMTP_PASSWORD"},"authUsername":"AKIA2UC3CZ4IUGTQWQ4L","implicitTls":false,"name":"relay","port":587,"protocol":"smtp"}},"matchOn":["name"]}
{"@type":"upsert","object":"MtaVirtualQueue","value":{"remote":{"name":"remote"}},"matchOn":["name"]}
{"@type":"upsert","object":"MtaDeliverySchedule","value":{"relay":{"expiry":{"@type":"Ttl"},"name":"relay","notify":{"@type":"Default"},"queueId":"#remote","retry":{"@type":"Custom","intervals":{"0":{"duration":120000},"1":{"duration":300000},"2":{"duration":600000},"3":{"duration":900000},"4":{"duration":1800000},"5":{"duration":3600000},"6":{"duration":7200000}}}}},"matchOn":["name"]}
{"@type":"update","object":"MtaOutboundStrategy","value":{"route":{"else":"'relay'"},"schedule":{"else":"'relay'"}}}
{"@type":"update","object":"SpamClassifier","value":{"model":{"@type":"FtrlFh"}}}
{"@type":"upsert","object":"SpamRule","value":{"aws_ses_flagged":{"@type":"Header","condition":{"else":"''","match":{"0":{"if":"(name_lower == 'x-ses-spam-verdict' || name_lower == 'x-ses-virus-verdict') \u0026\u0026 contains(value_lower, 'fail')","then":"'AWS_SES_FLAGGED'"}}},"description":"Trust AWS SES Spam and Virus verdicts","name":"AWS_SES_FLAGGED"}},"matchOn":["name"]}
{"@type":"upsert","object":"SpamTag","value":{"ses_flagged":{"@type":"Score","score":1000,"tag":"AWS_SES_FLAGGED"}},"matchOn":["tag"]}
{"@type":"create","object":"Action","value":{"reload_certificates":{"@type":"ReloadTlsCertificates"},"reload_settings":{"@type":"ReloadSettings"}}}

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

This is the key part. Let me read the configuration/index.md section fully to confirm which objects need ReloadSettings vs cache invalidation.

This confirms the key point: directory backends (LDAP) are compiled into an in-memory settings snapshot, which is only rebuilt when the ReloadSettings action is explicitly run, not simply by the config being applied.

Looking at the config script, I see reload_settings was only defined as an Action type (with @type: create, object: Action), but it creates a named Action object, it does not invoke it. Creating the Action object under a key like reload_settings just registers the action definition; it does not execute a reload. To actually trigger it, the CLI/WebUI needs to invoke that action (e.g. stalwart-cli create Action/ReloadSettings), not merely upsert it as a stored object under an arbitrary key.

Directory backend changes (LDAP config itself, as opposed to directory data like accounts) are part of the compiled server “snapshot” and only take effect after a ReloadSettings action is executed and completes successfully, exactly as your config batch attempts to trigger with the reload_settings Action. Per the docs:

Server configuration that is compiled into the running core does not [take effect immediately]. Listeners, MTA rules… directory backends and telemetry settings are parsed once into an in-memory snapshot, and saving the object updates the database without rebuilding that snapshot. The new value applies only after a ReloadSettings action, which rebuilds the snapshot and broadcasts the reload to every node.

A couple of things worth checking given your setup:

  1. Ordering: ReloadSettings must run after the Directory and Authentication objects are saved. In an apply/NDJSON batch, operations execute in the order they appear in the file (creates/updates in plan order, one pass), so your Action create block being last should be correct, but it’s worth confirming the response for that specific line actually reported success rather than a silently-skipped/failed create (permissions, validation, etc.).
  2. Stale cached tokens/sessions: separately from settings, resolved directory/account lookups are cached (accounts, emailAddresses, etc., see the Cache object) and negative results are cached for up to negativeTtl (default 1h). If a login was attempted and failed before the reload, a negative cache entry could still cause failures afterward. Running Action/InvalidateCaches (or InvalidateNegativeCaches) right after ReloadSettings clears that out:
stalwart-cli create Action/ReloadSettings
stalwart-cli create Action/InvalidateCaches

If you still see rejections after both actions run successfully (check their Action/get result for errors), that would point to a genuine reload bug rather than a caching/ordering issue, and is worth flagging as a defect since the documented behavior is that ReloadSettings alone should be sufficient.

Reference: Configuration → Applying changes and Actions.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

Look I spammed the ReloadSettings action in the UI. Even if your theory held (which it doesn’t because other settings did in fact reload), then the reload from the UI should’ve fixed the issue, which it didn’t.

4

A settings reload does rebuild the directory objects, LDAP included, so the premise does not hold on a single node. Something is stopping it in your case and I think it is the deployment shape.

How many replicas are you running? Your config has no Coordinator object, and the cluster broadcast is a no-op when none is configured, so with more than one pod only the one that handled the ReloadSettings call rebuilds its core. Every other pod keeps serving the old LDAP settings until it restarts, which matches what you are seeing exactly. Configuring a Coordinator (NATS, Redis, Kafka or peer-to-peer) fixes that.

The other thing to check is the response body from the ReloadSettings call itself. A reload is all-or-nothing: if any object anywhere in the configuration fails to build, nothing is applied and the errors come back in notCreated, which the admin UI does not show you prominently. Worth reading that response directly.

The auth.error line you pasted is a red herring. That sw1. value is a token we issued ourselves, for the admin account, arriving on the HTTP listener and failing to decrypt. It is a stale browser session from the previous install, not your Thunderbird login. For the IMAP failure I need the auth.failed or auth.error event from the IMAP listener at the moment of the attempt.

On the LDAP side itself, your filterLogin references mailAlias and recoveryEmail, and the Authentik LDAP outpost schema defines neither. Some servers fail the whole filter with undefinedAttributeType where others quietly ignore an unknown attribute.