Start the node with STALWART_RECOVERY_MODE=1. That skips every background service and brings the management API up on port 8080. If it starts clean, the fault is in one of those services, and that alone cuts the search space a long way. To run normally minus the suspect path, create a ClusterRole with the pushNotifications task disabled and point STALWART_ROLE at it. If that starts too, you have mail flowing while we work out the rest.
The ordering in your log (accessToken miss, then role hit, then store.data-iterate) is an access token being built: permissions get resolved, then an ACL query iterates. That runs on every authentication, not only in the push notification sweep, so it pins the moment but not the caller. What the offset tells us is solid though. We build with lto = true and codegen-units = 1, so a stable offset from the load base on both Docker and bare metal means the same monomorphised function every crash, which in turn means one specific stored record type is being read back. I cannot tell you which one from here, and that is something we need your help with.
addr2line will never give you anything against the shipped binary. Our release profile sets debug = false and strip = true, so the symbols are gone before the binary ever ships. Quickest way through is to reproduce the crash under a build that still has them:
git clone https://github.com/stalwartlabs/stalwart
cd stalwart
git checkout v0.16.22
In the top-level Cargo.toml, under [profile.release], change debug = false to debug = 2 and strip = true to strip = false. Leave opt-level, lto and codegen-units exactly as they are; changing those changes the code path and you may lose the bug. Then build with the same features as the Docker image:
cargo build --release -p stalwart --no-default-features \
--features "sqlite postgres mysql rocks s3 redis azure nats enterprise"
Point that binary at a copy of the database, not the live one. Two nodes against the same PostgreSQL will fight over the same keys and you will get failures that have nothing to do with this bug.
For the core dump, on the bare metal VM the systemd route is easiest. Add LimitCORE=infinity to the unit, systemctl daemon-reload, restart, and let it crash. Then:
coredumpctl list stalwart
coredumpctl gdb stalwart
If systemd-coredump is not installed, ulimit -c unlimited before launching plus a writable /proc/sys/kernel/core_pattern gets you the same file. Inside gdb the four things I want are:
thread apply all bt full
frame 1
info registers rdi rsi rdx
x/64xb $rdi
The register dump matters as much as the backtrace here. The faulting instruction is a load from the first argument, so whatever rdi holds is the bad pointer, and the bytes around it tell us whether we are walking a relative pointer out of a stored record or chasing freed memory. Those two have very different fixes.
RUST_BACKTRACE=1 will not help, so do not bother setting it. A SIGSEGV does not unwind, so no Rust backtrace is ever printed.
On traces, you already have trace level running, which is what I need. Send the whole window from process start through to the fault, not just the last few lines. The account id is the useful part: your paste shows key = 7, collection = "accessToken" immediately before the crash, so account 7 is the first thing I would look at.
You can pull the suspect record straight out of PostgreSQL. Push subscriptions live in table p under a ten byte key: account id as a big-endian u32, then collection 7, then field 44, then a zero document id.
SELECT (get_byte(k,0)<<24)|(get_byte(k,1)<<16)|(get_byte(k,2)<<8)|get_byte(k,3)
AS account_id,
length(v) AS len,
encode(v, 'hex') AS value
FROM p
WHERE length(k) = 10 AND get_byte(k,4) = 7 AND get_byte(k,5) = 44;
That should be a short list. If you would rather not paste record contents into a public thread, send them privately to [email protected].
One more thing, and it may turn out to be the whole story. Did this deployment come up from v0.15.x or older, and if so, how was that upgrade done? I am asking specifically whether you went through the migration proxy or did the offline steps from UPGRADING/v0_16.md by hand. The offline path requires every node to be stopped for the duration, and on Kubernetes a rolling update does the opposite by default: it runs old and new pods against the same database at the same time. A v0.15.x pod writing records while a v0.16.x pod is migrating produces exactly the kind of record that would read back as garbage, and the symptom would be permanent and survive an export and import, which matches what you are seeing. If that is what happened, this is a migration bug on our side and I want to fix the migration, not just this one record.
Also useful to know: how many replicas were running during the upgrade, and whether the crash predates the export and import or first appeared after it.