Stalwart-migrate 1.0.1 segfault on aarch64

Issues
1

Issue Description

Running stalwart-migrate run on Ubuntu 24.04, arm64 results in segfault.

Expected Behavior

No segfault!

Actual Behavior

Segfault!

Reproduction Steps

Just follow the regular migration - prepare, then run.

Relevant Log Output

Offsets

Frame Address in core vaddr (for symbols) File offset (for bytes on disk)
#0 (PC) 0xc527b0b9b074 0x3b9b074 0x3b8b074
#1 0xc527b0a9f414 0x3a9f414 0x3a8f414
#2 0xc527b0a9e8a0 0x3a9e8a0 0x3a8e8a0
#3 0xc527b0a9cfd0 0x3a9cfd0 0x3a8cfd0
#4 0xc527b0a9cea4 0x3a9cea4 0x3a8cea4
#5 0xc527b09799d4 0x39799d4 0x39699d4

From coredump:

(gdb) bt
#0 0x0000c527b0b9b074 in ?? ()
#1 0x0000c527b0a9f414 in ?? ()
#2 0x0000c527b0a9e8a0 in ?? ()
#3 0x0000c527b0a9cfd0 in ?? ()
#4 0x0000c527b0a9cea4 in ?? ()
#5 0x0000c527b09799d4 in ?? ()
#6 0x0000c527aee30b38 in ?? ()
#7 0x0000c527af709180 in ?? ()
#8 0x0000c527af708524 in ?? ()
#9 0x0000c527af70c9a0 in ?? ()
#10 0x0000c527b02a9940 in ?? ()
#11 0x0000e87c5e1a582c in start_thread (arg=0xe87c5e3b6b80) at ./nptl/pthread_create.c:447
#12 0x0000e87c5e20bb4c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone3.S:76
(gdb) info threads
Id Target Id Frame

  • 1 Thread 0xe87c30f9e840 (LWP 112312) 0x0000c527b0b9b074 in ?? ()
    2 Thread 0xe87c3fdee840 (LWP 112417) 0x0000e87c5e1a1d6c in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
    futex_word=0xe87c3fde7658) at ./nptl/futex-internal.c:57
    3 Thread 0xe87c3d38e840 (LWP 112470) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    4 Thread 0xe87c4c3cd840 (LWP 112415) 0x0000e87c5e1a1d6c in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
    futex_word=0xe87c4c3c6658) at ./nptl/futex-internal.c:57
    5 Thread 0xe87c3cf9e840 (LWP 110677) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    6 Thread 0xe87c3f7ee840 (LWP 111905) 0x0000e87c5e20be74 in __GI_epoll_pwait (epfd=3, events=0xc527bd263db0, maxevents=1024, timeout=10624, set=0x0)
    at ../sysdeps/unix/sysv/linux/epoll_pwait.c:40
    7 Thread 0xe87c4fffe840 (LWP 110267) 0x0000e87c5e1a1d70 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
    futex_word=0xc527bd254668) at ./nptl/futex-internal.c:57
    8 Thread 0xe87c3d7ae840 (LWP 112468) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    9 Thread 0xe87c3d59e840 (LWP 112469) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    10 Thread 0xe87c5deae840 (LWP 110263) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    11 Thread 0xe87c31fbe840 (LWP 112314) 0x0000e87c5e1a1d6c in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
    futex_word=0xe87c31fbd6a8) at ./nptl/futex-internal.c:57
    12 Thread 0xe87c3fffe840 (LWP 112416) 0x0000e87c5e1a1d6c in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
    futex_word=0xe87c3fff7658) at ./nptl/futex-internal.c:57
    13 Thread 0xe87c317ae840 (LWP 112313) 0x0000e87c5e1a1d6c in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
    futex_word=0xe87c317ad6a8) at ./nptl/futex-internal.c:57
    14 Thread 0xe87c32fde840 (LWP 111083) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    15 Thread 0xe87c3fbde840 (LWP 112418) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    16 Thread 0xe87c5e0be840 (LWP 110262) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    17 Thread 0xe87c3dfbe840 (LWP 111084) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    18 Thread 0xe87c4c5dd840 (LWP 112324) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    19 Thread 0xe87c17ffe840 (LWP 112311) 0x0000e87c5e1a1d6c in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xe87c17ffddd8, op=393,
    expected=0, futex_word=0xc527bd24e408) at ./nptl/futex-internal.c:57
    20 Thread 0xe87c3e7ce840 (LWP 111085) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    21 Thread 0xe87c3efde840 (LWP 111086) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    22 Thread 0xe87c5d90d840 (LWP 110264) 0x0000e87c5e1a1d6c in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
    futex_word=0xc527bd263a68) at ./nptl/futex-internal.c:57
    23 Thread 0xe87c5d0fd840 (LWP 110265) 0x0000e87c5e1a1d6c in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
    futex_word=0xc527bd263a6c) at ./nptl/futex-internal.c:57
    24 Thread 0xe87c5c8ed840 (LWP 110266) 0x0000e87c5e1a1d6c in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
    futex_word=0xc527bd263a6c) at ./nptl/futex-internal.c:57
    25 Thread 0xe87c3c78e840 (LWP 112471) 0x0000e87c5e1a1d6c in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0,
    futex_word=0xe87c3c7812e8) at ./nptl/futex-internal.c:57
    26 Thread 0xe87c5e3b6440 (LWP 110261) syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
    27 Thread 0xe87c4c7ed840 (LWP 112315) 0x0000e87c5e207d4c in __GI_munmap () at ../sysdeps/unix/syscall-template.S:120
    (gdb) p $_siginfo
    $1 = {si_signo = 11, si_errno = 0, si_code = 1, _sifields = {_pad = {240, 0 <repeats 27 times>}, _kill = {si_pid = 240, si_uid = 0}, _timer = {si_tid = 240,
    si_overrun = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _rt = {si_pid = 240, si_uid = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}},
    _sigchld = {si_pid = 240, si_uid = 0, si_status = 0, si_utime = 0, si_stime = 0}, _sigfault = {si_addr = 0xf0}, _sigpoll = {si_band = 240, si_fd = 0},
    _sigsys = {_call_addr = 0xf0, _syscall = 0, _arch = 0}}}
    (gdb) p/x $_siginfo._sifields._sigfault.si_addr
    $2 = 0xf0
    (gdb) x/6i $pc-8
    0xc527b0b9b06c: clrex
    0xc527b0b9b070: mov x9, x8
    => 0xc527b0b9b074: ldr x8, [x8, #240]
    0xc527b0b9b078: cmp x8, x19
    0xc527b0b9b07c: b.ne 0xc527b0b9b070 // b.any
    0xc527b0b9b080: str x22, [x9, #240]
    (gdb) info registers
    x0 0x85e4 34276
    x1 0xe87c30f9d5c0 255620095268288
    x2 0xe87c30f9d2e8 255620095267560
    x3 0x1 1
    x4 0x0 0
    x5 0xc527bdb9ad90 216774477458832
    x6 0xfffffffffffffff0 -16
    x7 0x3 3
    x8 0x0 0
    x9 0x0 0
    x10 0x30 48
    x11 0x5 5
    x12 0x0 0
    x13 0x0 0
    x14 0xfffffffffffff000 -4096
    x15 0x0 0
    x16 0x1 1
    x17 0xe87c5e1aa5c4 255620852393412
    x18 0x5 5
    x19 0xe87c4c3c6570 255620552615280
    x20 0xc527bdb9b0c0 216774477459648
    x21 0x21080 135296
    x22 0xe87c3c781200 255620288090624
    x23 0xe87c3c781200 255620288090624
    x24 0xe87c4c3c6570 255620552615280
    x25 0xe87c4c3c6570 255620552615280
    x26 0xe87c30f9d619 255620095268377
    x27 0x0 0
    x28 0xc527bdb9b280 216774477460096
    x29 0xe87c30f9cf30 255620095266608
    x30 0xc527b0a9f414 216774258324500
    sp 0xe87c30f9cf30 0xe87c30f9cf30
    pc 0xc527b0b9b074 0xc527b0b9b074
    cpsr 0x80001000 [ EL=0 BTYPE=0 SSBS N ]
    fpsr 0x10 [ IXC ]
    fpcr 0x0 [ Len=0 Stride=0 RMode=0 ]
    tpidr 0xe87c30f9ef80 0xe87c30f9ef80
    tpidr2 0x0 0x0
    (gdb) info proc mappings
    Mapped address spaces:

        Start Addr           End Addr       Size     Offset objfile
0xc527ad000000     0xc527addc3000   0xdc3000        0x0 /usr/local/bin/stalwart-migrate
0xc527addd2000     0xc527b0f87000  0x31b5000   0xdc2000 /usr/local/bin/stalwart-migrate
0xc527b0f96000     0xc527b11e7000   0x251000  0x3f76000 /usr/local/bin/stalwart-migrate
0xc527b11f6000     0xc527b1219000    0x23000  0x41c6000 /usr/local/bin/stalwart-migrate
0xe87c5e0c0000     0xe87c5e0c1000     0x1000        0x0 /usr/lib/aarch64-linux-gnu/libdl.so.2
0xe87c5e0c1000     0xe87c5e0df000    0x1e000     0x1000 /usr/lib/aarch64-linux-gnu/libdl.so.2
0xe87c5e0df000     0xe87c5e0e0000     0x1000     0xf000 /usr/lib/aarch64-linux-gnu/libdl.so.2
0xe87c5e0e0000     0xe87c5e0e1000     0x1000    0x10000 /usr/lib/aarch64-linux-gnu/libdl.so.2
0xe87c5e0f0000     0xe87c5e0f1000     0x1000        0x0 /usr/lib/aarch64-linux-gnu/libpthread.so.0
0xe87c5e0f1000     0xe87c5e10f000    0x1e000     0x1000 /usr/lib/aarch64-linux-gnu/libpthread.so.0
0xe87c5e10f000     0xe87c5e110000     0x1000     0xf000 /usr/lib/aarch64-linux-gnu/libpthread.so.0
0xe87c5e110000     0xe87c5e111000     0x1000    0x10000 /usr/lib/aarch64-linux-gnu/libpthread.so.0
0xe87c5e118000     0xe87c5e120000     0x8000        0x0 /opt/container-data/stalwart/migrate-state/state.sqlite-shm
0xe87c5e120000     0xe87c5e2ba000   0x19a000        0x0 /usr/lib/aarch64-linux-gnu/libc.so.6
0xe87c5e2ba000     0xe87c5e2cd000    0x13000   0x19a000 /usr/lib/aarch64-linux-gnu/libc.so.6
0xe87c5e2cd000     0xe87c5e2d0000     0x3000   0x19d000 /usr/lib/aarch64-linux-gnu/libc.so.6
0xe87c5e2d0000     0xe87c5e2d2000     0x2000   0x1a0000 /usr/lib/aarch64-linux-gnu/libc.so.6
0xe87c5e2e0000     0xe87c5e365000    0x85000        0x0 /usr/lib/aarch64-linux-gnu/libm.so.6
0xe87c5e365000     0xe87c5e37f000    0x1a000    0x85000 /usr/lib/aarch64-linux-gnu/libm.so.6
0xe87c5e37f000     0xe87c5e380000     0x1000    0x8f000 /usr/lib/aarch64-linux-gnu/libm.so.6
0xe87c5e380000     0xe87c5e381000     0x1000    0x90000 /usr/lib/aarch64-linux-gnu/libm.so.6
0xe87c5e384000     0xe87c5e3ab000    0x27000        0x0 /usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
0xe87c5e3c2000     0xe87c5e3c4000     0x2000    0x2e000 /usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
0xe87c5e3c4000     0xe87c5e3c6000     0x2000    0x30000 /usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1

Stalwart Version

v0.16.x

Installation Method

Docker

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

Additional Context

stalwart-migrate status --state ./migrate-state
State database: ./migrate-state/state.sqlite
tool_version 1.0.1
edition enterprise
source_def_hash eac838c9ccd547f73f088f438c223eb6
target_def_hash 7fbcc15151bc4da024cd6b880c74bbd0
target_schema_version 7
categories settings,principals,mail,calendars,contacts,files,sieve,identities,push_subscriptions,spam_samples,reports,archive,spam_model,queue,tasks,telemetry
created_at 2026-10-27T10:19:24Z
source_store directory /opt/container-data/stalwart/data
target_claim 693c6a5a52f1b3766bbfdca4b163a2da
claim_generation ffe6637f34a8633499680d1bee91a23a
claim_generation_pending ffe6637f34a8633499680d1bee91a23a
final_done not done

Accounts: 8
done 4
full 4

Recent runs:
#12 incremental running started 2026-10-27T11:02:19Z
#11 incremental interrupted started 2026-10-27T10:59:05Z
#10 incremental interrupted started 2026-10-27T10:58:42Z
#9 incremental interrupted started 2026-10-27T10:40:31Z
#8 incremental interrupted started 2026-10-27T10:38:27Z

Reports in the latest run:
none

Reports in all runs:
info changed 23
warning dropped 61

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

This isn’t covered in the current docs. A community member or staff member will follow up.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

Oh so we created the same issue at the same time…

4

Same crash here: stalwart-migrate 1.0.1 (standard build), aarch64, Debian-based (glibc 2.43), Oracle Cloud Ampere A1 with 4 vCPUs and 23 GiB RAM. The source was Stalwart 0.16.27 on RocksDB, running during the run, with a file system blob store (blob_mode=Shared). The target was a new, empty RocksDB directory.

The crashing PC and the next two frames match yours exactly, reached through a different call path:

Signal: 11 (SEGV) si_code: SEGV_MAPERR   (binary sha256 c1a2d6fa52dfbfbb965f0aa49bff8d2b0896043c1b7e518290ef65d17e1bf6c2)
#0  +0x3b9b074    #7  +0x198d664    #14 +0x360dfd8
#1  +0x3a9f414    #8  +0x19193cc    #15 +0x3616b54
#2  +0x3a9e8a0    #9  +0x1c9d308    #16 +0x35fdd1c
#3  +0x3d5f834    #10 +0x1bfb0a4    #17 +0x3606f9c
#4  +0x397be8c    #11 +0x1886268    #18 +0x32a9940
#5  +0x3319e64    #12 +0xfcebd8     #19 start_thread (libc)
#6  +0x18bac50    #13 +0x360f7a0

It happened on the first run after prepare, with default concurrency (concurrency=16, --account-split default). That was about 75 s in, with roughly 76,000 emails written while two large accounts copied in parallel at about 2,000 emails/s. Nothing was printed before the crash, and there was no memory pressure (about 3 GiB in use).

Workaround: re-running the same command with --concurrency 4 --account-split 2 resumed and finished: 887,590 emails in 18.7 min, 0 errors. Two more incremental runs, run --final and its verification also went through at that concurrency, and we’re now on 1.0.0. So it looks concurrency-dependent, which may help anyone else who hits it.

I can’t share the core dump (it holds mailbox contents), but I’m happy to run gdb commands against it while it’s still around.

5

Tried with that workaround but still crashes for me.

6

for me changing those parameters does not help either.

7

Thanks for the core dumps, @rcfa and @mzhaase. With the crash addresses we could match the faulting instruction in the 1.0.1 binary, and from there it wasn’t hard to find.

It’s a bug in RocksDB’s write queue that only our aarch64 builds expose. In early October we upgraded the toolchain the arm64 binaries are built with, and the new one compiles RocksDB’s atomic compare-and-swap differently. Under concurrent writes that operation can fail spuriously on arm64, and RocksDB then follows a NULL pointer. That’s the si_addr = 0xf0 in your dumps. x86_64 builds can’t hit it, and it also explains why lowering --concurrency only helps sometimes: it makes the crash less likely without ruling it out.

The fix goes out in the next weekly Stalwart release and in the next stalwart-migrate release, for both the gnu and musl aarch64 builds. Your existing state database is fine; once you have the new binary, just run the same stalwart-migrate run again and it picks up where it stopped.

If you’d like to finish before then, building stalwart-migrate from source on the arm64 machine with the system compiler avoids the problem. The steps are in the README.

8

Ok, finished it with migrate 1.0.3 and that worked fine. Thanks! I especially liked that although it crashed a couple times, the migrate tool was very robust in just picking it back up.