[How-To] Fixing “An item with the same key already exists” after version mismatch in Stalwart v0.16.x (PostgreSQL backend)
Background: After deploying Stalwart with an older Docker image (while the version string in the configuration already referenced a newer version), several accounts and domains became broken or uncreatable. This post documents the full repair procedure.
Symptoms
-
Trying to create a user account gives: “An item with the same key already exists”
-
Some accounts appear broken or missing in the admin UI
-
Some domains show errors despite existing in the database
Root Cause
When a mismatched Stalwart Docker image version is used together with a newer version string in the configuration, data gets written to PostgreSQL in an older binary format. Specifically:
-
The VERSION byte in the d-table (main object storage) is written as 0x01 instead of the expected 0x00 for Stalwart v0.16.2
-
During migration or re-deployment with the correct image, accounts/domains written in the old format become invisible or broken to Stalwart
-
When you then try to delete and re-create those accounts via the admin UI, Stalwart’s internal cleanup only partially succeeds — it removes the main d-table entry but leaves ghost index entries in the b-table (SUBSPACE_REGISTRY_IDX) and g-table (SUBSPACE_REGISTRY_PK) behind
-
These orphan entries then permanently block re-creation of the same usernames
The affected tables are:
| Table |
Subspace |
Purpose |
d |
SUBSPACE_STORE |
Main object storage (accounts, domains) |
b |
SUBSPACE_REGISTRY_IDX |
Index entries (key only, no value) |
g |
SUBSPACE_REGISTRY_PK |
Global PrimaryKey unique locks (key + value) |
The Fix (step by step)
Step 1 — Fix VERSION byte corruption in d-table
Stalwart v0.16.x expects VERSION=0x00 as the first byte of every value in the d-table. An older image writes VERSION=0x01. Fix all corrupted domain and account entries:
sql
-- Find corrupted domain entries (prefix 0x0025)
SELECT encode(k,'hex'), get_byte(v,0) as version_byte
FROM d
WHERE substr(k,1,2) = '\x0025'::bytea
AND get_byte(v,0) != 0;
-- Fix domain entries
UPDATE d SET v = set_byte(v, 0, 0)
WHERE substr(k,1,2) = '\x0025'::bytea
AND get_byte(v,0) != 0;
-- Fix account entries (prefix 0x0000)
UPDATE d SET v = set_byte(v, 0, 0)
WHERE substr(k,1,2) = '\x0000'::bytea
AND get_byte(v,0) != 0;
Step 2 — Clean ghost entries from b-table (SUBSPACE_REGISTRY_IDX)
After attempting to delete broken accounts through the admin UI, ghost entries remain in the b-table. Find and remove entries referencing non-existent account IDs:
sql
-- Find ghost entries (replace 11,12,13 with your ghost account IDs)
SELECT encode(k,'hex') FROM b
WHERE (
substr(k,1,2) = '\x00dd'::bytea OR
substr(k,1,2) = '\x0000'::bytea
)
AND (
get_byte(k, octet_length(k)-4)*16777216 +
get_byte(k, octet_length(k)-3)*65536 +
get_byte(k, octet_length(k)-2)*256 +
get_byte(k, octet_length(k)-1)
) IN (11, 12, 13);
-- Backup first, then delete
CREATE TABLE b_backup_ghost AS SELECT k FROM b WHERE ...;
DELETE FROM b WHERE ...;
Step 3 — The critical fix: clean ghost entries from g-table (SUBSPACE_REGISTRY_PK)
This is the most important step and the direct cause of the “item with same key already exists” error. The g-table stores global unique PrimaryKey locks for email addresses. Format:
Key: \xffff 00f2 [username_bytes] [domain_id as 8-byte BE uint64]
Value: [object_type 2 bytes] [account_id as 8-byte BE uint64]
0xffff00f2 = global unique lock for Property::Email (242 = 0xF2).
List all email unique locks and check if the referenced account ID still exists:
sql
SELECT encode(k,'hex'), encode(v,'hex')
FROM g
WHERE substr(k,1,4) = '\xffff00f2'::bytea
ORDER BY k;
The last 8 bytes of the value column are the account ID (big-endian). If that account no longer exists in the d-table, it is a ghost entry blocking account creation.
sql
-- Backup
CREATE TABLE g_backup_ghost AS
SELECT k, v FROM g WHERE
k = '\xffff00f2<username_hex><domain_id_8byte_BE>'::bytea;
-- Delete ghost locks
DELETE FROM g WHERE
k = '\xffff00f2<username_hex><domain_id_8byte_BE>'::bytea;
Step 4 — Restart Stalwart
bash
kubectl rollout restart deployment/mail- -n
Account creation should now work normally.
Key Lesson
Always make sure the Docker image version matches the version string in your configuration. Using an older image with a newer version string causes data to be written in an incompatible binary format. When those mismatched entries are subsequently deleted through the admin UI, Stalwart’s cleanup leaves ghost entries in the b and g index tables that permanently block re-creation of the same usernames/domains.
Environment
Hope this helps someone else who ran into the same issue!