Idempotent apply of non-singleton objects

Questions
1

Your question

We’re running the official Docker image (v0.16.9) on NixOS and manage our config declaratively now - stalwart-cli (1.0.8, installed separately since it’s not in the image) runs apply on every deploy from plan files in git.

What I want is that the same plan converges no matter the current state, empty or already populated. For singletons that’s straightforward, update always works. For non-singletons I’m not sure what the intended pattern is.

destroy {filter} + create is state-independent on its own, but as soon as another object references the one I’m replacing, the destroy gets rejected, e.g.:

destroy failed for id <id>: error: objectIsLinked
  Object id:  Directory#<id>
  Linked by:  Authentication#singleton

(That’s just one example - a Directory referenced by Authentication. The same applies to anything referenced by another object.)

Putting the referencing object’s update in the same plan doesn’t help, since apply runs all destroys before the creates/updates, so the link is still there when the destroy fires. And a plain update by id only works if the object already exists, so it isn’t state-independent - it breaks on a fresh server.

I’ve read through these but couldn’t quite work out how this case is meant to be handled:

So for non-singletons that are referenced anywhere, I can’t find a single plan that converges from both states. What’s the recommended way to handle this? Is there ordering control within a plan, a create-or-update behaviour, or is the intended approach to split it into multiple applies (clear the references first, then replace and re-link)?

Thanks.

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

Let me research the declarative deployment documentation and CLI apply behavior thoroughly.

The docs cover the exact problem you’re hitting. The key insight is in the destroy ordering section. Here’s what’s happening and the fix:

The objectIsLinked error is caused by a destroy ordering mistake. The two-pass model reverses the destroy list at runtime (to handle dependency teardowns). This means if you list your destroy operations in the “logical” order (e.g., Directory before Authentication), the CLI re-reverses them to execute Authentication first - which is what you want - but only if you write the destroys parents-first (same order as your creates).

From the docs on the “Dependency ordering” section:

Destroys must be listed in the same order as the creates. The reverse pass will then take them down children-first, matching foreign-key constraints.

A common pitfall: writing destroys in reverse-of-creates order (children-first) makes the apply re-reverse them at runtime to parents-first, which fails on objectIsLinked. The fix is to write destroys forwards (parents-first); the apply does the reversal.

So for your Directory → Authentication case, your plan should look like this:

{"@type":"destroy","object":"Directory","value":{"name":"my-dir"}}
{"@type":"destroy","object":"Authentication"}
{"@type":"create","object":"Directory","value":{"dir-1":{"name":"my-dir",...}}}
{"@type":"update","object":"Authentication","value":{"directoryId":"#dir-1"}}

The destroy pass executes them in reverse - so Authentication is cleared first, then Directory is destroyed without the objectIsLinked error. The create pass then rebuilds Directory and re-links Authentication to it.

This is a single idempotent plan: on a fresh server, the destroys find nothing to delete (no-op), and the create/update proceed normally. On a populated server, the destroys cleanly unlink and remove before rebuilding.

Reference: Declarative bulk operations - Dependency ordering

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

Thanks for the suggestion, but that plan doesn’t actually apply. The destroy Authentication line gets rejected right away at dry-run:

error: cannot destroy singleton `Authentication`

So the parents-first destroy trick doesn’t help here: the object holding the reference is a singleton, and singletons can’t be destroyed. I also can’t clear the link with an update instead, because in a single plan all destroys run before the creates/updates, so Authentication.directoryId is still pointing at the Directory when the destroy fires, and I get objectIsLinked.

So as far as I can tell, for a non-singleton that’s referenced by a singleton there’s no single plan that’s state-independent: destroy+create is blocked by the link, and a plain update by id only works once the object already exists (breaks on a fresh server).

What’s the recommended way here? Is the intended approach two separate applies — first an update setting directoryId to null, then destroy+create+re-link in a second apply - accepting the short window where the reference is null? Or is there a way to make an update run before the destroy pass within one plan?

And the same applies to your own snapshot output: a full export emits destroy Directory but only an update for the Authentication singleton — so re-applying that export to an already-populated server hits the exact same objectIsLinked. It only round-trips against a fresh server. Is reconcile-on-deploy against a live server simply not supported for objects a singleton references, or is there a pattern I’m missing?

4

I don’t think there is a solution.

I raised a similar post here: Recommended idempotent pattern for apply plans given non-nullable singleton references?

5

You can somewhat work around this using multiple plans, and just ignoring errors for some of the plans, but its very hacky and isn’t fully idempotent.

I am not personally using docker, but here is the shape of what I’m doing

  • some plans pass --continue-on-error to stalwart-cli apply (see my abstraction if you’re interested)
  • I use multiple plans to control ordering (see my config if interested)
6

I don’t think there is a solution.

To clarify there absolutely could be a solution, the current config design should effectively disallow non-nullable references. In fact the current implementation just puts an invalid reference inside of SystemSettings.defaultDomainId which actually makes it impossible to restore the config to the default state once it’s set.

Additionally, the lack of an upsert (or more ideally a create-or-update-to-match) operator is what makes having multiple plans required.

Idempotent configs are a solved problem—the docs sound too fluent about idempotency for the design to be this broken. I just don’t think enough (human?) thought was put into the current design; the project’s stance against taking contributions, leads me to be unable to help.

I was initially really excited about this project, but this is making me have second thoughts.

7

The Authentication object is a singleton and cannot be destroyed. You should first create the default domain and then update the Authentication object with its id in the defaultDomainId property.
Also, the Authentication object is not fully idempotent as the defaultDomainId property is not nullable. However, it is not advised to create a plan that deletes domains and then creates them again each time it is executed. Accounts, aliases and DKIM keys are all linked to domains. Domains are management objects, not configuration, and for that reason it is not a good idea to treat them as idempotent settings.

We had to disable open PRs due to the volume of low-quality, AI-generated submissions flooding the Stalwart codebase. Without a dedicated team to review and triage that kind of contribution, keeping the queue open simply isn’t sustainable. Stalwart is far from the only open source project to restrict contributions this way, and the decision reflects resource constraints, not a lack of care for open source contributions.
We’d also ask that feedback stay constructive. Questioning whether sufficient thought went into the design isn’t a fair characterization, and it’s not the kind of tone that moves a conversation forward productively.

8

I want to start by saying how much I respect the work you’ve put into Stalwart — it’s genuinely impressive, and I’ve been following the project with a lot of enthusiasm.

But I have to be honest: I’ve been trying to migrate from v0.15.5 to v0.16.10 for a week now, and the configuration experience has become so painful that I’m seriously starting to consider moving to a different mail server — not because of the migration itself, but purely because of the new configuration system.

Here’s what I’m running into:

The declarative bulk operations approach makes it nearly impossible to achieve a clean, idempotent configuration — even for basic things. For example, to set a basic SystemSettings object, you must provide a defaultDomainId. But to have a defaultDomainId, you first need to create a Domain object. This is a classic chicken-and-egg problem: the Domain object makes the configuration non-idempotent (subsequent applies fail with “object already exists” errors), and at the same time — as you yourself noted — it is not advised to treat domains, accounts, aliases, and DKIM keys as idempotent settings, since they are management objects rather than configuration

On top of that, the strict operation ordering — destroy first, then update, then create — makes it extremely difficult to write a configuration that applies cleanly. You constantly run into errors like “cannot update this because the object doesn’t exist yet”, and in most cases that object is a reflink (#) that fundamentally cannot be created first, since destroy runs before create.

Snapshots, in theory, could help — but recreating all objects on every apply doesn’t feel safe to me in a production environment.

I genuinely feel stuck. A week in, and I still can’t get the server into a clean, reproducible state. It’s exhausting.

I think having even a minimal static config file for foundational objects — network listeners, basic system settings, and similar primitives — would make a world of difference. It doesn’t have to cover everything, just the pieces that need to exist before anything else can reference them.

I really hope this feedback is useful. I want to keep using Stalwart — I just need the configuration story to be workable.

9

I apologize that was uncalled for and wasn’t constructive.

10

Please try using CLI v1.0.9 which now supports an upsert operation. Before trying it, take quick look at the updated apply and snapshot documentation. Using the new upsert command should be straightforward as it tries to match regular objects (not singletons) by the display name defined in the schema (the “label” property). If the object does not define a label (or if you want to define a custom field for matching), then take a look at the matchOn property in the upsert command.