System Sieve Script - Discard Duplicate Email

Issues
1

Issue Description

I have a script to discard emails if the sender, recipient, subject and body are the same. This works as user sieve script but not a system sieve script attached to the data stage.

Expected Behavior

System Sieve script should discard duplicate emails.

Actual Behavior

Warning in the log - Sieve action not supported reason = “Unsupported event”, details = “DuplicateId { id: “<unique id>”, expiry: 180, last: false }”

Duplicate email is delivered to recipient.

Reproduction Steps

require [
    "duplicate",
    "variables",
    "extracttext",
    "foreverypart",
    "mime"
];

set "s_from" "";
set "s_to" "";
set "s_subj" "";
set "s_body" "";


if header :matches "From" "*"    { set "s_from" "${1}"; }
if header :matches "To" "*"      { set "s_to" "${1}"; }
if header :matches "Subject" "*" { set "s_subj" "${1}"; }


foreverypart {
    if header :mime :type "Content-Type" "text" {
        extracttext :first 4096 "s_body";
    }
}

set "dedupe_key" "${s_from}|${s_to}|${s_subj}|${s_body}";

if duplicate :uniqueid "${dedupe_key}" {
    discard;
    stop;
}

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

PostgreSQL

Blob Storage

PostgreSQL

Search Engine

Internal

Directory Backend

Internal

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

2

This is a known limitation: the duplicate extension’s event (DuplicateId) is only handled by the personal/user Sieve script runtime, not by the trusted system-script event loop that runs at SMTP stages (EHLO/RCPT/DATA). Looking at the trusted script event loop in the SMTP scripts module, its match arm only handles IncludeScript, ListContains, Reject/queue actions, CreatedMessage, and SetEnvelope events; any other event (including the duplicate extension’s DuplicateId check) falls through to the “Unsupported event” branch, logs the warning you saw, and simply stops processing further script logic rather than discarding the message. That’s why duplicate works when the same script runs as a per-user Sieve script (handled by the full JMAP Sieve runtime) but not when attached as a system script on the MtaStageData stage.

There’s no supported workaround documented for using duplicate in a trusted/system script at the moment. If you need deduplication at the SMTP stage, you’d need to implement equivalent logic using a lookup store (e.g. string :list against a key/value store) that you update yourself, since the duplicate action itself isn’t wired into the trusted-script event loop.

Reference: Trusted Interpreter docs

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

3

Followup - There is a “Duplicate Expiry” configuration option in the system script interpreter that allows admins to set “Default expiration time for IDs stored by the duplicate extension from trusted scripts”.

This seems redundant if the DuplicateID is only handled by the user script run time.

4

duplicate only works in user scripts. It relies on per-mailbox tracking that doesn’t exist at the SMTP stage, so the system interpreter turns the event down, logs that warning and lets the message through. That leaves the duplicateExpiry setting on the system interpreter with nothing to do, and the docs should say so; I’ll fix them.

A system script can get the same result from the in-memory store. Keep your set, header and foreverypart lines as they are and swap the duplicate test for this:

require ["variables", "extracttext", "foreverypart", "mime", "vnd.stalwart.expressions"];

set "dedupe_key" "${s_from}|${s_to}|${s_subj}|${s_body}";
let "dedupe_id" "'dedupe-' + hash(dedupe_key, 'sha256')";

if eval "key_exists('', dedupe_id)" {
    discard;
    stop;
}
eval "key_set('', dedupe_id, '1', 604800)";

The last argument to key_set is the expiry in seconds, seven days here. The functions are described at Reference | Stalwart