Incremental spam training does not advance last_id, causing retained samples to be repeatedly processed

GoldenSW · September 21, 2026, 8:40am

Issue Description

In Stalwart 0.16.22, spam_train(false) loads the persisted trainer and selects samples starting at trainer.last_id + 1. However, last_id is updated only when it equals zero, so it stops advancing after the first successful training run.

Retained samples added afterward are consequently processed again in later runs. Deduplication is local to each run and does not prevent this.

This affects ordinary user feedback: add_account_spam_sample(), used by IMAP spam/ham corrections, passes hold_sample=true, producing delete_after_use=false.

Verified through source inspection and an isolated reproduction of the selection/counter logic. The same code is present on main.

Expected Behavior

After successful incremental training, the persisted cursor should advance past the newly processed samples.

A subsequent run with no new samples should perform no training and leave the ham/spam total_seen counters unchanged.

Actual Behavior

The cursor remains at the highest sample ID from the first successful training run. Later retained samples are repeatedly treated as new, incrementing counters and being submitted to training again.

An isolated control-flow reproduction produced:

Run 1: 100 ham + 100 spam added
Processed: 200; last_id: 200; ham: 100; spam: 100

Run 2: one new ham + one new spam added
Processed: 2; last_id: 200; ham: 101; spam: 101

Run 3: nothing added
Processed: 2; last_id: 200; ham: 102; spam: 102

IDs are simplified. The balanced classes exclude intentional reservoir replay for class balancing as the explanation.

At the default 12-hour interval, a retained sample could be processed approximately 360 times over 180 days if runs succeed and nothing removes the sample or resets/rebuilds the trainer. This is a scheduling-based estimate, not a production measurement. Classification-accuracy impact has not been measured.

Reproduction Steps

Suggested integration reproduction for an isolated instance; not executed here:

Start with a fresh classifier and 100 distinct valid ham samples plus 100 distinct valid spam samples. Set delete_after_use=false and keep their blobs available.
Run SpamFilterMaintenance with maintenanceType=“train” and wait for successful completion.
Add two new distinct retained samples: one ham and one spam.
Run maintenanceType=“train” again and wait for completion.
Without adding or changing samples, run maintenanceType=“train” a third time.
Inspect SpamTrainer.last_id, the reservoir ham/spam total_seen counters, and the training-completed event.

The code path predicts that the third run reprocesses the two samples from step 3 instead of reporting zero work. Use “train”, not “retrain”.

Relevant Log Output

No live-server training logs are attached. The results above come from an isolated reproduction of the source control flow, not from running the Stalwart binary.

Stalwart Version

v0.16.x

Installation Method

Binary (Linux)

Database Backend

RocksDB

Blob Storage

RocksDB

Search Engine

Internal

Directory Backend

Internal

Additional Context

Selection, deduplication and cursor assignment:

github.com/stalwartlabs/stalwart

crates/spam-filter/src/modules/classifier.rs

v0.16.22


      
          // Fetch blob hashes for samples
          let mut samples = Vec::new();
          let mut duplicate_samples = Vec::new();
          let mut remove_entries = false;
          let object_id = ObjectType::SpamTrainingSample.to_id();
          let from_key = ValueKey::from(ValueClass::Registry(RegistryClass::Item {
              object_id,
              item_id: trainer.last_id + 1,
          }));
          let to_key = ValueKey::from(ValueClass::Registry(RegistryClass::Item {
              object_id,
              item_id: u64::MAX,
          }));
          let mut seen_samples = AHashSet::new();
          let mut spam_count = 0;
          let mut ham_count = 0;
          self.store()
              .iterate(
                  IterateParams::new(from_key, to_key).descending(),
                  |key, value| {

This file has been truncated. show original

Trainer persistence:

github.com/stalwartlabs/stalwart

crates/spam-filter/src/modules/classifier.rs

v0.16.22


      
          // Store updated trainer and classifier
          let ham_count = trainer.reservoir.ham.total_seen;
          let spam_count = trainer.reservoir.spam.total_seen;
          let classifier = Archiver::new(match &trainer.trainer {
              SpamTrainerClass::FtrlFh(fh_trainer) => spamfilter::SpamClassifier::FhClassifier {
                  classifier: fh_trainer.build_classifier(),
                  last_trained_at: now(),
              },
              SpamTrainerClass::FtrlCfh(ccfh_trainer) => spamfilter::SpamClassifier::CcfhClassifier {
                  classifier: ccfh_trainer.build_classifier(),
                  last_trained_at: now(),
              },
          });
          self.blob_store()
              .put_blob(
                  SPAM_TRAINER_KEY,
                  &Archiver::new(trainer)
                      .serialize()
                      .caused_by(trc::location!())?,
                  self.core.email.compression,

This file has been truncated. show original

Counter increment:

github.com/stalwartlabs/stalwart

crates/nlp/src/classifier/reservoir.rs

v0.16.22


      
          impl<T: Clone + Eq> SampleReservoir<T> {
              pub fn update_reservoir(&mut self, item: &T, is_spam: bool, capacity: usize) {
                  let class = if is_spam {
                      &mut self.spam
                  } else {
                      &mut self.ham
                  };
          
                  class.total_seen += 1;
          
                  if class.buffer.len() < capacity {
                      class.buffer.push(item.clone());
                  } else if let Some(buf) = class
                      .buffer
                      .get_mut(rand::rng().random_range(0..class.total_seen as usize))
                  {
                      *buf = item.clone();
                  }

User-generated samples are retained:

github.com/stalwartlabs/stalwart

crates/email/src/message/ingest.rs

v0.16.22


      
                  self.add_spam_sample(
                      account_id,
                      batch,
                      (&metadata.inner.blob_hash).into(),
                      part.from().unwrap_or_default().to_string(),
                      thread_name(part.subject().unwrap_or_default()).to_string(),
                      is_spam,
                      true,
                      span_id,
                  );
              }
          
              Ok(())
          }
          
          fn add_spam_sample(
              &self,
              account_id: u32,
              batch: &mut BatchBuilder,
              hash: BlobHash,

This file has been truncated. show original

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

on

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

on

helpbot · September 21, 2026, 8:40am

Your source analysis matches what’s in crates/spam-filter/src/modules/classifier.rs on the referenced tag. The cursor update logic is:

if trainer.last_id == 0 {
    trainer.last_id = id;
}

This only sets last_id the very first time it’s zero (i.e., on the very first successful run), and is never updated again in subsequent runs, even though new samples are processed each time and merged into the reservoir/counters. So your reproduction’s behavior — last_id staying frozen at the value from run 1 while total_seen keeps incrementing on runs 2 and 3 — lines up with what the code does: since the range scan always starts at last_id + 1 and last_id never advances past the first run, previously-processed retained (non-delete_after_use) samples keep re-entering the selection window and get deduplicated only within that single run, not across runs.

This is a source-level defect report rather than a documentation question, so I can’t point you to docs that describe or contradict this behavior — there isn’t a doc page covering the training cursor internals. I don’t have a way to confirm here whether a fix is already staged or how it will be prioritized.

This is an automated reply from the Stalwart Help Bot. Other community members may follow up if this answer is incomplete or wrong.

stalwart · September 25, 2026, 7:37am

This has been fixed in v0.16.24 (yet to be released).

Topic		Replies	Views
Spam classifier: removing a same-day duplicate training sample clears the blob link of the sample that is kept Issues	2	28	October 4, 2026
WARN Training sample not found Questions	4	90	June 19, 2026
Make spam samples Editable Questions	3	59	September 30, 2026
Improving Stalwart's built-in AI spam classification: hosted API vs. local LLM and a small source experiment General	0	71	August 8, 2026
Spamtest sieve function only returns binary spam or ham, no intermediate values Issues	3	62	August 23, 2026

Incremental spam training does not advance last_id, causing retained samples to be repeatedly processed

Issue Description

Expected Behavior

Actual Behavior

Reproduction Steps

Relevant Log Output

Stalwart Version

Installation Method

Database Backend

Blob Storage

Search Engine

Directory Backend

Additional Context

I have reviewed the documentation and FAQ and confirm that my issue is NOT addressed there.

I have searched this support forum (open and closed topics) and confirm this is not a duplicate.

Related topics