Performance Testing

Questions
1

Your question

On our sales call, I think Mauro mentioned that there may be a repo that I could use to stress test our Stalwart instances. Is this true? If not, is there some way to put the pedal to the metal and give it a workout?

I understand that topics in this category are triaged by a bot first but a human reply will follow up. If I’d prefer a human-only reply, I’ll add the no-ai tag to my topic.

on

2

That is correct, the stress testing scripts are located under stalwart/tests/resources/scripts at main · stalwartlabs/stalwart · GitHub

3

I utilized those scripts, with a few modifications.

We loaded in 1000 users, and gave them each 50 emails in their inbox to start.
We ran the tests from our webmail VM.
Stalwart and Postgres VM’s were on a single Enterprise SSD. 8 Cores and 8GB Ram on each VM.

We ran 5 x 90 threads of the tests.

--------------------------------------------------------------------------------------------------------------------------------------
Persistent-connection stress test report (wall time: 120.77s, sessions: 90)
--------------------------------------------------------------------------------------------------------------------------------------
Action            OK   Err  Skip  Reconn    Min ms    Max ms    Avg ms    Med ms    P95 ms    P99 ms    Std ms    Ops/s        MB
--------------------------------------------------------------------------------------------------------------------------------------
SMTP SEND       3185     0     0       0     85.20   2000.64    251.41    193.64    636.78   1154.63    195.64     26.4     105.6
IMAP APPEND     3115     0     0       0    264.96   1885.50    746.63    699.59   1157.75   1399.51    197.83     25.8     102.5
IMAP FETCH      3113     0     0       0    141.99   2088.11    770.19    785.25    991.01   1112.80    166.70     25.8       0.0
IMAP DELETE     3205     0     0       0    298.08   2038.24   1208.01   1230.32   1504.80   1604.74    227.59     26.5       0.0
--------------------------------------------------------------------------------------------------------------------------------------
TOTAL          12618     0     0       0     85.20   2088.11    744.63    741.68   1413.78   1525.69    394.26    104.5     208.1
--------------------------------------------------------------------------------------------------------------------------------------

So the result was the above X 5.

With the 450 non-stop hammering sessions, we could do 129 incoming emails per second with a mix of SMTP and IMAP happening.

Each session is one simulated user holding one open IMAP connection and one open SMTP connection continuously for the full 2-minute test, and in a tight loop with zero pause between actions, doing:

  • 25% chance: send a new email
  • 25% chance: append a message directly into their inbox
  • 25% chance: search their mailbox and fetch a random message
  • 25% chance: search their mailbox, flag, and delete a random message

This seems way harsher than what the average user would put it through.

Could the Stalwart team give any feedback, does this seem in line with previous testing in any way? I’d like to make sure I’m not way off the mark with these results.

4

The production postgres cluster will run on nvme backed Ceph storage. The blob storage will be on SATA SSD’s in some form ZFS Zpool running on TrueNAS with Min.IO. The overhead with S3 storage is significant, but as I understand it, once written, the email is rarely written to again on S3 and is mostly just reads from there on out. The metadata gets updated in postgres, thereby reducing the storage workload vs legacy mail servers which are all file based. Is this about right?

5

Could the Stalwart team give any feedback, does this seem in line with previous testing in any way? I’d like to make sure I’m not way off the mark with these results.

Yes, we usually recommend tailoring the tests to match the protocols and usage patterns of your existing deployment, but for a general test, what you did is correct. A few notes though:

  • If you want to evaluate raw store performance, disable the spam filter, since it performs dozens of DNS queries per message against various DNS RBLs. Disabling DMARC, SPF and DKIM is also a good idea when evaluating raw performance, though it’s less critical than disabling the spam filter.
  • To reduce disk I/O and increase IMAP/JMAP performance, we recommend adjusting the caches (see Caching | Stalwart). As a rough guide, the cache uses about 80 bytes of memory per message and is evicted automatically based on usage patterns.

The overhead with S3 storage is significant, but as I understand it, once written, the email is rarely written to again on S3 and is mostly just reads from there on out.

That’s correct, blobs are immutable. They’re only accessed when the user retrieves the full email content, which should happen just once with well-behaved IMAP clients using client-side caching.

The metadata gets updated in postgres, thereby reducing the storage workload vs legacy mail servers which are all file based. Is this about right?

Legacy mail servers typically use Maildir, which is file based. The storage workload of Maildir is actually lower than Postgres+S3 for writes and similar for reads. The main advantage of Postgres+S3 is clustering and high availability. Maildir isn’t scalable or distributed, so you end up with silos and single points of failure.