This article applies to:

• Trustwave MailMarshal (SEG)

Question:

Procedure:

So you think you need to get MailMarshal really flying? This guide should give you a range of things to consider.

MailMarshal performs exceptionally well straight out of the box without doing anything special. So, unless you are experiencing problems, or have a lot of mail to deal with, often it is best to simply leave MailMarshal to get on with it.

If you are concerned about performance, some basic items to consider are:

• Keep MailMarshal up to date
• Use current hardware
• Use a reliable local DNS server
• Configure disk locations appropriately
• Use a fast virus scanner
• Optimize user matching
• Optimize other rules
• Use a separate computer for Array Manager/SQL Server

Note: Much of the material in this article is condensed from the following documents available on the Trustwave MailMarshal documentation page (requires login):

• Trustwave MailMarshal Sizing Guide
• Trustwave MailMarshal Policy Implementation Sizing Guide

For more details, see these documents.

Keep MailMarshal up to date

As with any software, it is important to keep SEG up to date. Acceptance testing showed that MailMarshal SMTP 6.0 was about 40% faster than MailMarshal 5.5. SEG 7.X provided additional speed-enhancing features. SEG 8.X with 64-bit architecture throughout is even faster but does require additional resource for 64-bit structures. Even with additional scanning layers, MailMarshal 10.X on modern systems has greater throughput than ever.

Notably, you can split MailMarshal components onto multiple servers. This allows you to enhance performance and scale up MailMarshal for very large sites (100,000 email users or more).

Use current hardware

Suggested minimum hardware for single-server installations is as follows:

• 1,000 users: 4 cores 2.6GHz (Core i5 or similar), 20GB HD free, 2GB RAM
• 10,000 users: 4 cores 3GHz, (better Core i7 or similar), 100GB HD free, 4GB RAM

There is some discussion of hardware scaling below. For additional hardware configuration recommendations, see Q10829.

Let's start with the basics

The following numbers are based on traffic through Marshal's former gateway (which serviced about 750 users for three companies). Experience has shown these numbers to be fairly representative of most organizations.

The average message size is 45K with a breakdown as follows:

By 8:00am only 2% of the business related mail volume has arrived. However by 6:00pm 95% of the business related volume has arrived. The flow of data between these times is relatively linear. (Link bandwidth is not an issue). However, spam generally arrives around the clock.

The "average" corporation sends and receives around 20 messages per person per day to the Internet (most of which is spam). So, a 10,000 user site might ship 200,000 mail messages a day with the majority in a ten hour period.

You should review the number of messages and total volume for your organization.

How does mail flow through MailMarshal?

MailMarshal has three core services: a Receiver, an Engine, and a Sender.

• The Receiver accepts incoming mail traffic and writes it to the Incoming directory.
• The Engine unpacks the mail into the Unpacking directory, and if the message passes, moves it to the ProcessedOK directory.
• The Sender then accepts the mail for delivery, moves it into the Sending directory, and proceeds to deliver it to the next destination.

Discussion

From this simple overview there are a number of things we can learn:

1. The email message files are typically moved unchanged from directory to directory. To make this file transfer efficient, the Incoming, ProcessedOK, and Sending directories should all be on the same logical disk. This avoids the operating system having to copy the file.
2. The MailMarshal Engine unpacks mail into the Unpacking directory. This directory does not have to be on the same partition as the other SEG directories. The unpacking process creates a lot of temporary files and directories and although Windows Server caches file reads and writes very well, it does not like the file creations and deletes very much. The best solution is to run the Unpacking directory on its own partition on a separate physical disk. A write caching disk controller can also help here, but introduces some other potential problems.  
3. You have probably also noticed that MailMarshal logs a lot of detail about what is going on. This is all written to text files created in the Logging directory and kept by default for 5 days. NTFS, like virtually every other file system, does not like being run out of space. Keep your disk free space high, certainly no more than 60% full.

The MailMarshal Engine

The Engine does the hard work of implementing the majority of the rules you configure in MailMarshal. Every message is unpacked into its components, and then a user configurable set of rules is run against each component. These rules can include Policy Group user criteria, Rule user Criteria, additional Rule Criteria, and Rule actions.

Discussion

User matching in MailMarshal is relatively efficient. Direct matching is better than large numbers of wild carded matches. So, the following user matching examples are fine: user@domain.com, *@domain.com (this is a special case - direct domain match). Avoid using lots of user*@domain or user@*.domain constructs. Unless you have tens of thousands of users, user matching is not a performance issue. MailMarshal will easily handle direct matching of 100,000 names without a problem, although you would not normally configure MailMarshal this way. Typically, you match whole domains.

It is also usually more efficient to have standard incoming and outgoing policy groups, and then set up exceptions in the Rules. Exceptions can be domains, or small user groups. For example:

When a message arrives
   Except where addressed either to or from 'management group'
etc

Additional Rule Criteria are where the biggest gains can be made. Choice of virus scanner is particularly important.

Most installations use a virus scanner with MailMarshal, and some use more than one.

To ensure the best performance with virus scanning, you should use only one virus scanner. You should use a DLL based scanner such as McAfee for Marshal. DLL based scanners are generally 10 times faster than command line scanners, because they do not have to start up each time they are called.

For example: With a command line scanner, a new Windows process has to be started for each message. The scanner then has to load and unpack and check its signature files (say 970 milliseconds elapsed), then it finally gets around to scanning the files we asked it to (say another 30ms).  

On the other hand, a DLL scanner is already memory resident (zero millisecs to load and check signature files) it scans the files immediately taking say 50 ms. Although the scanning time may be higher, in a MailMarshal environment the DLL scanner is still a clear winner.  

The TextCensor takes up the next biggest chunk of processing time.  If possible, avoid lots of small scripts and create a few larger scripts, as each script requires the input files to be parsed. Also, try to avoid very short matches with wildcards. For instance, do not use f* in a script, as it has to do more work for any word that starts with a f.  Instead try to enumerate all the cases in the script or at least increase the word stem length. If you wanted to spot "free mail", "fast mail", "faster mail" then "f* mail" works but listing all three cases is a lot better. A good compromise might be "free mail" and "fast* mail".

DNS Blocklist lookups as used by Receiver Rules and the SpamCensor can also be a major source of delay, because DNS lookups take time. It is essential to use a local and well connected DNS server for SEG lookups. To minimize impact on performance, use only one DNS blocklist. You can also purchase a local feed of some DNS blocklists. Having a local copy would significantly improve performance.

Note: Current versions of MailMarshal include a local cache for DNS information that can help substantially with repeated lookups.

More Information

A more detailed discussion of the impact of rule conditions and actions, virus and malware scanning, and also of the Image Analyzer add-in, is provided in the MailMarshal Policy Implementation Sizing Guide.

The Receiver and the Sender

There is not much to tune here. These services open sockets to other machines on the Internet. Sockets are resources allocated from the non-paged pool in Windows. The non-paged pool is fixed at 1/8 of physical memory. Sockets are consumed at a rate of two for every mail message (one in and one out). Each socket enters a timed wait state for around two minutes as it is released. Therefore if we are sustaining two messages per second, we will have 480 sockets in a timed wait state, plus maybe another 120 active sockets.

Add to this the resources for the threads and open files to support the active connections. Remember that sockets consume memory. For higher throughput on 64 bit systems that can address a lot of memory, you may need to add memory to support the greater number of sockets.

If the Receiver is slow to respond, this may be because you have enabled DNS Blocklist or reverse PTR lookups. This is another reason to ensure you are using a local and reliable DNS server for MailMarshal lookups.

How Many Threads?

Administrators often think that increasing the number of threads for the MailMarshal processes will give better performance. MailMarshal provides two standard sets of thread settings, one for smaller and one for larger sites. You can configure this setting in the Configurator > Server and Array Properties > Advanced > Advanced Options > Server Threads.

• For the Engine, current versions of MailMarshal default to 2x the number of processor cores. Based on experience, more Engine threads do not help. Very large core counts will not help in most cases: there just isn't that much mail to process.
  
  • Caution: Sophos for Marshal should be limited to 10 Engine threads due to a cap on the number of instances of the scanner.
• You can configure additional Sender and Receiver threads, but be aware that additional threads consume system resources such as memory.
• If the Sender connection latency is very high, ensure your DNS PTR record is set up correctly. Many sites try to do reverse lookups when you connect.

If MailMarshal cannot send mail quickly to an internal email server it is probably because the internal server cannot resolve who is connecting to it. Ensure the PTR record for the MailMarshal server is in the DNS that your internal server uses, or alternatively add the MailMarshal host into the internal server's hosts file.

Database Logging

SQL Server Express Edition, which ships with MailMarshal, is a free version of Microsoft SQL. It has a limited maximum database size (10GB in current versions). If you are concerned with performance then chances are you will also be logging enough data to exceed this limit.  Depending on your data retention needs, you may need to purchase SQL Server.

For details of the typical volume of logging data and SQL server software recommendations, see Q10829.

In current versions of MailMarshal, all database logging is through the Array Manager.

• To optimize database activity, if you are using SQL Express on the MailMarshal server, put the database on its own physical disk.
• You can get a significant boost in performance by installing the Array Manager with SQL Server on another machine.
• SQL "always on" configurations can be slower because of mirroring requirements.

By default MailMarshal retains logging information for 100 days. However, if you archive messages for longer, MailMarshal keeps database records about these messages as long as they are archived.

• If you do not require long reporting history or archives, configure the appropriate settings.
  • For longer retention of forensic archiving with no local resource requirement, consider the Trustwave Cloud Email Archiving service.
• Do set up a regular SQL server maintenance plan with backups and log truncation.
• Do not skimp on free disk! Many database operations such as backing up, restoring, or moving to a new version of SQL software will require at least enough free disk to replicate the database.

Also, allow significant extra memory for SQL. SQL Server will use all the memory it can get to keep tables and queries cached. Memory is particularly important when the Console and Reports are running.

Multiple disks and partitioning

To optimize disk usage, on a single server MailMarshal installation you would want the following disk partitions:

• A system partition for all the software, Windows operating system, SQL, and MailMarshal
• A partition for the Unpacking directory
• A partition for the SQL database
• A partition for all the other MailMarshal quarantine folders

Ideally, the Unpacking partition and the SQL database would each be on a separate physical drive. Also, if data protection for archives is important, the quarantine folders should be on a mirrored drive.

Multiple Servers

MailMarshal allows you to configure multiple email processing servers with the same Array Manager, configuration, and database. Multiple servers allow redundancy and scalability.  A decision to use multiple servers will depend on the existing email environment and what load balancing and/or redundant features are already in place.

Based on testing by Trustwave, a MailMarshal installation with all services including SQL Server on a single server will comfortably handle over 10,000 users if properly resourced. Moving the Array Manager and SQL Server to another machine will significantly increase the throughput and the number of users that a single node can handle.

Notes:

Of historical interest: In 2004, Marshal was processing mail for 750 users using MailMarshal 5.5 on a Pentium 166 with a 13GB IDE disk, with both a command line and a DLL based virus scanner running.  This configuration is no longer recommended!