How do I read MailMarshal Exchange Log Files?

This article applies to:

• Trustwave ECM/MailMarshal Exchange 7.X

Question:

Information:

This article provides a short overview of how to read MailMarshal Exchange Log files.

Each MailMarshal service creates a text log file. A new file is created at least daily. If the log reaches 10MB, a new file is started. The logs are deleted after five days by default. The logs are located in the logging subfolder of the installation (for instance, c:\Program Files (x86)\Marshal\M86 MailMarshal Exchange\Logging). The files are named with the name of the service, date, and sequence letter (for instance, MEXControllerJun13b.log).

You can change the location using the Server Tool. You can change the file size and retention with Registry settings. Contact Support for details.

Note: If you set up a rule to archive or quarantine messages, you can choose to save log files. When you select this option, MailMarshal will extract the log information for each message to a separate file. You can view these extracts in the MailMarshal Console. Normally this is the best way to retain logging information for longer than 5 days.

The logs are useful for troubleshooting purposes. Processing information and problems encountered are recorded in the log files.

What is logged?

• The MEXArrayManager log includes information about the database connection and database writing, directory connector updates, SpamCensor updates, configuration changes, and configuration reloads.
• The MEXController log is present on email processing servers and includes information about the array manager connection and configuration retrieval.
• The MEXTransportAgent log contains information about messages that were retrieved from Exchange for processing, and the message ID and file name assigned for each message. 
• The MEXEngine log contains a record of rule processing for each message. This includes for each message a list of the rules that were checked, what rules if any triggered, and what actions MailMarshal performed.
• Other tools and services, such as the Message Release executable, also create log files.

How does email flow?

To troubleshoot email processing, you should understand how MailMarshal processes email.

1. When an email message is sent within Exchange, it passes to the Replay directory where third party Transport Agents can retrieve it The MailMarshal Exchange Transport Agent monitors this directory for messages not processed by MailMarshal Exchange (or that have been changed by other Transport Agents since MailMarshal Exchange processed them).

2. For any message requiring processing, the Transport Agent creates a MML file containing the message, in the Incoming folder.

3. The Engine picks up each MML file in turn, unpacks the email and its contents by layers, then applies the rules. After actioning the rules, if a message is not blocked the Engine places the file in the ProcessedOk folder.

4. Another Engine thread returns messages to the Agent via the Replay directory 
   • In version 7.1, where possible messages are returned through an object in memory and not through the Replay directory.

Note: The message number (starting with b00 for ordinary messages) is unique. This number identifies an individual message throughout the MailMarshal system. Therefore when looking through the logs (or in the Console) you can track particular messages through each of the services by searching via the message number.

In the log files you will also notice the 4 digit number (for example 0116) before the time stamp of each line of log. This is the thread ID. This number is also useful to track when reading the raw logs. Normally MailMarshal services are working on multiple threads at the same time. To trace a particular message you need to follow the thread ID which may skip lines in the logs. If you are interested in engine action, it is much easier to archive files and view the log excerpts, because these excerpts contain only the information about the specific message.

Examples

MEXTransportAgent log for a new email from person1@example.com

0055 13:24:49.831 A new instance of MEXTransportAgent was constructed.
0055 13:24:49.831 OnSubmittedMessageHandler() - Processing email
0055 13:24:49.831 OnSubmittedMessageHandler() - subject <a test>
0055 13:24:49.831 OnSubmittedMessageHandler() - from <person1@example.com>
0055 13:24:49.878 OnCategorizedMessageHandler() - Processing email
0055 13:24:49.878 OnCategorizedMessageHandler() - subject <a test>
0055 13:24:49.878 OnCategorizedMessageHandler() - from <person1@example.com>
0055 13:24:49.878 CATEGORIZED: CheckProcessed returned <False>.
0055 13:24:49.878 CATEGORIZED: HandleCategorizedMessage: New message - passing to engine for processing
0055 13:24:49.894 CATEGORIZED: Generating new MM-Id.
0055 13:24:49.894 CATEGORIZED: Extracting msg B4d0173510000.000000000001.0001.mml
0055 13:24:49.925 CATEGORIZED: As: C:\Program Files (x86)\Marshal\M86 MailMarshal Exchange\Queues\Incoming\~B4d0173510000.000000000001.0001_1400.1506562.1.mml
0055 13:24:49.925 CATEGORIZED: Creating mml message B4d0173510000.000000000001.0001.mml from C:\Program Files (x86)\Marshal\M86 MailMarshal Exchange\Queues\Incoming\~B4d0173510000.000000000001.0001_1400.1506562.1.mml
0055 13:24:49.972 CATEGORIZED: Adding envelope to: C:\Program Files (x86)\Marshal\M86 MailMarshal Exchange\Queues\Incoming\~B4d0173510000.000000000001.0001_1400.1506562.1.mml
0055 13:24:50.019 CATEGORIZED: Passing message to engine.

MMEngine log:

1940 13:37:57.921 Thread 2 already working on B000000053.00000001.mml
0116 13:37:57.911 Thread 2 Starting to unpack <B000000053.00000001.mml>
0116 13:37:58.262 Type=MAIL, size=407272, Name=B000000053.00000001.mml
0116 13:37:58.262 Type=MHDR, size=582, Name=MsgHeader.txt
0116 13:37:58.262 Type=MBODY, size=2, Name=Quoted-Printable
0116 13:37:58.262 Type=MBODY, size=318, Name=Quoted-Printable_1
0116 13:37:58.262 Type=ZIP, size=1223, Name=example.zip
0116 13:37:58.262 Type=XLS, size=11776, Name=example.xls
0116 13:37:58.262 Type=JPG, size=295017, Name=24.jpg
0116 13:37:59.812 1 user(s) match rule -Test
0116 13:37:59.812 Name=U1\B000000053.00000001.mml (MAIL,407272) False
0116 13:37:59.812 Name=U2\MsgHeader.txt (MHDR,582) False
0116 13:37:59.812 Name=U2\Quoted-Printable (MBODY,2) False
0116 13:37:59.812 Name=U2\Quoted-Printable_1 (MBODY,318) False
0116 13:37:59.812 Name=U2\example.zip (ZIP,1223) False
0116 13:37:59.812 Name=U3\example.xls (XLS,11776) False
0116 13:37:59.812 Name=U2\24.jpg (JPG,295017) False

When the MailMarshal Engine processes an email, it first declares the contents of the email including the type, size and name (as you can see in the entries above at time 13:37:58.262). This information is essential to any further actions such as unpacking. (For instance, the file could be corrupt. You can confirm what MailMarshal has found by looking here).

The Engine unpacks the email into its parts and attachments, then unpacks each part for as many layers as are required. Each of these layers is normally identified by U1 (level 1) U2 (level2) and so on.

• In U1 you would expect the entire message (mml file), in U2 you would expect to see the email parts (msg header, body), U3 would have the next level of attachment contents, in this case example.xls and 24.jpg (being the items attached in the email).

The Engine log records which rules have been applied to a message. For each rule, the log shows the number of users this rule applies to. In the example above 1 user(s) match rule -Test. ("Test" being the rule name)

Each item in the message contents is scanned using the criteria defined in the rules that match. If the rule triggers for that item, it is actioned as "True" and if not "False". Depending on the actions if any, the message is passed through the remaining rules.

Notes:

See additional information in the following articles:

• Q14022 - General tips on reading MailMarshal service logs
• Q10545 - How does MailMarshal Exchange work? (message flow)