Changing the Nested PDF unpacking limit


This article applies to:

  • Trustwave MailMarshal (SEG) 
  • Trustwave ECM/MailMarshal Exchange 7.X
  • PDF documents

Question:

  • How do I change the limit of recursive unpacking of nested items in PDF documents?
  • Error message: PDF Error - Document too deeply nested

Information:

PDF documents can contain nested items. Some automatically generated documents use thousands of levels of nesting. These documents can cause MailMarshal unpacking to fail.

MailMarshal has a default limit of 500 levels of recursive unpacking of nested PDF elements. If a document exceeds this level of nesting, MailMarshal will move the message to the Deadletter - Unpacking folder and log the following message: PDF Error - Document too deeply nested.

You can change the allowed level of nesting. However, note the following recommendations:

  • If you set the value above approximately 1000 to 1200 levels, the PDF unpacking is likely to start failing due to memory problems.
  • If you set the value below 200 levels, MailMarshal may not successfully process moderately complex PDF documents, resulting in the '"PDF Error - Document too deeply nested" exception.

Procedure:

To change the allowed level of recursive unpacking of nested PDF elements, set an entry in the Registry on the MailMarshal Array Manager computer.

  • In MailMarshal 10.0 and above, open the Management Console and navigate to Advanced Settings. Add a new value:
    • Name: Engine.MaxPDFNesting
    • Type: Integer
    • Value: The desired level of recursion
  • In MailMarshal 8.X and below, and ECM, open the Registry Editor on the Array Manager. Within the base registry key, navigate to \Default\Engine
    • In version 8.X: HKEY_LOCAL_MACHINE\SOFTWARE\Trustwave\Secure Email Gateway\Default\Engine
    • For information about the registry location for each version, see article Q10832.
    • Enter the desired level of recursion as a new DWORD value named MaxPDFNesting. Use Decimal numbers to avoid confusion.
  • Save your registry settings or configuration settings.
  • Commit the configuration changes and restart the MailMarshal Engine service on each node.

As always, take due care when editing the Registry.


Last Modified 4/1/2020.
https://support.trustwave.com/kb/KnowledgebaseArticle12243.aspx