TLS/SSL sending connection errors with SEG 7.5.7 and above

This article applies to:

• Trustwave SEG 7.5.7 and above (issue fixed by default in 8.X and above)
• Sending email via TLS

Symptoms:

• Sending via TLS fails.
• SSL connection errors:
  • SSL_ERROR_SYSCALL
  • Error: SSL_ERROR_SSL error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

Causes:

• The remote system is using outdated and insecure TLS cipher suites.
• SEG 7.5.7 includes a TLS/SSL library that contains new Diffie-Hellman cipher suites. The remote systems are not compatible with these suites and fail to negotiate.
• Software known to be affected includes Exchange 2003 and older Sendmail installations.

Resolution:

SEG version 8.0 and above corrects this issue by default, by excluding DH authentication as explained below.

• Note that the TLSCipherList functionality described below overrides default settings and will not be changed on upgrade. If you added this setting Trustwave recommends you remove it after upgrading (be aware that the Registry hive location changes at 8.0; see article Q10832).

With 7.5.7, you can configure SEG to communicate with these outdated servers by changing the available TLS cipher suites for sending. The changes remove the affected ciphers from the list available to be used.

• Note: Do not change the enabled SSL/TLS versions, as that change will not have any effect on the issue.

For Exchange 2003, the strongest list of ciphers is:
ALL:!aDH:!ADH:!aNULL:!RC4:!IDEA:!MD5:!EXPORT:@STRENGTH

For some Linux/Sendmail systems the following list has been reported to work:
ALL:!DH:!aNULL:!RC4:!IDEA:!MD5:!EXPORT:@STRENGTH

Procedure:

To set specific cipher suites for TLS:

1. On the Array Manager, edit the Registry (10.X: use Advanced Settings in the Management Console)
2. Navigate to the SEG Sender key:
   • In version 8.X: HKEY_LOCAL_MACHINE\SOFTWARE\Trustwave\Secure Email Gateway\Default\Sender
   • 10.X: value names have the prefix Sender. (Sender dot).
   • For full details of the location for each product version, see article Q10832.
3. Add a String value (REG_SZ) TLSCipherList
4. Set the string value to a valid cipher string.
5. Commit configuration and then restart the Sender service.
6. When TLS is enabled for a service, the custom cipher list is logged on service startup. For example:
   3952 09:07:24.264 TLS enabled: using TLSv1 TLSv1.1 TLSv1.2  (Custom cipher list: ALL:!aDH:!ADH:!aNULL:!RC4:!IDEA:!MD5:!EXPORT:@STRENGTH)

Warning: If an entry exists and is not valid, the Sender service will stop. This warning applies even if TLS is disabled for the service.

If the registry value described entry is blank or does not exist, the "minimum cipher strength" setting from TLS properties (low, medium, or high) is used. However, due to other restrictions, current versions of MailMarshal effectively do not use any "low" security ciphers by default.

Notes:

• As always, take due care when editing the Registry. Trustwave recommends that you back up the Registry before making changes.
• You should update the TLS configuration on the affected Exchange or Sendmail server if possible, or urge the owner of the server to do so.

The cipher listings suggested include the following entries: