What is the log analysis memory usage with Firewall Suite?

This article applies to:

• WebTrends Firewall Suite 4.X

Question:

Information:

Trustwave recommends that you have available memory at least 1/10 of the log file size as a general guide. Many factors can influence this number such as format, content, filters applied by the profile, structure of the Web site, number of unique visitors and number of Web pages. For example: 100MB of log files will generally require 10MB of free memory, 1GB will generally require 100MB of free memory. While virtual memory is helpful, disk swapping will impact performance.

A simple test shows that the amount of memory used increases logarithmically with log file size.

During the initial phase of analysis, tables are being created and populated. Memory is consumed quickly. Once they have been created and used for a while, the tables grow essentially linearly with each new line processed.

For example, a test was conducted with a 450MB NSCA format log file with an average line length of 100 characters. After the initial memory explosion, the Log Analyzer consumed about 10KB for every 1000 lines of log file. (Or, 10KB for each 100KB in the file.)

This provides a simple rule of thumb:

• In short, enough memory is needed to hold 1/10 of your log file on average.

This article was previously published as:

NETIQKB344