This article applies to:
- WebMarshal 7.X
- FileFilter
Question:
- What is the memory usage of FileFilter?
- What are the practical limits on FileFilter URL lists?
Information:
WebMarshal FileFilter is designed to allow large numbers of URLs to be loaded from a local source.
- When thousands of URLs are included in standard WebMarshal categories, configuration reload and Console response can be very slow.
- Maintaining the URLs in files outside the WebMarshal configuration avoids this issue.
To allow good lookup performance, the FileFilter lists are kept in memory on the processing servers. The database format uses significantly more storage than the text URLs. Processing servers must have enough memory to hold the lists.
Memory requirements and performance
Memory requirements for FileFilter are generally linear. Lookup performance is generally not affected by the size of the database.
- Each URL with no path part consumes about 3.6 KB.
- Each URL with a path part consumes about 6.7 KB.
- Dividing the database into categories adds overhead.
If you are planning to use large FileFilter lists, be sure that the processing servers have enough memory.
Examples:
- 30,000 URLs in a single category, without path part, consume about 100 MB of memory.
- 30,000 URLs in a single category, including path part, consume about 180 MB of memory.
- 300,000 URLs in a single category, without path part, consume about 1 GB of memory.
- 10 categories of 30,000 URLs each, without path part, consume about 1.3 GB of memory.
Update behavior:
When the database is updated, a second copy is created. Memory used by FileFilter doubles for a few seconds.
- The update is performed daily by default, between 2AM and 3AM server time.