What is the WELF log file format?

This article applies to:

• WebMarshal 6.X
• WebMarshal 7.X
• Security Reporting Center 2.X
• WebTrends Firewall Suite

Question:

Information:

WELF is the WebTrends Enhanced Log file Format.

The WELF Reference defines the WebTrends industry standard log file exchange format. Any firewall or VPN system logging to this format will be compatible with Firewall Suite 2.0 and later, Firewall Reporting Center 1.0 and later, and Security Reporting Center 2.0 and later.

WebMarshal 6.X and WebMarshal 7.X "Traffic Logging" logs can be created in the WELF format.

• For full details of the fields logged by WebMarshal, see Trustwave Knowledgebase article Q21119.

Log File Format

A log file is made up of records. Each record makes up a single line of the file. Records must be in chronological order. The earliest record is the first record in the file; the most recent record is the last record in the file. The WebTrends Enhanced Log Format places no restrictions on log file names or log file rotation policies.

Record Format

A record is terminated by the character sequence carriage return-line feed (0x0D-0x0A). There may be no carriage-returns or line-feeds within a record; this format results in a single record per line.

Each record is made up of fields. The record identifier field (id=) must be the first field in a record. All other fields can appear in any order.

Aside from a few required fields, you can decide which optional fields are included in the record. You may want some fields to appear in only certain records because they are only relevant to certain types of activity (for example, the operation on an HTTP request).

Some optional fields may be left out if the firewall vendor chooses, but doing so typically results in reports that are less complete. Refer to the field descriptions to determine which fields are required for tables.

Sample Record
(In a real log file, the record would reside on one line.)

Note: This sample record should give you a sense of what a record looks like. It does not contain all the fields that are available and described in this document. Additional sample records are fully documented in the HELP Index built into the Firewall Suite product.

id=firewall time="2000-2-4 12:01:01" fw=192.168.0.238 pri=6 rule=3 proto=http src=192.168.0.23 dst 6.1.0.36 rg=www.webtrends.com/index.html op=GET result 0 rcvd=1426

Required Fields (WELF)

• Record identifier
  The id= field identifies the type of record. For log files conforming to this document, the type will always be firewall. For example,

  id=firewall

• Date/time
  The time= field shows the date and time of the event, in terms of local time. The form of the date/time field is shown below (Note: Since this field contains spaces, it must be enclosed in double quotes):

  time="yyyy-mm-dd hh:mm:ss" (where):

yyyy: year (always 4 digits)
mm: number between 1 and 12 (inclusive) to represent the month (1 or 2 digits)
dd: day of the month, 1 based (1 or 2 digits)
hh: hour, based on 24-hour clock (1 or 2 digits)
mm: minute (1 or 2 digits)
ss: second (1 or 2 digits)

For example,

6:00 a.m. on January 1, 2000 would be represented as:
time="2000-1-1 6:0:0"

It could also be represented as:
time="2000-01-01 06:00:00"

6:00 p.m. on the same day would be represented as:
time="2000-01-01 18:00:00"

• Firewall IP address or name
  The fw= field identifies the firewall that generated the log record. This is most often represented as an IP address or a machine name. Firewall Suite uses this field for licensing. The user's licensing is simplified if the firewall is consistent in logging this field. In other words, a particular firewall should always log its IP address or always log its machine name, but not both. If a firewall is logging its IP address, it should always log the IP address of the internal network interface or always log the IP address of the external network interface, not a mixture of the two.

  An example using the IP address of the firewall:
  fw=192.168.0.238

  An example using the machine name of the firewall:
  fw=ACME_FIREWALL

• Priority of the record
  The pri= field specifies the priority of the event. The following is a list of valid values:

0 - emergency
1 - alert
2 - critical
3 - error
4 - warning
5 - notice
6 - information
7 - debug

Messages are placed in various tables based on the priority. Messages with priorities 0, 1, and 2 are included in the critical errors tables, messages with priorities of 3 and 4 are included in the errors and warnings tables, and messages with priorities of 5, 6, and 7 are included in the informational messages tables. For example:

pri=0
pri=5

Optional Fields (WELF)

The following fields for the WebTrends Enhanced Log File Format are optional:

• Rule
  The rule= field specifies the rule that triggered the log entry. This field is used to generate tables that help the user understand that the rules they have set up are working properly. Three tables are based on the rule field: internal IP addresses triggering firewall rules, external IP addresses triggering firewall rules, and protocols triggering firewall rules. Most firewalls log this field as an integer identifying a particular rule. However, rules could also be identified by name and logged as such in this field. For example:

rule=4
rule=12

• Protocol
  The proto= fields specifies the protocol used by the event. A large number of tables and graphs depend on the presence of the protocol field. Although this it is not a required field, without it, reports lack important information. Some firewalls do not log the protocol, but log the service. If this is the case, the service can be logged in this field. For example,

  proto=http
proto=ftp
proto=snmp

  Default protocol mapping
  Firewall Suite includes a file called wtprotocols.txt that maps protocol fields found in log files to types of traffic that appear in reports (for example, pop3 in the log file is displayed as e-mail in the report).

The following is an extract from the wtprotocols.txt file that ships with WebTrends Firewall Suite:

[web]
http
https
80/tcp
[email]
pop3
smtp
smap
[ftp]
ftp
ftp-data
[telnet]
telnet
[realaudio]
realaudio

Map new protocols
Your log files may contain protocols not included in this file. Map them using the Protocols tab in the Firewall Options dialog in the GUI (a mapping changes file named protocols.txt is created). Or you can create the protocols.txt file and map new protocols. Follow the syntax of the wtprotocols.txt file: use the types of traffic designations enclosed in square brackets and list new protocols for that type of traffic, each on a single line. Note: Unmapped protocols are grouped for reports in a type of traffic designation called "other."

• Duration

The duration= field specifies the time that is required to perform the operation, in seconds. For example, for an FTP file transfer, the duration is the amount of time used to perform the transfer. Although Firewall Suite tracks this field, it is not currently shown in any tables or graphs. We recommend that if the this information is available, it should be logged so that it can be used in the future. For example, to indicate that an operation required 3 minutes exactly, the duration field could look like this:

duration=180.00

or like this:

duration=180

• Bytes Transferred

The sent= field specifies the number of bytes transferred from the source to the destination. For example,

sent=1426
sent=512

• Bytes Received

The rcvd= field specifies the number of bytes transferred from the destination to the source. For example:

rcvd=1426
rcvd=512

• Source

The src= field specifies the IP address that generated the event. For example:

src=192.168.0.23
src 6.0.2.1

• Source Name

The srcname= fields is a more user-friendly version of the src= field. For example,

srcname=mickm@example.com
srcname=www.example.com
srcname=JIMS_SYSTEM

• Destination

The dst= field specifies the IP address that received the event. For example,

dst=192.168.0.23
dst 6.0.2.1

• Destination Name

The dstname= field is a more user-friendly version of the dst= field. For example,

dstname=EXAMPLE_SERVER
dstname=www.example.com

• User

If users are authenticating through the firewall, then the authenticated user name can be logged in the user= field. For example,

user=JohnB
user=MarySmith

• Operation

For HTTP and FTP requests, the op= field is the operation such as GET, POST, etc. For example,

op=GET
op=POST

• URL Accessed

For HTTP and FTP requests, the arg= field is the URL accessed. For example,

arg=/PRODUCTS/GOODIES/GIFS/IWAWARD2.gif
arg=/PRODUCTS/GOODIES/download.htm?Product=Standard

• Result Code

For HTTP requests, the result= field is the standard result code, such as 200 for success, 304 for returned from cache, etc. For example,

arg 0
arg=304
arg=404

• VPN

The vpn= field identifies a particular VPN. This value is used to generate tables showing the most highly used VPNs and tables correlating particular users to particular VPNs. For example,

vpn="NY Branch VPN"
vpn=Sales

• Type

The firewall vendor can use the type= field to cause records to be placed into the tables relating to VPN events or relating to firewall management events. (Other categories may be defined in the future.)
A record can be put into more than one category by separating values by commas.
The currently defined types are:

· vpn - the record is a VPN event.

· mgmt - the record is a firewall management event. For example,

type=vpn
type=mgmt
type=vpn,mgmt

• Message

The msg= field is the basis for the tables showing detailed Critical Events, Errors and Warnings, VPN events, and Firewall Management events. Firewall Suite generates summary tables showing these types of events. Firewall Suite will also generate detailed tables associating users with these events. To make this happen, the user(s) need to be identified using the Src=, Srcname=, Dst=, Dstname=, or User= fields. For example,

msg="VPN starting"
msg="Possible port scan detected"
msg="Firewall configuration changed"

• Referring Site

For incoming web records, the ref= field contains the referring site. For example,

ref=http://search.yahoo.com/bin/search?p=trends%20internet

• Agent

For incoming or outgoing web records, the agent= field contains the agent (usually the browser).

agent="SPRY_Mosaic/v8.32 (Windows 16-bit)"
agent="Microsoft Internet Explorer/4.40.308 (Windows 95)"
agent="Mozilla/3.0 (WinNT; I)"

• Category

The cat= field contains the categories to which the accessed site belongs. It is used only for firewalls or proxies capable of categorizing web sites. For example, www.msnbc.com might be categorized as "General News", "Investment", and "Entertainment". If a site belongs to more than one category, these categories should be given in the same cat= field, with a comma separating each category. Note: If a field contains spaces, it must be enclosed in double quotes.

Use this field should only if dst= (or dstname=) is also present.
For example:

dst=www.msnbc.com cat=News
dst=www.msnbc.com cat="General News"
dst=www.msnbc.com cat=News,Investment,Entertainment
dstname=www.msnbc.com cat=News,Investment,Entertainment

• Category Action

The cat_action= field contains the action taken for the category value of the cat= field. For example, access to gambling web sites may be blocked.
This field should only be present if cat= is also present.
Possible values for this field are: block and pass.
For example:

dst=www.gambling.com cat=Gambling cat_action=block
dst=www.msnbc.com cat=News cat_action=pass

• WebMarshal Cache Result

The wmcache= field contains the result of a WebMarshal cache lookup request.

• Cache status is not written for HTTPS or FTP requests. Only HTTP supports caching.
• This custom field cannot be reported by Security Reporting Center.

Possible values for this field are: HIT, MISS, REFRESH_HIT or REFRESH_MISS.

• wmcache=HIT indicates that the item was served from cache without checking the origin server.
• wmcache=MISS indicates that the item was not in cache and had to be retrieved from the origin server.
• wmcache=REFRESH_HIT indicates that the cache item required revalidation, and that revalidation was successful.
• wmcache=REFRESH_MISS indicates that the cache item required revalidation, and that the origin server sent back new data.

• WebMarshal Classification Result

Two fields record the result of WebMarshal classification:

dclass= indicates a WebMarshal Domain Classification
fclass= indicates a WebMarshal File Classification

If WebMarshal records multiple classifications, they are included as a comma separated list within double quotes.

For instance: dclass="Safe Sites,Search Engines"

Sample Records

See the document entitled WebTrends Enhanced Log Format (WELF) For Firewalls & VPNs or the HELP Index built into the Firewall Suite product for examples of records that conform to the WebTrends Enhanced Log File Format. Included among the examples provided are:

• Sample Web Records
• Sample E-mail Records
• Sample Telnet Records
• Sample FTP Records
• Sample RealAudio Records
• Sample Management Records
• Sample Error Messages

This article was previously published as:

NETIQKB1301