But there is another advantage to switching to text. They take up considerably less disk space. Here are some figures:

Number of Records in 235 MBs of log data:

235 MB of TMG’s W3C text logs contains 326,824 records. An SQL Express database of the same size (mdf and ldf files) contains only 40,308 records. In other words, w3C text logs can store over 8 times as much data in the same amount of disk space.

A rule of thumb:

By switching to W3C text logs, the disk space taken by your log files will be roughly 12% of the SQL Express or MSDE log files. This can be reduced even further by compressing your text logs.

• MSDE/SQL logs: budget for 5 KB per record
• W3C Text logs: budget for 0.71 KB per record

How many records your ISA or TMG server creates per day will depend on the number of users in your organization and how much traffic they generate, but about 16,000 records per user is a reasonable estimate.

A real world example

If you are hitting 500 GB of SQL Express/ MSDE logs per month (about 86,128,205 records), simply switching to W3C text logs will reduce this down to 61 GB.

Once imported into a WebSpy Storage, the storage size would be roughly 53 GB (87% of the original W3C text logs).

With NTFS compression applied to the Storage folder, the WebSpy Storage would be roughly 13.4 GB (22% of the original W3C text logs).

Applying NTFS compression to your WebSpy Storages folder is certainly a good idea. This does not impact performance. If anything, it may improve performance slightly as there is less disk fragmentation within the storage.

Disadvantages and Alternatives

Please be aware that by changing your logging to text, the default reporting functionality within TMG will no longer work. However, the reporting supplied by WebSpy Vantage should more than adequately replace this feature.

If you are still concerned about changing the logging method, you can utilize a script published by Microsoft to convert your SQL Express logs to W3C text.  You can then keep the text logs and set some more stringent data retention policies on the SQL Express logs, such as clearing logs every week. You can download this script as part of the Microsoft Forefront Threat Management Gateway (TMG) 2010 Tools & Software Development Kit.

Additional Resources

• Here’s a great article by Marc Grote at isaserver.org on the pros and cons of the different logging options in ISA and TMG. It also takes you through how to exclude fields to reduce the amount of data being logged:
  http://www.isaserver.org/tutorials/Microsoft-Forefront-TMG-Logging-options-Forefront-TMG.html
• Also take a look at Richard Hicks’ blog regarding MSDE performance with ISA Server 2006:
  http://tmgblog.richardhicks.com/2009/10/31/msde-performance-with-microsoft-isa-server-2006/
• Here’s another article on isaserver.org by Richard Hicks on the logging enhancements in TMG 2010
  http://www.isaserver.org/articles/Logging-Enhancement-Microsoft-Forefront-Threat-Management-Gateway-TMG-2010.html

The figures above were produced using some sample logs received from customers with similar (but not exactly the same) logging settings. If you have changed to text logging, I’d be very interested to hear the sort of disk savings you are seeing, and I’m sure others would to. So please leave a comment below.

So let’s use WebSpy Vantage to drill into that Anonymous user and find out what is going on.

One way to do this is to run an Ad-hoc analysis on the Summaries screen and drilldown into the Anonymous user to view all the information about that user. However, TMG and ISA tend to log a lot of information that may not be relevant to this particular investigation, so I’ve created some report templates (one for ISA and one for TMG) and a set of Aliases that pull out some relevant information.

Download our Anonymous Traffic Investigation Report

If you’re running WebSpy Vantage download the Anonymous Traffic Report Templates & Aliases

Then open the .Templates file on the Reports tab, and the .Aliases file on the Aliases tab. Once you have both files opened, go to the Reports tab and click either the ‘Anonymous Traffic Investigation (ISA)’ or the ‘Anonymous Traffic Investigation (TMG)’ report. Then click the ‘Generate report’ link and run the report template on your ISA or TMG storage.

The report gives you the ability to drill into the Allowed, Denied and Failed traffic to see a list of the unauthenticated IPs, Sites, Rules responsible for blocking or allowing the traffic, unauthenticated Applications and Result Codes.

Main causes of anonymous traffic

What you will probably find is that most of the Anonymous traffic is being denied by your TMG or ISA firewall. When a client first requests a web page, the proxy will challenge the client for authentication. These events are often logged with the result code 12209 meaning ‘authorization is required to fulfill the request’. These requests are therefore denied by the proxy until the client’s credentials are authenticated.

Have a look at the amount of traffic being denied and then checkout the Result Codes associated with the denied traffic. Chances are you’ll see ‘proxy authentication required’ appear predominantly.

If you also look at the Applications section you may also find that Windows Updates are sailing through your TMG or ISA firewall unauthenticated.

Filter out unauthenticated traffic from Reports

The most logical next step is to filter out the information you do not want in your reports. You’ll probably still want to include Windows Update traffic in your reports, but you’re probably not so interested in the ‘proxy authentication required’ information. So let’s filter that out.

To do this:

1. Go to the Reports tab and select the report you want to filter (such as your Organization report)
2. Click ‘Edit Template’, then click ‘Template Properties’.
3. In the filter section at the bottom of the dialog, click Add | Field value filter.
4. Select the ‘Result Code’ summary and select the Status Code Names (ISA-FTMG) alias.
5. On the toolbar, search for Authorization, and check the following two items:
   • The server requires authorization to fulfill the request. Access to the Web Proxy filter is denied.
   • The server requires authorization to fulfill the request. Access to the Web server is denied. Contact the server administrator.
6. Ensure the ‘Exclude’ radio button is selected and click OK.

If you decide that you don’t care about seeing ANY unauthenticated traffic in your reports, you can always simply filter out the Anonymous user from your reports.

To do this:

1. Go to the Reports tab and select the report you want to filter (such as your Organization report)
2. Click ‘Edit Template’, then click ‘Template Properties’.
3. In the filter section at the bottom of the dialog, click Add | Field value filter.
4. Select the ‘Username’ summary.
5. On the toolbar, click Add and type ‘anonymous’. Click OK.
6. Ensure the Exclude radio button is selected and click OK.

Hopefully this article improves your understanding of the ‘anonymous’ user, and gives you some actions to take for your specific reporting situation.

If you have any questions, please leave a comment below.

Check your version in Help | About. If you are running 2.2.0.27 or above, then you already have this update. If not, make sure you update to your software by selecting Tools | Check for updates.

TMG or UAG? What is the difference?

TMG is an outgoing proxy that protects your internal users from malware, viruses and the like. TMG generates some great web proxy log files to import into WebSpy Vantage allowing you to monitor where your users are going on the Internet, how much they’re downloading etc.  TMG, unlike ISA, now has deep packet inspection for HTTPS traffic, plus a bunch of other new features.

UAG is an incoming proxy that provides employees, partners and vendors secure remote access to corporate resources such as Outlook Web Access (OWA) and Sharepoint (MOSS). It utilizes the TMG engine, but this is mainly just to protect the UAG server (more on this topic here http://technet.microsoft.com/en-us/library/ee522953.aspx).

TMG can also publish your OWA and MOSS sites, but this is no longer recommended by Microsoft. They recommend using a dedicated UAG server to perform this function.

Upgrading to TMG

If you’re thinking about migrating your ISA server (2004 or 2006) to TMG, you may like to check out this migration guidance video with Mohit Saxena (Senior Technical Lead) and Jim Harrison (Program Manager). http://edge.technet.com/Media/ISA-to-TMG-Migration-Guidance/

Microsoft Forefront TMG Migration Video

Reporting on TMG

If you’re using TMG at the moment, we invite you to analyze your web proxy and/or firewall logs using WebSpy Vantage and tell us what you think!  Download your copy of WebSpy Vantage here, and import your logs using the Microsoft FTMG format:

Microsoft Forefront Threat Management Gateway