This video follows on from #3, and will cover the Profiles section of the software, and how to use site categorization.

In this video we’ll look at how to import some data into a storage, use the Summaries section, and start customizing Aliases.

This video is a high level overview, giving you a general insight into the software and its different parts.

You can also read through these steps on this page: Analyzing SonicWALL log files with WebSpy.

Creating and Importing SonicWALL log files

Analyzing SonicWALL log files

We intend to make some SonicWALL specific report templates available on our SonicWALL how to page soon.

Until then, feel free to create your own templates, or modify our existing web reports to include the extra goodies contained in the SonicWALL logs.

TIP: To modify an existing web report, right-click the report and choose ‘Duplicate template’. Then choose the “SonicWall Web” schema. You’ll then have a report template that you can modify to include all the SonicWALL summaries, such as Categories, and Source and Destination Interface.

If you need some assistance getting the report(s) you need, feel free to contact me, or support@webspy.com.

But there is another advantage to switching to text. They take up considerably less disk space. Here are some figures:

Number of Records in 235 MBs of log data:

235 MB of TMG’s W3C text logs contains 326,824 records. An SQL Express database of the same size (mdf and ldf files) contains only 40,308 records. In other words, w3C text logs can store over 8 times as much data in the same amount of disk space.

A rule of thumb:

By switching to W3C text logs, the disk space taken by your log files will be roughly 12% of the SQL Express or MSDE log files. This can be reduced even further by compressing your text logs.

• MSDE/SQL logs: budget for 5 KB per record
• W3C Text logs: budget for 0.71 KB per record

How many records your ISA or TMG server creates per day will depend on the number of users in your organization and how much traffic they generate, but about 16,000 records per user is a reasonable estimate.

A real world example

If you are hitting 500 GB of SQL Express/ MSDE logs per month (about 86,128,205 records), simply switching to W3C text logs will reduce this down to 61 GB.

Once imported into a WebSpy Storage, the storage size would be roughly 53 GB (87% of the original W3C text logs).

With NTFS compression applied to the Storage folder, the WebSpy Storage would be roughly 13.4 GB (22% of the original W3C text logs).

Applying NTFS compression to your WebSpy Storages folder is certainly a good idea. This does not impact performance. If anything, it may improve performance slightly as there is less disk fragmentation within the storage.

Disadvantages and Alternatives

Please be aware that by changing your logging to text, the default reporting functionality within TMG will no longer work. However, the reporting supplied by WebSpy Vantage should more than adequately replace this feature.

If you are still concerned about changing the logging method, you can utilize a script published by Microsoft to convert your SQL Express logs to W3C text.  You can then keep the text logs and set some more stringent data retention policies on the SQL Express logs, such as clearing logs every week. You can download this script as part of the Microsoft Forefront Threat Management Gateway (TMG) 2010 Tools & Software Development Kit.

Additional Resources

• Here’s a great article by Marc Grote at isaserver.org on the pros and cons of the different logging options in ISA and TMG. It also takes you through how to exclude fields to reduce the amount of data being logged:
  http://www.isaserver.org/tutorials/Microsoft-Forefront-TMG-Logging-options-Forefront-TMG.html
• Also take a look at Richard Hicks’ blog regarding MSDE performance with ISA Server 2006:
  http://tmgblog.richardhicks.com/2009/10/31/msde-performance-with-microsoft-isa-server-2006/
• Here’s another article on isaserver.org by Richard Hicks on the logging enhancements in TMG 2010
  http://www.isaserver.org/articles/Logging-Enhancement-Microsoft-Forefront-Threat-Management-Gateway-TMG-2010.html

The figures above were produced using some sample logs received from customers with similar (but not exactly the same) logging settings. If you have changed to text logging, I’d be very interested to hear the sort of disk savings you are seeing, and I’m sure others would to. So please leave a comment below.

Microsoft TMG Log showing Bytes Sent consistently larger than Bytes Received

This issue has been confirmed by the Microsoft Forefront TMG team, and unfortunately there is no ETA for a fix.

We obviously don’t want our reports showing incorrect usage figures, so we’ve fixed our TMG loader so that it imports the ‘bytesrecvd’ field into the Bytes Sent aggregate, and the ‘bytessent’ field into the Byte Received aggregate.

But what if Microsoft release a fix? What we’ve done is implemented a loader property to allow you to turn off this behavior. This will allow you to import your old logs with the fields reversed, and your new logs with the fields the right way around.

To access the loader property:

• On the import wizard, select the Microsoft FTMG format and click the Properties button on the toolbar
• Select Microsoft FTMG from the drop down list
• Notice the option to ‘Reverse Bytes Sent and Received to compensate for bug in TMG’s logging’. Leave this checked until Microsoft issue a fix.

Microsoft Forefront TMG Loader Option to Reverse Bytes Sent and Received

This fix is available in Vantage build 2.2.0.48 (and above) which has been released as an auto update. So simply select Tools | Check for updates to ensure you have this fix.

So let’s use WebSpy Vantage to drill into that Anonymous user and find out what is going on.

One way to do this is to run an Ad-hoc analysis on the Summaries screen and drilldown into the Anonymous user to view all the information about that user. However, TMG and ISA tend to log a lot of information that may not be relevant to this particular investigation, so I’ve created some report templates (one for ISA and one for TMG) and a set of Aliases that pull out some relevant information.

Download our Anonymous Traffic Investigation Report

If you’re running WebSpy Vantage download the Anonymous Traffic Report Templates & Aliases

Then open the .Templates file on the Reports tab, and the .Aliases file on the Aliases tab. Once you have both files opened, go to the Reports tab and click either the ‘Anonymous Traffic Investigation (ISA)’ or the ‘Anonymous Traffic Investigation (TMG)’ report. Then click the ‘Generate report’ link and run the report template on your ISA or TMG storage.

The report gives you the ability to drill into the Allowed, Denied and Failed traffic to see a list of the unauthenticated IPs, Sites, Rules responsible for blocking or allowing the traffic, unauthenticated Applications and Result Codes.

Main causes of anonymous traffic

What you will probably find is that most of the Anonymous traffic is being denied by your TMG or ISA firewall. When a client first requests a web page, the proxy will challenge the client for authentication. These events are often logged with the result code 12209 meaning ‘authorization is required to fulfill the request’. These requests are therefore denied by the proxy until the client’s credentials are authenticated.

Have a look at the amount of traffic being denied and then checkout the Result Codes associated with the denied traffic. Chances are you’ll see ‘proxy authentication required’ appear predominantly.

If you also look at the Applications section you may also find that Windows Updates are sailing through your TMG or ISA firewall unauthenticated.

Filter out unauthenticated traffic from Reports

The most logical next step is to filter out the information you do not want in your reports. You’ll probably still want to include Windows Update traffic in your reports, but you’re probably not so interested in the ‘proxy authentication required’ information. So let’s filter that out.

To do this:

1. Go to the Reports tab and select the report you want to filter (such as your Organization report)
2. Click ‘Edit Template’, then click ‘Template Properties’.
3. In the filter section at the bottom of the dialog, click Add | Field value filter.
4. Select the ‘Result Code’ summary and select the Status Code Names (ISA-FTMG) alias.
5. On the toolbar, search for Authorization, and check the following two items:
   • The server requires authorization to fulfill the request. Access to the Web Proxy filter is denied.
   • The server requires authorization to fulfill the request. Access to the Web server is denied. Contact the server administrator.
6. Ensure the ‘Exclude’ radio button is selected and click OK.

If you decide that you don’t care about seeing ANY unauthenticated traffic in your reports, you can always simply filter out the Anonymous user from your reports.

To do this:

1. Go to the Reports tab and select the report you want to filter (such as your Organization report)
2. Click ‘Edit Template’, then click ‘Template Properties’.
3. In the filter section at the bottom of the dialog, click Add | Field value filter.
4. Select the ‘Username’ summary.
5. On the toolbar, click Add and type ‘anonymous’. Click OK.
6. Ensure the Exclude radio button is selected and click OK.

Hopefully this article improves your understanding of the ‘anonymous’ user, and gives you some actions to take for your specific reporting situation.

If you have any questions, please leave a comment below.

• How to import your log files and explore the information recorded by IronPort using the Summaries screen
• How to open the customized IronPort Report Templates and Aliases
• How to generate reports
• How to import your organizational structure and report on departments
• How to setup the Web Module and publish reports

PART 1: Importing log files & exploring your IronPort summaries

Once you have exported your IronPort access logs (see http://www.webspy.com.au/vendors/ironport/howto.aspx#ftp), this video takes you through importing your logs into WebSpy Vantage and analyzing data on the Summaries screen.

PART 2: Opening the customized IronPort Templates & Aliases, and running reports

This video takes you through opening the IronPort-specific report templates and aliases and generating a report that provides an overview of your organization’s Internet usage.

PART 3: Importing your Organization structure & generating department reports

This video shows you how to import your organizational structure into WebSpy Vantage from a directory server (such as Active Directory) using LDAP, and then generating a report that contains information on your newly imported departments.

PART 4: Using the Web Module.

This video takes you through configuring and using the WebSpy Vantage Web Module. Specifically, it takes you through the following tasks:

• Configuring the Web Module for Windows Authentication
• Adding a Web Module to Vantage
• Publishing reports to the Web Module
• Adding permissions for a user
• Synchronizing the Web Module
• Using the Dynamic Reports tab

PART 5: A quick word about tasks & conclusion

This video summarizes the actions taken in the previous four videos and also briefly discusses how to automate the reporting processing using scheduled tasks.

I hope this helps! Let me know if you have any questions by leaving a comment below.

The log databases are stored in an SQL Express instance named MSFW. By default these databases cannot be accessed by a remote computer. I’d first like to say that we recommend changing TMG’s logging to W3C text files, as these logs are about 5-6 times faster to import, and you don’t need to worry about the steps below.

But if you need to stick with the SQL Express logging, here are the basic steps to enable access to the logs from a remote computer:

Enable TCP access to the MSFW instance

To do this:

1. Log into your Forefront TMG server using administrator credentials.
2. Select Start | All Programs | Microsoft SQL Server 2008 | Configuration Tools | SQL Server Configuration Manager.
3. Expand SQL Server Network Configuration and select Protocols for MSFW
4. Right-click TCP/IP and select Enable
5. Click OK on the Warning dialog informing you that “changes will not take effect until the service is stopped and restarted.”

Enabling TCP/IP on the MSFW instance

Set the listening Port on the MSFW instance

Once TCP/IP is enabled on the MSFW instance, you need to set it to listen on port 1433

1. Select Protocols for MSFW under SQL Server Network Configuration
2. Right-click TCP/IP and select Properties.
3. Click the IP Addresses tab and scroll to the IPAll section at the bottom of the list.
4. Change the TCP Port to 1433 and ensure nothing is entered in TCP Dynamic Ports (Delete the ‘0′ value  if present). Click OK and click OK on the Warning dialog.

Setting the Port on the MSFW instance

Change the listening port on the ISARS instance

The ISARS SQL instance also listens on port 1433 and this can cause connection issues. Change this instance to use port 1434:

1. Still in SQL Server Configuration Manager, select Protocols for ISARS under SQL Server Network Configuration
2. Right-click TCP/IP and select Properties.
3. Click the IP Addresses tab and scroll to the IPAll section at the bottom of the list.
4. Change the TCP Port to 1434 and ensure nothing is entered in TCP Dynamic Ports. Click OK and click OK on the Warning dialog.

Changing the port on the ISARS instance

Restart the Services

For the above changes to take effect, you need to restart the SQL Server (ISARS) and then the SQL Server (MSFW) services in that order.

1. Go to Start | Administrative Tools | Services
2. Right-click the SQL Server (ISARS) service and select Restart.
3. Right-click the SQL Server (MSFW) service and select Restart.

Test the connection from the WebSpy machine

You should now be able to connect to the MSFW databases from a remote computer. To test the connection, we recommend that you install SQL Management Studio on the machine running WebSpy and try to connect to <TMGservername>\MSFW, 1433 (replace <TMGservername> with your actual server name or IP address). For example TMGServer\MSFW, 1433 or 192.168.0.10\MSFW, 1433.

As long as you are logged into Windows with a user account that is a local administrator on the TMG server, you should be able to connect without issue.

Importing the TMG Log files into WebSpy Vantage

Once you have established a connection, you can import your logs using WebSpy Vantage like so:

Create a new Storage

Select Database Connection

Select the Microsoft FTMG Loader

Click Add

Enter TMGServer\MSFW and port 1433

Successfully Imported WebProxy Logs

The screenshots above also illustrate using a database mask of *WEB* to only import the WebProxy logs. If you only want to import the Firewall logs, set the database mask to *FWS*. If you want to import both the WebProxy and Firewall logs, leave the database and table masks set to *.

Now that you have your log files imported, you can run a quick ad-hoc analysis on the Summaries screen or generate any of Vantage’s default web of firewall reports. M

Make sure you also download our Forefront TMG specific Aliases and report template. For more information, see our Forefront TMG How To page.

If you have any questions or encounter any hurdles, please leave a comment below.

When imported into WebSpy Vantage, the result is shown in a new summary called ‘Group’ which you can add to your reports.

We also added support for the custom fields Bytes Sent and Bytes Received. Due to the absence of a header in the IronPort access log, Bytes Received and Bytes Sent fields must both be present to be detected, and the Received field must precede the Sent field.

We also added support for the custom fields Request Body Size and Response Body Size. These fields can be included in your access log by adding %q (Request body size) and %b (Response body size) in the ‘Custom Fields’ edit box. Due to the absence of a header in the IronPort access log, Request Body Size and Response Body Size fields must both be present to be detected, and the Request field must precede the Response field.

We’ve also noticed that the values in the Bytes Sent and Bytes Received fields do not necessarily add up to the value logged for ‘Size’. We’re discussing this issue with our friends at IronPort and we will hopefully post a solution or explanation soon..
The information we first received about these fields indicated they represented Bytes Sent and Bytes Received. This is the way they are represented in the builds below (2.2.0.29). We will release a new build soon, with the field names changed to Request body size and Response body size. Body size is different to bytes sent/received as it does not include bytes from packet headers etc.

We’re yet to issue an automatic update for the Vantage applications, so in the mean time you can download the latest builds here:

Vantage Ultimate:
ftp://ftp.webspy.com/webspy/Builds/VantageUltimate2.2.0.29.zip

Vantage Web Module:
ftp://ftp.webspy.com/webspy/Builds/VantageWebModule2.2.0.8.exe

Vantage Giga:
ftp://ftp.webspy.com/webspy/Builds/VantageGiga2.2.0.29.zip

Vantage Premium:
ftp://ftp.webspy.com/webspy/Builds/VantagePremium2.2.0.29.zip

To apply the Vantage update, close Vantage and extract the downloaded file into Vantage’s installation folder (Usually c:\Program Files\WebSpy\Vantage <flavour> 2.2). Overwrite the existing files.

To apply the Web Module update, uninstall the Vantage Web Module from Add/Remove Programs (Programs and Features in Windows 7/Server 2008), then run the downloaded exe file, making sure you specify the same server, virtual directory and data location that your Web Module was previously using.

We will be releasing this as a public auto-update soon. Let us know if you have any issues.

Here’s a quick video outlining some of the differences between TMGs Reporting, and what can be achieved using WebSpy Vantage. The video does not illustrate all the limitations outlined below, so please read on.

Whats is in the Forefront TMG report?

The default TMG report contains the following sections

• Summary
• Web Usage
• Application Usage
• Traffic and Utilization
• Security
• Malware Protection
• URL Filtering
• Network Inspection System

Each section contains overviews such as ‘Top users’ and ‘Top Sites’.

If your reporting requirements can be satisfied with these overviews – that’s great! Unfortunately, when you start thinking about what system administrators and other people in your organization actually need to make informed decisions, this report is quite limiting.

The 8 Limitations of Microsoft Forefront TMG’s Reporting

Here is what I consider to be the 8 main limitations of Microsoft Forefront TMG’s reporting functionality.

1. No Drilldowns

Want to see the sites that the top 5 users accessed? Want to see the users that downloaded the most traffic from youtube? These are fairly standard reporting requirements that simply cannot be achieved using the inbuilt TMG reporting.

WebSpy Vantage lets you either interactively drilldown into a user or site, or produce a regular report that includes further details about what your top users have actually been up to.

2. No Filtering

When you generate a report in TMG, you can only filter the report by a date range. There is no way to filter out anonymous (unauthenticated) traffic or exclude traffic coming from advertising servers (such as doubleclick and 2mdn.net) that tend to dominate most of the top 10 sites.

This can easily be achieved using WebSpy’s software. Check out my video on how to remove clutter from your web reports.

3. No Customization

Customization of each overview in the TMG report is limited to the number of items to show (e.g. top 5 or top 50 users), and the sort order (Incoming Bytes, Outgoing Bytes, Requests and Total Bytes).

What about the time a user spent browsing the web, or the number of users that visited a specific site? There is no way to add custom columns such as total browsing time, average session time, or number of users/sites/IPs to the report tables.

Or say you simply want to change your top users chart from a bar to pie to easily see the percentage used. Nope sorry!

If you do make one of the two available customizations in a TMG report, you then get the annoying Apply / Discard message to save changes to the configuration database.

All of these customizations can be achieved using WebSpy Vantage, and it doesn’t touch your TMG server to apply a change to a report.

4. Limited Report Distribution

When you generate a report, you get the option to email it to a specific email address. What if you would like to create a report for every department, and then email it to the managers of each department? Or better yet, host the report on a secure web server where department managers can log in and view their reports?

WebSpy Vantage Ultimate comes with a secure ‘Web Module’ specifically for this purpose and managers still receive a link to the report via email.

5. Cluttered ‘Top Sites’ List

The ‘Top sites’ list can become particularly cluttered due to the inclusion of sub-domains. I don’t want to mentally add up the size values from farm1.static.flickr.com, farm2.static.flickr.com, and farm3.static.flicr.com – I just want to know how much was downloaded from flickr.com.

This is compounded by the inability to exclude sites that are merely placing advertising banners on the actual sites users are visiting (as mentioned in the ‘No Filtering’ limitation above).

WebSpy Vantage breaks URLs down into separate components and lets you analyze each part separately. Look at the Site Domains summary to remove sub-domains and see only flickr.com. Or perhaps you want to see the keywords a user entered into search engines like Google? Or perhaps the top pages accessed within a website? No problem. Just include the Site Keywords or Site Resource summaries in your Vantage reports.

6. No Grouping or Aliasing

There is no way to group users into departments or locations, or IP addresses into subnets, or extensions such as .html, .pdf or .exe into file types. The ability to group and represent raw log data in more meaningful ways, as offered by WebSpy Vantage, can increase the value of a report tremendously.

7. No Productivity Assessment

One of the major features introduced in TMG since ISA Server 2006 is the included URL categorization technology.

Although the TMG report gives you an overview of the categories that have been visited, the report does not use this information to display a productivity assessment for your users.

WebSpy Vantage not only provides this assessment, but also the ability to customize the categories that are deemed productive as this can vary wildly depending on the industry and organization.

8. Not browser independent

This is a minor limitation that can be a major annoyance. The report that TMG produces is a HTML report that only displays correctly in Internet Explorer. As Forefront TMG is a Microsoft product, this is not exactly surprising, but still very annoying if IE is not your default browser.

How to get awesome reports from Forefront TMG

If you have had personal experience with any of the above limitations, you’ve probably been hunting for an alternative solution. I strongly recommend checking out the WebSpy Vantage range of products, and if you would like secure report distribution via the ‘Web Module’, Vantage Ultimate is what you are after.

If you agree or disagree with anything in this article, I encourage you to leave your thoughts in the comments.

Cheers!

Scott

TMG or UAG? What is the difference?

TMG is an outgoing proxy that protects your internal users from malware, viruses and the like. TMG generates some great web proxy log files to import into WebSpy Vantage allowing you to monitor where your users are going on the Internet, how much they’re downloading etc.  TMG, unlike ISA, now has deep packet inspection for HTTPS traffic, plus a bunch of other new features.

UAG is an incoming proxy that provides employees, partners and vendors secure remote access to corporate resources such as Outlook Web Access (OWA) and Sharepoint (MOSS). It utilizes the TMG engine, but this is mainly just to protect the UAG server (more on this topic here http://technet.microsoft.com/en-us/library/ee522953.aspx).

TMG can also publish your OWA and MOSS sites, but this is no longer recommended by Microsoft. They recommend using a dedicated UAG server to perform this function.

Upgrading to TMG

If you’re thinking about migrating your ISA server (2004 or 2006) to TMG, you may like to check out this migration guidance video with Mohit Saxena (Senior Technical Lead) and Jim Harrison (Program Manager). http://edge.technet.com/Media/ISA-to-TMG-Migration-Guidance/

Microsoft Forefront TMG Migration Video

Reporting on TMG

If you’re using TMG at the moment, we invite you to analyze your web proxy and/or firewall logs using WebSpy Vantage and tell us what you think!  Download your copy of WebSpy Vantage here, and import your logs using the Microsoft FTMG format:

Microsoft Forefront Threat Management Gateway

Most frequently suggested bypass methods includes the use of public proxies, circumventors and http tunneling. I don’t wish to go into details on any of these methods as their use is NOT recommended. However, it does prove a point: The main reasons organizations block certain websites is to prevent security risks and unproductive internet usage. Although, it is an indisputable fact that employees’ use of virus ridden public proxies, and other elaborate methods, to overcome blocking efforts can in fact increase security risks and unproductive behavior – making matters even worse.

Obviously all employees do not take these measures, but isn’t it enough that some do? Yes, the same high risk and time consuming bypassing “techniques” could be used when trying to stay anonymous from internet monitoring software. However, there are two main differences:

1. Using internet monitoring software reduces the need to block. Employees will be able to access the legitimate sites that often end up blocked thanks to a “block worthy” word in a corporate blog, or something of similar virtuousness. Not blocking means less time and effort spent trying to bypass blocking solution. After all, my mailbox is not full of alerts on how to bypass internet monitoring software.
2. Using internet monitoring software will allow employers to detect who is up to no good trying to bypass blocking rules or browse anonymously. For example, if an employee continuously use public proxies or tunneling, an internet monitoring solution (or at least a good internet monitoring solution) can assist the employer in tracking down the offender. (Please have a look at “How to Improve Public Proxy Management” blog for more info.)

This blog simply adds to the convincing case against organizations’ excessive use of blocking and filtering solutions. Porn sites, known malicious virus and phishing sites – by all means, block the living daylight out of them. But as for the rest, as for news site, online shopping sites, social networking and general interest sites – Don’t block, monitor.

I want to avoid repeating myself so please have a look at previous blog for the full story on “The Cost of Blocking Employee Internet Usage”

If you’re considering upgrading your ISA Server to TMG, this means that you can start your deployment using the Release Candidate, and simply switch it to a licensed version with no additional configuration changes once the full release is available. At least, that is what Vladimir Holostov (Lead Program Manager, Release Manager for Forefront TMG 2010) states on the Forefront TMG (ISA Server) Product Team Blog:

“The final product will be released later this year and you can expect it to behave exactly like the Release Candidate. You can install Forefront TMG 2010 RC today and upgrade to a licensed version once available without changing the configuration of your deployment.”

To offer some peace of mind for organizations considering the deployment, Vladimir also mentions that “Forefront TMG 2010 RC is deployed at three major Microsoft sites located around the world in Haifa, Bellevue and Redmond. More than 20,000 employees are already protected by TMG and these deployments have already accumulated more than 5,000 hours of runtime, performing extremely well under heavy load”.

No major features have been added to the Release Candidate since Beta 3, however there have been improvements geared around tightening up security, reliability and performance and telemetry. For more information about the release candidate, please visit the
Forefront TMG (ISA Server) Product Team Blog.

You can also download the release candidate here

I mentioned in my last blog posting that WebSpy has introduced support for reporting on Microsoft Forefront TMG log formats in the Vantage product range. To try it out, please make sure you have installed Vantage 2.2 (any flavour – Premium, Giga or Ultimate), and then select Tools | Check for updates to download build 2.2.0.10 or above. You can then import your TMG log files by selecting the Microsoft FTMG loader in the import wizard.

Importing Microsoft Forefront Threat Management Gateway Log Files

We’re very interested to hear your thoughts on the reporting functionality, so please go ahead and give it a go!

We’ve got a great deal at the moment where you get 20% off Live and Sentinel if you purchase them online with Analyzer Standard.

If you convert your logs to text using this script, they won’t import into WebSpy Vantage or Analyzer due to an extra line break in the header of the file (after #fields:).

We’ve therefore created a modified version of the script that creates compatible log files for WebSpy software.

Download the modified MSDEToText script:
MSDEToText.zip -26 KB

Also make sure the file names of your output log files contain the word WEB (for Web Proxy logs) or FWS (for Firewall Logs) as Analyzer and Vantage use these strings to automatically detect the type of ISA log file.

Happy converting!

Watching a single video on YouTube will probably generate a list of about three to five sites such as lax-v41.lax.youtube.com, www.youtube.com, img.youtube.com, and so on. Your list of top sites also probably contains hits to ad servers and tracking servers, such as doubleclick.net, google-analytics.com and imrworldwide.com. All this clutter gets in the way of determining what sites were ‘intentionally’ visited.

Fortunately there are a few simple steps you can take to exclude this information from your reports. Watching is much easier than reading, so I thought I’d create a video demo to walk you through the process.

By the way, this is the first video demo of what I hope will be many more to come. I created it using TechSmith’s Camtasia Studio which is by far the best screen recording software I’ve used. All the zooming you see throughout the demonstration is completely auto-magical! It’s a brilliant piece of software that has saved me hours of time. Props to the guys at TechSmith! The one pitfall of Camtasia is that it seems to make me sound like a geek with a raw Aussie accent… I hope they fix that in the next version.

Anyway, I hope you find this useful.

To some it might be obvious, but plenty of search queries used by visitors coming to our site contains phrases such as; “Why monitor internet usage important” and “Why analyze log files”.

Majority of benefits directly relate to the network device being monitored so I will structured the business benefits based on this.

Web Proxy Servers

Web proxy servers maintain log files listing every request, from outgoing traffic, made to the proxy server. By monitoring and reporting on log files from web proxy servers you will be able to identify aspects such as: who is accessing external sites, what sites are being accessed, when the sites were accessed, how much time was spent on the sites, how the user navigates through the sites, what page or search phrase referred the user to the sites, and the type and size of data downloaded from the sites. Use this information to:

• Maximize Employee Productivity
  Identify employees who excessively use corporate Internet resources for recreational purposes. Effectively publishing and communicating Internet usage policies and making employees aware of monitoring activities, and corresponding breach consequences, will assist in reducing personal Internet use.
• Ensure Policy Compliance
  Identify misuse and ensure compliance with acceptable Internet usage policies by monitoring which sites are being viewed, for how long, what is being downloaded and by whom.
• Ensure Legal Compliance
  Mitigate risk of costly liability and litigation issues by ensuring compliance with acts and regulations relating to Internet usage.
• Reduce & Verify Bandwidth costs
  Assess bandwidth usage and identify excessive downloading from particular websites, of specific files, and by which employee. Verify accuracy of Internet Service Provider’s charges.
• Understand and Reward Acceptable usage
  Please read my previous blog covering this area.

Web Servers

Web servers maintain log files listing every request from incoming traffic made to the server. Reporting on these log files can tell you: who is accessing the internal site, what pages are being accessed, when the pages were accessed, how much time was spent on each page, how visitors navigated through the pages, what site or search phrase referred the visitor to the site, and the type and size of data downloaded from the site. Use this information to:

• Verify Effectiveness of Online Campaigns
  View the most common sites referring traffic to your own website to validate the effectiveness of online marketing initiatives. Display search terms commonly used in search engines referring to your company’s website to optimize the website’s search ranking and maximize bids on the correct search terms for online pay-per-click campaigns. Or why not use the search phrases to inspire a new blog post .
• Optimize Website Performance
  Prioritize web page sequences, improve navigation, improve browser support and reduce link breaks by monitoring incoming website traffic, commonly accessed pages, user agents (browsers) accessing your website, client and server errors.

Email and messaging

Every time an email or messaging server sends or receives information they store log files containing data about the sender, the receiver, timing of delivery or receipt, subject line, size of attachment and, depending on the server, name of attachment and content of message. Use this information to:

• Reduce Bandwidth costs
  Identify emails and messages with large attachments, who sent them, and if they were work related.
• Protect Confidential Information
  Monitor email and instant messaging activity to protect the transmission of confidential organizational information.
• Mitigate Litigation Risks
  Mitigate risk of costly liability and litigation issues by ensuring compliance with acts and regulations in relation to sexual harassments, bullying and discrimination that can arise from improper email and messaging usage.
• Maximize Email Virus Protection
  Analyze log files from email virus scanning software, or devices, to identify source of viruses. Identify who sent the virus, who received it, attachment name and how your virus scanner dealt with it.

Network and security devices

Network devices, such as switches, routers and proxies, and security devices, such as firewalls, anti-virus, spyware and spam applications, store log files containing data about network activity and the external and internal traffic that has been blocked or filtered. Use this information to:

• Improve Network Management
  Investigate traffic between computers, ports or applications to diagnose network problems. Gather information to help decide which protocols to prioritize over others. Better manage network resources and troubleshoot certain events.
• Strengthen Security Controls
  Verify the configuration of a network’s firewall and its control of network traffic. Identify and investigate security breaches, determine the source of email viruses and manage their organizational impact.
• Maximize Effectiveness of Existing Blocking & Filtering Solution
  Review websites that employees have been denied and granted access to in order to validate the effectiveness of existing Internet filtering service.

Event logs

Designed to provide an audit trail of system use, event logging records the actions that occur within the system, such as users logging in, failure of a component to start, or an attempt to print a document.

Every event that occurs across a network can be recorded in an event log file. The list of events that are recorded by default can be modified to reflect the needs of the organization’s system. Use this information to:

• Monitor failed authentication attempts
  Identify users trying to access files and folders they are not authorized to access, or the system failing to provide legitimate user access.
• Prevent data loss and leakage
  Identify the access, modification or printing of confidential files to prevent information leakage or identify the person behind accidental or deliberate data loss.
• Ensure employees adhere to specified work schedules
  Monitor event logs that record when an employee’s computer has been powered on or shut down.

Hopefully this will give readers a better understanding of the benefits involved. Perhaps it can be helpful when explaining to employees / employer why and how your Internet and network resources need to be monitored and reported on.

Well, it’s now been released and can be downloaded from the Microsoft Download Center:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=e05aecbc-d0eb-4e0f-a5db-8f236995bccd

One of the major improvements in Beta 3 is URL filtering which leverages Microsoft Reputation Services (MRS). With regards to this, Microsoft says:

“At the time of this release, the MRS database content is being populated and updated continuously as part of the initial beta service offering. As this process continues, URL filtering categorization accuracy and comprehensiveness will increase. A telemetry package designed for improving the quality of URL filtering database and collecting your feedback is planned to be released soon. Please check back for updates in August.”

Support for importing Microsoft TMG log files into your favorite WebSpy product is coming soon so stay tuned! Subscribe to the WebSpy blog RSS feed or follow us on twitter if you want to be notified as soon as it’s available.

According to the latest blog from TMG’s Product Unit Manager, David B. Cross, Beta 3 will be released in the next couple of weeks. As for the full release, David says that they are still on track for Q4 this calendar year.

Beta 3 will introduce URL filtering that is ‘fully integrated’ with TMG’s web policy rules, and also utilizes Microsoft Reputation Services.

Microsoft are also introducing Intrusion Prevention and Detection (IPS/IDS) capabilities in TMG. These systems will utilize a technology they’re calling Network Inspection System (NIS) that detects attacks using signatures of known vulnerabilities, downloaded from the Microsoft Malware Protection Center. For more information on NIS see http://blogs.technet.com/isablog/archive/2009/04/12/exercising-nis-with-test-signature.aspx

If you’re currently using ISA 2004 or 2006, upgrading to TMG will consist of exporting rules and settings from ISA, then importing them into a clean installation of TMG. TMG will also only run on Windows Server 2008.

Improving the on-box reporting has not been a focus for the TMG development team, so analyzing TMG’s web proxy and firewall logs is still the best way to go for in depth reporting.

If you’re interested in reporting on your TMG log files stay tuned! We’re currently implementing support for the SQL Express, W3C and Native text logs. WebSpy Vantage is likely to be the first application to include the feature, with Analyzer and Live soon to follow.

All going well, you can expect to see TMG support in your favourite WebSpy app within the next month or so. If you want to be notified once we’ve added support, just leave a comment below.

Cheers!
Scott.

The fix can be applied to WebSpy Analyzer Giga 2.3, Analyzer Premium 4.3 or Analyzer Standard 4.3. If you’re not running the latest version, download it now!

You can download the new loader build that we created today at either of these locations:
USA West Coast (FTP)
USA East Coast (FTP)

Then extract the zip file into Analyzer’s installation folder (usually C:\Program Files\WebSpy\Analyzer flavour 4.3\) and overwrite the existing file.

Then go to the storages screen and select your Sophos storage(s) and click ‘Reload all hits’. This will re-import your log files using the modified loader and will populated the ‘Blocked’ summary appropriately. To check it out, go to the Summaries screen and run a Full Analysis. Then go to the ‘Blocked’ summary and you should see two items – ‘Blocked’ and ‘Not Blocked’. Drilldown into whichever one you care about to analyze the sites, users, files, browsing times, size downloaded etc. Go nuts!

You can also filter out blocked hits (or Not Blocked hits) from your reports. On the Reports Screen, click Generate a new report and go through the report wizard with this filter (this example shows filtering out blocked hits).

Analyzer Report Wizard - Select Custom Filters

Analyzer Report Wizard - Selecting the 'Blocked' Summary as a Filter

Analyzer Report Wizard - Adding the items that you want to filter

Analyzer Report Wizard - final filter to exclude 'Blocked' hits

Then proceed through the report wizard to generate your report. This filter can be applied to any report as well as analyses on the Summaries screen (using the same options in the Analysis Wizard).

Happy analyzing!