The Australian Attorney-General’s Department recently confirmed ongoing discussions about implementing a data retention regime in Australia requiring ISPs to hold customers web browsing history and private emails and make both available on request from government agencies. Industry insiders said the regime being considered by the Australian Government could see data held for up to ten years, much longer than EU Directive time of 24 months.

While reading this, looming in the back of mind is the labour party’s proposed mandatory internet filtering policy. The proposed policy was originally aimed at keeping children safe online but in reality it involves blacklisting and blocking a plethora of online material deemed inappropriate by the government. Opponents of the policy don’t dispute the worth of providing tools to help parents protect their children, but take issue with the expense, side-effects and technical issues of this scheme. Find out more from Electronic Frontiers Australia’s filtering fact sheet.

Although using a slightly different approach, the ultimate goals behind the two regimes is making the internet a safer place, circumventing crimes and facilitating investigation of suspected criminals. But is snooping on individuals’ private browsing and controlling access to material really the way to go? Regardless of the questionable effectiveness and obvious drawbacks, such as costs, there are some major concerns regarding privacy and freedom. Do we really want the government to decide what we can view online and have the ability to access our own personal browsing history? I am aware the intentions are good but it does send chills down my spine.

In relation to the data retention scheme, Electronic Frontier Australia (EFA) chair Colin Jacobs said, “At some point data retention laws can be reasonable, but highly-personal information such as browsing history is a step too far”. Jacobs added, “You can’t treat everybody like a criminal. That would be like tapping people’s phones before they are suspected of doing any crime.”

Jacobs does raise valid concerns. Where do you draw the line? Is accepting government access to everyone’s browsing history a precursor to tapping people’s phones? Setting up surveillance cameras in everyone’s home? Thought police?

Extract from Orwell’s 1984:
“There was of course no way of knowing whether you were being watched at any given moment. How often, or on what system, the Thought Police plugged in on any individual wire was guesswork. It was even conceivable that they watched everybody all the time. But at any rate they could plug in your wire whenever they wanted to. You had to live—did live, from habit that became instinct—in the assumption that every sound you made was overheard, and, except in darkness, every movement scrutinized.”

Alright, I know we’re far from the dystopia described in Orwell’s 1984 but I can’t help drawing parallels between the personal privacy lost to the state. Orwell once explained that the scene of the book (1984) is laid in Britain in order to emphasize that the English-speaking races are not innately better than anyone else, and that totalitarianism, if not fought against, could triumph anywhere.

Side Note

WebSpy is pro-internet access and provide businesses, government departments and educational organizations an alternative to blocking and filtering software. We emphasizes that organizational internet usage should be managed using an honest and open monitoring approach where acceptable internet usage policies are clearly communicated to employees and students.

Employers need to ensure their internet resources are used in a productive, secure and legal manor. Private filtering and monitoring by a government is a completely different kettle of fish and we have deliberately refrained from public comments on Australia’s proposed filtering policy. However, with the new discussions on government access to private web browsing and email records we felt the need to at least raise a few concerns and if nothing else point out the difference between private and organizational internet reporting.

Whilst we do see benefits in families and individuals monitoring their own internet usage and bandwidth cost, as a company we do NOT support government legislation allowing national filtering or access to private browsing records.

What are your thoughts?

One of Forefront TMG’s major strengths is obviously its URL categorization and filtering abilities. Since TMG now takes care of the threat management aspects, clients converting from other solutions, such as ISA Server, no longer need a third party filtering solution and will most likely save a considerable amount of money.

However, the reporting functionality included in Forefront TMG are not much different from ISA Server 2006, i.e. very little flexibility or customization for those with reporting requirements beyond general overviews cluttered with irrelevant information.

We’ve blogged a lot about TMG reporting in the past and have now uploaded new and dedicated WebSpy Vantage and Microsoft Forefront TMG pages outlining:

• 10 Reasons to Use WebSpy Vantage to Report on Forefront TMG
• How to:
• Set-up TMG Logging for WebSpy
• Import TMG Logs into WebSpy Vantage
• Forefront TMG Report Templates and Aliases (created to make your life a lot easier)
• Run Reports
• Analyze and Drilldown into Data

Have a look at WebSpy Vantage and Microsoft Forefront TMG.

Hopefully it can assist you in your quest for sophisticated Forefront TMG reporting.

Here’s a quick video outlining some of the differences between TMGs Reporting, and what can be achieved using WebSpy Vantage. The video does not illustrate all the limitations outlined below, so please read on.

Whats is in the Forefront TMG report?

The default TMG report contains the following sections

• Summary
• Web Usage
• Application Usage
• Traffic and Utilization
• Security
• Malware Protection
• URL Filtering
• Network Inspection System

Each section contains overviews such as ‘Top users’ and ‘Top Sites’.

If your reporting requirements can be satisfied with these overviews – that’s great! Unfortunately, when you start thinking about what system administrators and other people in your organization actually need to make informed decisions, this report is quite limiting.

The 8 Limitations of Microsoft Forefront TMG’s Reporting

Here is what I consider to be the 8 main limitations of Microsoft Forefront TMG’s reporting functionality.

1. No Drilldowns

Want to see the sites that the top 5 users accessed? Want to see the users that downloaded the most traffic from youtube? These are fairly standard reporting requirements that simply cannot be achieved using the inbuilt TMG reporting.

WebSpy Vantage lets you either interactively drilldown into a user or site, or produce a regular report that includes further details about what your top users have actually been up to.

2. No Filtering

When you generate a report in TMG, you can only filter the report by a date range. There is no way to filter out anonymous (unauthenticated) traffic or exclude traffic coming from advertising servers (such as doubleclick and 2mdn.net) that tend to dominate most of the top 10 sites.

This can easily be achieved using WebSpy’s software. Check out my video on how to remove clutter from your web reports.

3. No Customization

Customization of each overview in the TMG report is limited to the number of items to show (e.g. top 5 or top 50 users), and the sort order (Incoming Bytes, Outgoing Bytes, Requests and Total Bytes).

What about the time a user spent browsing the web, or the number of users that visited a specific site? There is no way to add custom columns such as total browsing time, average session time, or number of users/sites/IPs to the report tables.

Or say you simply want to change your top users chart from a bar to pie to easily see the percentage used. Nope sorry!

If you do make one of the two available customizations in a TMG report, you then get the annoying Apply / Discard message to save changes to the configuration database.

All of these customizations can be achieved using WebSpy Vantage, and it doesn’t touch your TMG server to apply a change to a report.

4. Limited Report Distribution

When you generate a report, you get the option to email it to a specific email address. What if you would like to create a report for every department, and then email it to the managers of each department? Or better yet, host the report on a secure web server where department managers can log in and view their reports?

WebSpy Vantage Ultimate comes with a secure ‘Web Module’ specifically for this purpose and managers still receive a link to the report via email.

5. Cluttered ‘Top Sites’ List

The ‘Top sites’ list can become particularly cluttered due to the inclusion of sub-domains. I don’t want to mentally add up the size values from farm1.static.flickr.com, farm2.static.flickr.com, and farm3.static.flicr.com – I just want to know how much was downloaded from flickr.com.

This is compounded by the inability to exclude sites that are merely placing advertising banners on the actual sites users are visiting (as mentioned in the ‘No Filtering’ limitation above).

WebSpy Vantage breaks URLs down into separate components and lets you analyze each part separately. Look at the Site Domains summary to remove sub-domains and see only flickr.com. Or perhaps you want to see the keywords a user entered into search engines like Google? Or perhaps the top pages accessed within a website? No problem. Just include the Site Keywords or Site Resource summaries in your Vantage reports.

6. No Grouping or Aliasing

There is no way to group users into departments or locations, or IP addresses into subnets, or extensions such as .html, .pdf or .exe into file types. The ability to group and represent raw log data in more meaningful ways, as offered by WebSpy Vantage, can increase the value of a report tremendously.

7. No Productivity Assessment

One of the major features introduced in TMG since ISA Server 2006 is the included URL categorization technology.

Although the TMG report gives you an overview of the categories that have been visited, the report does not use this information to display a productivity assessment for your users.

WebSpy Vantage not only provides this assessment, but also the ability to customize the categories that are deemed productive as this can vary wildly depending on the industry and organization.

8. Not browser independent

This is a minor limitation that can be a major annoyance. The report that TMG produces is a HTML report that only displays correctly in Internet Explorer. As Forefront TMG is a Microsoft product, this is not exactly surprising, but still very annoying if IE is not your default browser.

How to get awesome reports from Forefront TMG

If you have had personal experience with any of the above limitations, you’ve probably been hunting for an alternative solution. I strongly recommend checking out the WebSpy Vantage range of products, and if you would like secure report distribution via the ‘Web Module’, Vantage Ultimate is what you are after.

If you agree or disagree with anything in this article, I encourage you to leave your thoughts in the comments.

Cheers!

Scott

If you’re using WebSpy Vantage, you are probably interested in filtering your log file imports by date (only import files from the month of June for example). The obvious way to do this is to specify a date filter using the filters page in the Input Wizard. The problem is Vantage will still check every record in every log file being imported to see if it matches the date filter. If you have months or years worth of logs in the folder being imported, that’s a lot of data that Vantage has to pointlessly sift through.

The good news is, if your log files contain the date in their file name, then you can use file masks to instruct Vantage to never touch these unwanted files.

A bit about file masks…

You can specify file masks such as *, *.log, *.gzip, *WEB*.w3c, etc to import logs with specific file extensions, or with specific strings in the file name (such as WEB or FWS to import only Microsoft ISA Web Proxy or Firewall logs respectively).

But if your log file contains the date in the file name, you can also use date modifiers in the file mask to select logs from a particular month, date or year.

Say you have log files that look like this:

• 20090801.log

• 20090802.log

• 20090803.log

and so on..

You can create a simple file mask to only import log files from the month of August very easily using 200908*, or 200908*.log.

Using date modifiers in file masks

But if you’re using Tasks to automatically create a new storage each month, you don’t want to have to worry about manually changing the file mask to 200909*.log when the first day of the next month rolls around.

So intsead, you can use a date modifier in the file mask that will automatically select the logs for the current month, every time your task runs. For the above example, the file mask looks like this:

• %[yyyyMM]* (you can also use %[yyyyMM]*.log)

When the task runs, %[yyyyMM] will be replaced with actual values from the current date. So if the task runs on the 1st of August 2009, the file mask will become 200909* (or 200909*.log).

Dealing with different date formats

You can also use date modifiers for log files that look like this:

• 2009-Aug-01.log

• 2009-Aug-02.log

• 2009-Aug-03.log

In this case the file mask looks like:

• %[yyyy-MMM]* - notice the three MMM's as opposed to two MM's used previously.

Vantage uses the custom date and time format strings available in the .NET framework, so for more information on whether to use m or M or MMM, please refer to this article http://msdn.microsoft.com/en-us/library/8kb3ddd4.aspx

Importing logs from previous months

If you would like to import logs from a previous month, this can also be done by adding an additional element to the date modifier. For example, to import the previous months logs you can use:

• %[-1m,yyyyMM]*

Notice the -1m meaning ‘minus one month’. You can also use -1d (for minus one day), or -1y (for minus one year).

More examples

Here are some more examples to give you an idea of what is possible using date modifiers.  Assuming the date is 14th of August 2009:

• %[-1y,yyyyMM]*.log will create a file mask of 200808*.log

• %[yyyy-MM-dd]*.log will create a file mask of 2009-08-14*.log

• %[-4d,yyyyMMdd]*.log will create a file mask of 20090810*.log

• %[1-m,-4d,yyyyMMdd]*.log will create a file mask of 20090710*.log

• %[-1y,1-m,-4d,yyyyMMdd]*.log will create a file mask of 20080710*.log

• ISALOG_%[-1m,yyyyMM]*_WEB_*.w3c will create a file mask of  ISALOG_200907*_WEB_*.w3c

• *%[-1m,yyyyMM]* will create a file mask of  *200907*

Adding a file mask

File masks are configured on the Input Selection page of the Input Wizard, when you select Add | Folder.

Adding a File Mask

There is also an option to save the literal date into the file mask when the task is run.  For more information on this option, please see my previous blog about this feature.

Other uses for date modifiers

Date Modifiers are a great way to speed up log file imports, but you can also use them when specifying storage names as well as report names. For example, if you specify a storage name of %[yyyyMM]_storage, this will create storages with the names 200907_storage, 200908_storage and so on. When selecting the storages to report on, you can click the Add button on the storage selection toolbar in the Report Wizard, and specify storages such as %[-1m,yyyyMM]_storage, to report on the previous month’s storage.  For more information, please see Automatic Importing and Reporting using Tasks.

I hope this helps someone out there. Let me know how you go!

Watching a single video on YouTube will probably generate a list of about three to five sites such as lax-v41.lax.youtube.com, www.youtube.com, img.youtube.com, and so on. Your list of top sites also probably contains hits to ad servers and tracking servers, such as doubleclick.net, google-analytics.com and imrworldwide.com. All this clutter gets in the way of determining what sites were ‘intentionally’ visited.

Fortunately there are a few simple steps you can take to exclude this information from your reports. Watching is much easier than reading, so I thought I’d create a video demo to walk you through the process.

By the way, this is the first video demo of what I hope will be many more to come. I created it using TechSmith’s Camtasia Studio which is by far the best screen recording software I’ve used. All the zooming you see throughout the demonstration is completely auto-magical! It’s a brilliant piece of software that has saved me hours of time. Props to the guys at TechSmith! The one pitfall of Camtasia is that it seems to make me sound like a geek with a raw Aussie accent… I hope they fix that in the next version.

Anyway, I hope you find this useful.