WARNING: With storage locking disabled, it is possible to import into a storage while running a report, and doing this may cause storage corruption. It is therefore very important if you decide to enable these features to ensure that a storage is not written to while running reports.

Due to the experimental nature of these features, they can only be enabled by including a config file next to Vantage’s executable. To enable multi-instance capabilities and disable storage locking:

1. Download the following config file:
   WebSpy.Vantage.exe.config
2. Close Vantage
3. Extract downloaded zip file into Vantage’s installation folder (usually c:\Program Files (x86)\WebSpy\Vantage <flavour> 2.2). If you already have a file of the same name in that location, make a backup of it before overwriting it with the  new file.
4. Run Vantage.

You can now run Vantage again to launch another instance of the application.

Be aware of:

Simultaneous reading and writing, and multiple writes

I just want to be very clear that if you run reports while importing, or import into the same storage simultaneously, storage corruption can occur. Storages are not designed to be unlocked for these reasons. The only reason we’ve provided this ability is so that you can READ from the a single storage  simultaneously (i.e. run two or more reports). Reading and writing, and multiple writing is NOT supported, but Vantage will attempt to do it if you ask it to, with undefined behavior.  Check your Tasks configuration and note when any import jobs are likely to occur to avoid running reports at these times.

Configuration Changes

When Vantage closes it writes all of it’s state to a series of files under c:\users\<user profile>\AppData\Roaming\WebSpy\Vantage <flavour> 2.2). When Vantage opens, it loads these files into memory. When  running multiple instances, these instances will be reading and writing the same files. So if you open two instances of Vantage, make a change to a report template in one instance, then close the application, the Vantage.Templates file will be updated. But when you close the second instance of the application, the Vantage.Templates file will be overwritten with a version that doesn’t include the change.

When making configuration changes (templates, tasks, aliases, organization etc), make sure only one instance is running (check Task Manager for the WebSpy.Vantage.exe process).

It’s Experimental!

There may be other undefined behaviors that we are yet unaware of, so we advise running this configuration in a test environment.

We’re providing these feature on an “as-is” basis, meaning we will not be providing technical support for issues that arise as a result. That said, we are certainly interested to hear about any issue to help us improve the feature.

Let us know how you go!

Of note, this release includes full support for Microsoft Exchange 2010 Tracking logs (previously supported with the Exchange 2007 loader, but missing a few fields), as well as JunOS (Juniper) Traffic Logs, IronPort Traffic Monitor Logs and Squid Syslog.

We’ve also included an experimental feature to allow multiple instances of WebSpy Vantage to run on the same operating system. The goal here is to run multiple reports at the same time using multiple instances of the application. To do this, we have also included a second experimental feature to disable storage locking. This allows multiple instances of Vantage to read from the same storage at once. These features can only be enabled by including a config file next to the Vantage’s executable. More on this feature here.

Here’s the full list of changes:

Application Changes

• New: Added suffix option to Import Windows Users wizard in Aliases.
• New: Date modifiers now supports h for hour and n for minute, e.g. %[-2h,yyyyMM - HH].
• New: Added tracing to storage publish task.
• Experimental: Multiple instances of Vantage can now be run simultaneously, by adding the multipleInstance key to the application config file.
• Experimental: Storage locking can be turned off to allow multiple instances of Vantage to run reports on a single storage simultaneously. This is done by adding the storageLocking key to the application config file.
• Fix: Import Organization merge options now appends attributes if keep existing user details is selected, and replaces attributes if update user details from the directory is selected.
• Fix: Import Organization merge no longer replaces user’s passwords.
• Fix: Fixed issue where no results were returned when filtering on time less than one day – such as past n hours.
• Fix: Storages are no longer duplicated in the Import new hits task dialog.
• Fix: Fixed issues where the Site Domain summary included sub-domains for European domains (.fr, .be etc).
• Fix: SQL server inputs now commit correctly if the user edits the input and only changes the port number.

Loader Changes

• New: IronPort Traffic Monitor Logs.
• New: Juniper JunOS Traffic Logs (SRX).
• New: Microsoft Exchange 2010.
• New: Squid Syslog.
• Improved: Astaro Security Gateway: Added support for an additional different syslog header.
• Improved: SonicWall: Split syslog format into Web and Firewall schemas, added support for User field, string-type Category field and split Protocol field.
• Fix: Microsoft FTMG: Changed type of Object Source field in from Int32 to String. Users will need to clear/field select/reload their storages before this change will apply.
• Fix: Astaro Mail Gateway: Improved format detection, fixed negative size issue, and Index out of bounds errors.
• Fix: IronPort WSA: Improved format detection.

How to update

To update your software, simply click Tools | Check for updates. Vantage Ultimate users will also need to update the Web Module in order to use the new loader formats that have been added. To update the Vantage Web Module, right-click the WebSpy system tray icon and select ‘Check for updates’. If you have issues with the Web Module update process, please see: http://www.webspy.com.au/forums/viewtopic.php?f=4&t=29

Let me know if you have any questions or issues!

You can also read through these steps on this page: Analyzing SonicWALL log files with WebSpy.

Creating and Importing SonicWALL log files

Analyzing SonicWALL log files

We intend to make some SonicWALL specific report templates available on our SonicWALL how to page soon.

Until then, feel free to create your own templates, or modify our existing web reports to include the extra goodies contained in the SonicWALL logs.

TIP: To modify an existing web report, right-click the report and choose ‘Duplicate template’. Then choose the “SonicWall Web” schema. You’ll then have a report template that you can modify to include all the SonicWALL summaries, such as Categories, and Source and Destination Interface.

If you need some assistance getting the report(s) you need, feel free to contact me, or support@webspy.com.

Take a look at our dedicated Astaro pages to get an idea of what can be achieved when analyzing Astaro Web Gateway log files with WebSpy Vantage.

I’ve created some quick videos to show you how to enable the correct logging options on the Astaro Security Gateway appliance, how to import these log files into Vantage, and analyze the data on the Summaries screen.

Configure Logging

The best way to configure logging is to setup a 3rd party syslog server (such as Kiwi Syslog) on a machine in your network, then configure the Astaro Security Gateway to send syslog messages to that server. The syslog server then creates log files that can be imported into WebSpy Vantage. This video takes you through that process.

Importing and Analyzing Astaro logs

Once you have successfully configured syslogging on your Astaro Security Gateway, you can import the log files into WebSpy Vantage and analyze activity on the Summaries screen. This video takes you through that process.

We intend to make some Astaro specific report templates available on our Astaro How To page soon.

Until then, feel free to create your own templates, or modify our existing web reports to include the extra goodies contained in the Astaro logs.

TIP: To modify an existing web report, right-click the report and choose ‘Duplicate template’. Then choose the “Astaro Security Gateway – Filter with category” schema. You’ll then have a report template that you can modify to include all the Astaro summaries, such as Actions and Categories.

If you need some assistance getting the report(s) you need, feel free to contact me, or support@webspy.com.

Every organization is different with regards to the volume of logs it creates, but I’ve averaged three data sets submitted to us by customers to produce the following estimates.

You will create roughly 0.9 MB of IronPort WSA access logs per user per day.

Once imported into a WebSpy Storage, the Storage will be about 90% of your original log file size. If you apply NTFS compression to the storage folder, the actual size on disk of the WebSpy Storage will be about 30% of your original log data.

So an organization with 1000 users will produce about 900 MBs of access logs per day. The default WebSpy Storage  will be 810 MB, but with NTFS compression, the size on disk will around 270 MB.

As I said, this is a rough guide based on the average of three sets of sample logs we have in house, so please run your own tests and if you can, let us know your values in the comments below.

But there is another advantage to switching to text. They take up considerably less disk space. Here are some figures:

Number of Records in 235 MBs of log data:

235 MB of TMG’s W3C text logs contains 326,824 records. An SQL Express database of the same size (mdf and ldf files) contains only 40,308 records. In other words, w3C text logs can store over 8 times as much data in the same amount of disk space.

A rule of thumb:

By switching to W3C text logs, the disk space taken by your log files will be roughly 12% of the SQL Express or MSDE log files. This can be reduced even further by compressing your text logs.

• MSDE/SQL logs: budget for 5 KB per record
• W3C Text logs: budget for 0.71 KB per record

How many records your ISA or TMG server creates per day will depend on the number of users in your organization and how much traffic they generate, but about 16,000 records per user is a reasonable estimate.

A real world example

If you are hitting 500 GB of SQL Express/ MSDE logs per month (about 86,128,205 records), simply switching to W3C text logs will reduce this down to 61 GB.

Once imported into a WebSpy Storage, the storage size would be roughly 53 GB (87% of the original W3C text logs).

With NTFS compression applied to the Storage folder, the WebSpy Storage would be roughly 13.4 GB (22% of the original W3C text logs).

Applying NTFS compression to your WebSpy Storages folder is certainly a good idea. This does not impact performance. If anything, it may improve performance slightly as there is less disk fragmentation within the storage.

Disadvantages and Alternatives

Please be aware that by changing your logging to text, the default reporting functionality within TMG will no longer work. However, the reporting supplied by WebSpy Vantage should more than adequately replace this feature.

If you are still concerned about changing the logging method, you can utilize a script published by Microsoft to convert your SQL Express logs to W3C text.  You can then keep the text logs and set some more stringent data retention policies on the SQL Express logs, such as clearing logs every week. You can download this script as part of the Microsoft Forefront Threat Management Gateway (TMG) 2010 Tools & Software Development Kit.

Additional Resources

• Here’s a great article by Marc Grote at isaserver.org on the pros and cons of the different logging options in ISA and TMG. It also takes you through how to exclude fields to reduce the amount of data being logged:
  http://www.isaserver.org/tutorials/Microsoft-Forefront-TMG-Logging-options-Forefront-TMG.html
• Also take a look at Richard Hicks’ blog regarding MSDE performance with ISA Server 2006:
  http://tmgblog.richardhicks.com/2009/10/31/msde-performance-with-microsoft-isa-server-2006/
• Here’s another article on isaserver.org by Richard Hicks on the logging enhancements in TMG 2010
  http://www.isaserver.org/articles/Logging-Enhancement-Microsoft-Forefront-Threat-Management-Gateway-TMG-2010.html

The figures above were produced using some sample logs received from customers with similar (but not exactly the same) logging settings. If you have changed to text logging, I’d be very interested to hear the sort of disk savings you are seeing, and I’m sure others would to. So please leave a comment below.

What’s New?

Clearswift SECURE Web Gateway W3C

Clearswift have just released the latest version of their SECURE Web Gateway, which includes a transaction log export function. This enables you to send transaction logs in W3C format to an off-box FTP server for analysis. If you are updating to the latest Clearswift SECURE Web Gateway, make sure you update your Vantage software to 2.2.0.55 in order to import your W3C Transaction logs. More information on using WebSpy Vantage with Clearswift SECURE Web Gateway.

Cisco Firewall Bandwidth loader

We have also introduced a new Loader for Cisco ASA, PIX and IOS Firewall devices. This new loader imports TCP, UDP, ICMP and GRE ’session close’ events into one schema, allowing you to aggregate size values across these events. This loader is called Cisco Firewall (Bandwidth) and is now available on the Loader Selection page of the Import Wizard. Previously, these events were imported into separate schemas so there was no great way to determine total bandwidth from your Cisco syslog files (without using Netflow and WebSpy FlowMonitor).

Palo Alto Networks and WatchGuard XTM

We’re also very happy to welcome Palo Alto Networks to the WebSpy supported log file list. Vantage now supports both the CSV and syslog file formats from your PA Firewall.

Another new addition is support for the latest WatchGuard XTM devices running firmware version 11.

Full List of Changes

Here’s the full list of changes included in this update:

• New: Clearswift SECURE Web Gateway W3C.
• New: Palo Alto Networks Firewall (CSV/Syslog)
• New: Cisco Firewall (Bandwidth): This new Cisco loader imports TCP, UDP, ICMP and GRE events from ASA, PIX and IOS syslogs into one schema to aggregate size values across these events.
• New: Added WatchGuard XTM: Currently http-proxy, https-proxy, smtp-proxy and firewall lines are supported.
• Fixed: ISA Server: Fixed format detection issues, and issues importing hits with very large size values.
• Fixed: IronPort WSA: Fixed format detection issues, as well as the import issue “Invalid value for DVS Scan Code”
• Fixed: Sophos WSA: Fixed format detection issues and invalid line issues.

How to update

To update your software, simply click Tools | Check for updates. To update the Vantage Web Module, right-click the WebSpy system tray icon and select ‘Check for updates’. If you have issues with the Web Module update process, please see: http://www.webspy.com.au/forums/viewtopic.php?f=4&t=29

Let me know if you have any questions or issues!

The default LDAP query when you first run through the Import Organization wizard should filter these computers objects out. The query is:
(&(objectCategory=person)(objectClass=user))

In Active Directory, computers do not generally have an objectCategory equal to Person. Computers usually have the objectCategory ‘Computer’.

If by chance your computers are not being excluded by this filter, you could exclude all objects without an email address. This of course assumes that all users you want to import have an email address populated in Active Directory. To exclude objects without email addresses, the filter becomes:

(&(objectCategory=person)(mail=*)(objectClass=user))

Another useful addition to the query is to exclude users that have been disabled in Active Directory. You usually disable an account when a person leaves the organization, but you still need their user profile in Active Directory for whatever reason. This query is slightly less obvious:

(&(objectCategory=person)(mail=*)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))

For information on what the numbers mean in the query above, see How to query Active Directory using a bitwise Filter

Another question I’m often asked is how to exclude specific OUs from a query. Unfortunately LDAP does not support this concept and the only way to do this is to run multiple queries on different root level DNs. This means running through the Import Organization wizard multiple times with a different Root Distinguished Name each time, and the ‘Merge’ options set to ‘Keep users that are no longer in the directory’ and ‘Keep existing user details’.

If you have other helpful LDAP queries, please leave a comment below.

An issue was introduced in build 2.2.0.42 when a change was made to the underlying data type of the Url Category field in the Microsoft TMG Web Schema. Unfortunately this change resulted in nulls being imported into the Url Category summary when importing from TMG’s SQL log files. Fortunately, importing W3C text logs were unaffected by this change.

This issue has now been fixed and released as an auto-update. To update your software simply select Tools | Check for updates.

If you have existing storages that have not been importing the URL Category, go to the storages screen and click ‘Reload All’. This will re-import your log files and populate the Url Category summary.

For all the customers affected by this issue, thank you for your patience and understanding.

Here’s the full list of changes:

• New: Juniper SA Series. Vantage can import and report on web traffic and VPN connections.
• New: Microsoft Forefront Protection for Exchange 2010 format.
• New: Avencis SSOx.
• Improved: IronPort WSA: Department and Message fields were sometimes returned as null. Fixed.
• Improved: Microsoft FTMG: Removed usage of deprecated “FilterInfo” field from W3C Web format.
• Improved: Microsoft IAS Radius: Added support for Source/Destination IP and port (field code 5000).

To update your Vantage application simply select Tools | Check for updates.

To update the Vantage Web Module, right-click the WebSpy icon in the Web Module server’s system tray, and select Check for updates. If you have any issues with the Web Module update process, please see my previous blog regarding Web Module Errors and Workarounds.

This release will be welcomed with open arms by many customers for the following reasons:

• General usability improvements in the Web Module
  Multi-select / delete options, Ajax progress indicators to avoid page refreshes, export from Dynamics Report tab and more (see below)
• Fixes to the Microsoft Forefront TMG loader
  See my other post: Microsoft Forefront TMG logs size fields the wrong way around. Also fixed ‘value cannot be null’ error when importing SQL logs.
• Fixes to storage corruption issues
  This build should prevent ‘Normalization Index’ storage corruption issues from occurring. This often occurred after importing data, editing some log inputs and reimporting.
• New loaders and more fixes
  See below for the full list

To update your Vantage application, simply choose Tools | Check for updates. To update the Web Module, right-click the WebSpy icon in your system tray and select ‘Check for updates’. If you have any issues updating the Web Module, please see my previous post Web Module Update Errors and Workarounds.

Web Module Changes:

• New: Task progress is now updated without refreshing the page
• New: Added multi-select / delete functionality to Reports, Analyses and Storages tables.
• New: Added export functionality to Dynamic Reports view.
• New: Added Performance section on the Options tab to enabling multi-processing (improves Analysis speed)
• Fix: Dynamic Reports view now supports Trend reports.
• Fix: Organization selector on Dynamic Reports view now always reflects updated data under IE6/7/8.Fix: Fixed javascript errors in IE when expanding the organization filter.
• Fix: Report template names are no longer truncated on the Dynamic Reports view.
• Fix: Fixed errors that may occur when collating reports on the Dynamic Reports page.
• Fix: Authentication errors are now logged with stack trace.

Vantage Changes

• Fixed: ‘Normalization index’ storage corruption problems.
• Fix: Report collation: Added support for collation of Min/Max aggregates on DateTime columns (time of first hit etc). Also added support for arrayed fields (for example, category fields with a comma separated list of categories)
• Fix: Import windows wizard now remembers settings for Import all or selected users
• Fix: Organization: Filtered LDIFs may now be imported when references to some users are missing (for example, if a user’s manager does not exist in the LDIF)
• Fix: Improved connection and error handling between Vantage and the Web Module.

Loader Changes

• New: BlueReef Sonar Total Management Module
• New: Microsoft Sharepoint 2007
• New: SmoothWall Guardian 7.0 format
• New: Sun One Proxy (Supported under Sun One Webserver)
• Fixed: Astaro: Improved format detection
• Fixed: Cisco: Strings in the IP fields of 113019 lines are now imported
• Fixed: IronPort WSA: Improved log format detection
• Fixed: Microsoft Exchange 2007: No longer raises issues regarding total-bytes or internal-message-id fields
• Fixed: Micorosft FTMG (Web) SQL: No longer encounters value could not be null errors
• Fixed: Microsoft FTMG: Added option to reverse bytes received/sent fields. See Microsoft Forefront TMG logs size fields the wrong way around
• Fixed: Microsoft IIS W3C: Now imports cs-method and connection ID
• Fixed: Sophos Web Appliance: Switched the outgoing and ingoing sizes so that they are now the correct way around
• Fixed: Fixed import new hits issue associated with W3C formats. You must reload your logs before this change will take affect. Formats affected include: BlueCoat, Clearswift, Microsoft Exchange 2007, Microsoft FTMG, Microsoft Windows Media Services, WebSpy Live Tracking Log

Enjoy!

Microsoft TMG Log showing Bytes Sent consistently larger than Bytes Received

This issue has been confirmed by the Microsoft Forefront TMG team, and unfortunately there is no ETA for a fix.

We obviously don’t want our reports showing incorrect usage figures, so we’ve fixed our TMG loader so that it imports the ‘bytesrecvd’ field into the Bytes Sent aggregate, and the ‘bytessent’ field into the Byte Received aggregate.

But what if Microsoft release a fix? What we’ve done is implemented a loader property to allow you to turn off this behavior. This will allow you to import your old logs with the fields reversed, and your new logs with the fields the right way around.

To access the loader property:

• On the import wizard, select the Microsoft FTMG format and click the Properties button on the toolbar
• Select Microsoft FTMG from the drop down list
• Notice the option to ‘Reverse Bytes Sent and Received to compensate for bug in TMG’s logging’. Leave this checked until Microsoft issue a fix.

Microsoft Forefront TMG Loader Option to Reverse Bytes Sent and Received

This fix is available in Vantage build 2.2.0.48 (and above) which has been released as an auto update. So simply select Tools | Check for updates to ensure you have this fix.

So let’s use WebSpy Vantage to drill into that Anonymous user and find out what is going on.

One way to do this is to run an Ad-hoc analysis on the Summaries screen and drilldown into the Anonymous user to view all the information about that user. However, TMG and ISA tend to log a lot of information that may not be relevant to this particular investigation, so I’ve created some report templates (one for ISA and one for TMG) and a set of Aliases that pull out some relevant information.

Download our Anonymous Traffic Investigation Report

If you’re running WebSpy Vantage download the Anonymous Traffic Report Templates & Aliases

Then open the .Templates file on the Reports tab, and the .Aliases file on the Aliases tab. Once you have both files opened, go to the Reports tab and click either the ‘Anonymous Traffic Investigation (ISA)’ or the ‘Anonymous Traffic Investigation (TMG)’ report. Then click the ‘Generate report’ link and run the report template on your ISA or TMG storage.

The report gives you the ability to drill into the Allowed, Denied and Failed traffic to see a list of the unauthenticated IPs, Sites, Rules responsible for blocking or allowing the traffic, unauthenticated Applications and Result Codes.

Main causes of anonymous traffic

What you will probably find is that most of the Anonymous traffic is being denied by your TMG or ISA firewall. When a client first requests a web page, the proxy will challenge the client for authentication. These events are often logged with the result code 12209 meaning ‘authorization is required to fulfill the request’. These requests are therefore denied by the proxy until the client’s credentials are authenticated.

Have a look at the amount of traffic being denied and then checkout the Result Codes associated with the denied traffic. Chances are you’ll see ‘proxy authentication required’ appear predominantly.

If you also look at the Applications section you may also find that Windows Updates are sailing through your TMG or ISA firewall unauthenticated.

Filter out unauthenticated traffic from Reports

The most logical next step is to filter out the information you do not want in your reports. You’ll probably still want to include Windows Update traffic in your reports, but you’re probably not so interested in the ‘proxy authentication required’ information. So let’s filter that out.

To do this:

1. Go to the Reports tab and select the report you want to filter (such as your Organization report)
2. Click ‘Edit Template’, then click ‘Template Properties’.
3. In the filter section at the bottom of the dialog, click Add | Field value filter.
4. Select the ‘Result Code’ summary and select the Status Code Names (ISA-FTMG) alias.
5. On the toolbar, search for Authorization, and check the following two items:
   • The server requires authorization to fulfill the request. Access to the Web Proxy filter is denied.
   • The server requires authorization to fulfill the request. Access to the Web server is denied. Contact the server administrator.
6. Ensure the ‘Exclude’ radio button is selected and click OK.

If you decide that you don’t care about seeing ANY unauthenticated traffic in your reports, you can always simply filter out the Anonymous user from your reports.

To do this:

1. Go to the Reports tab and select the report you want to filter (such as your Organization report)
2. Click ‘Edit Template’, then click ‘Template Properties’.
3. In the filter section at the bottom of the dialog, click Add | Field value filter.
4. Select the ‘Username’ summary.
5. On the toolbar, click Add and type ‘anonymous’. Click OK.
6. Ensure the Exclude radio button is selected and click OK.

Hopefully this article improves your understanding of the ‘anonymous’ user, and gives you some actions to take for your specific reporting situation.

If you have any questions, please leave a comment below.

• How to import your log files and explore the information recorded by IronPort using the Summaries screen
• How to open the customized IronPort Report Templates and Aliases
• How to generate reports
• How to import your organizational structure and report on departments
• How to setup the Web Module and publish reports

PART 1: Importing log files & exploring your IronPort summaries

Once you have exported your IronPort access logs (see http://www.webspy.com.au/vendors/ironport/howto.aspx#ftp), this video takes you through importing your logs into WebSpy Vantage and analyzing data on the Summaries screen.

PART 2: Opening the customized IronPort Templates & Aliases, and running reports

This video takes you through opening the IronPort-specific report templates and aliases and generating a report that provides an overview of your organization’s Internet usage.

PART 3: Importing your Organization structure & generating department reports

This video shows you how to import your organizational structure into WebSpy Vantage from a directory server (such as Active Directory) using LDAP, and then generating a report that contains information on your newly imported departments.

PART 4: Using the Web Module.

This video takes you through configuring and using the WebSpy Vantage Web Module. Specifically, it takes you through the following tasks:

• Configuring the Web Module for Windows Authentication
• Adding a Web Module to Vantage
• Publishing reports to the Web Module
• Adding permissions for a user
• Synchronizing the Web Module
• Using the Dynamic Reports tab

PART 5: A quick word about tasks & conclusion

This video summarizes the actions taken in the previous four videos and also briefly discusses how to automate the reporting processing using scheduled tasks.

I hope this helps! Let me know if you have any questions by leaving a comment below.

The log databases are stored in an SQL Express instance named MSFW. By default these databases cannot be accessed by a remote computer. I’d first like to say that we recommend changing TMG’s logging to W3C text files, as these logs are about 5-6 times faster to import, and you don’t need to worry about the steps below.

But if you need to stick with the SQL Express logging, here are the basic steps to enable access to the logs from a remote computer:

Enable TCP access to the MSFW instance

To do this:

1. Log into your Forefront TMG server using administrator credentials.
2. Select Start | All Programs | Microsoft SQL Server 2008 | Configuration Tools | SQL Server Configuration Manager.
3. Expand SQL Server Network Configuration and select Protocols for MSFW
4. Right-click TCP/IP and select Enable
5. Click OK on the Warning dialog informing you that “changes will not take effect until the service is stopped and restarted.”

Enabling TCP/IP on the MSFW instance

Set the listening Port on the MSFW instance

Once TCP/IP is enabled on the MSFW instance, you need to set it to listen on port 1433

1. Select Protocols for MSFW under SQL Server Network Configuration
2. Right-click TCP/IP and select Properties.
3. Click the IP Addresses tab and scroll to the IPAll section at the bottom of the list.
4. Change the TCP Port to 1433 and ensure nothing is entered in TCP Dynamic Ports (Delete the ‘0′ value  if present). Click OK and click OK on the Warning dialog.

Setting the Port on the MSFW instance

Change the listening port on the ISARS instance

The ISARS SQL instance also listens on port 1433 and this can cause connection issues. Change this instance to use port 1434:

1. Still in SQL Server Configuration Manager, select Protocols for ISARS under SQL Server Network Configuration
2. Right-click TCP/IP and select Properties.
3. Click the IP Addresses tab and scroll to the IPAll section at the bottom of the list.
4. Change the TCP Port to 1434 and ensure nothing is entered in TCP Dynamic Ports. Click OK and click OK on the Warning dialog.

Changing the port on the ISARS instance

Restart the Services

For the above changes to take effect, you need to restart the SQL Server (ISARS) and then the SQL Server (MSFW) services in that order.

1. Go to Start | Administrative Tools | Services
2. Right-click the SQL Server (ISARS) service and select Restart.
3. Right-click the SQL Server (MSFW) service and select Restart.

Test the connection from the WebSpy machine

You should now be able to connect to the MSFW databases from a remote computer. To test the connection, we recommend that you install SQL Management Studio on the machine running WebSpy and try to connect to <TMGservername>\MSFW, 1433 (replace <TMGservername> with your actual server name or IP address). For example TMGServer\MSFW, 1433 or 192.168.0.10\MSFW, 1433.

As long as you are logged into Windows with a user account that is a local administrator on the TMG server, you should be able to connect without issue.

Importing the TMG Log files into WebSpy Vantage

Once you have established a connection, you can import your logs using WebSpy Vantage like so:

Create a new Storage

Select Database Connection

Select the Microsoft FTMG Loader

Click Add

Enter TMGServer\MSFW and port 1433

Successfully Imported WebProxy Logs

The screenshots above also illustrate using a database mask of *WEB* to only import the WebProxy logs. If you only want to import the Firewall logs, set the database mask to *FWS*. If you want to import both the WebProxy and Firewall logs, leave the database and table masks set to *.

Now that you have your log files imported, you can run a quick ad-hoc analysis on the Summaries screen or generate any of Vantage’s default web of firewall reports. M

Make sure you also download our Forefront TMG specific Aliases and report template. For more information, see our Forefront TMG How To page.

If you have any questions or encounter any hurdles, please leave a comment below.

“I would like to know how much of my bandwidth is being eaten by each protocol. I will then use this information to determine if circuit may need to be increased due to increased traffic”.

This customer was collecting syslog messages from a Cisco Firewall, then using WebSpy Vantage to generate reports. In theory, this sounds like a fair plan. Unfortunately, the Cisco Firewall logs many different types of messages. Some to do with denied packets, some to do with authentication, some for vpn and so on. The information contained within each message changes. Some events include the size information that is required for any type of bandwidth assessment and some don’t. Correlating the required events to get any sort of accurate ‘bandwidth’ representation is a bit of a nightmare.

Fortunately, there’s a simpler method. If you search the Cisco website or the Internet for bandwidth utilization reporting, you’ll no doubt be pointed in the direction of NetFlow.

NetFlow is a network protocol developed by Cisco Systems to run on Cisco IOS-enabled equipment for collecting IP traffic information [Source Wikipedia http://en.wikipedia.org/wiki/Netflow]

There are a couple of commands to enter on your router to turn NetFlow on, and then you just need a NetFlow collector to receive the Netflow information and generate reports.

Fortunately WebSpy has developed a little tool called FlowMonitor that collects the Netflow information and writes a log file that can then be imported into WebSpy Vantage and reported on.

The FlowMonitor Management Console

Once your FlowMonitor logs are imported into WebSpy Vantage, you can run the default FlowMonitor report to see the size of traffic flowing between IP addresses, subnets, router interfaces or protocols. Alternatively you can create your own custom reports to see exactly what you want to see.

NetFlow doesn’t record usernames or URLs so it’s not great for reporting on the web sites your users are visiting, but it is great for network administration and trouble shooting. Identify chatty IP addresses, protocols that are chewing too much bandwidth, the times throughout the day when incoming or outgoing links become heavily utilized and so on.

For information on how to configure your router and deploy FlowMonitor, see the FlowMonitor Installation and User Guide. You can also download a free trial here.

FlowMonitor is a handy little tool. Ask your friendly WebSpy account manager about it today!

Vantage uses multiple threads to perform certain tasks. As general rule, the more threads being used simultaneously, the higher the CPU utilization. There are a few situations where Vantage uses multiple threads simultaneously:

CPU usage when importing log files

When importing more than one log file, each log will be imported with a separate thread. As CPU usage increases when more threads are used, importing a single log file won’t push your CPU, but importing a folder full of logs will.

CPU usage when generating reports

CPU performance can also be affected by the structure the report you are running. Report templates have what we call ‘Nodes’ in them. You can go into a report template, right-click | add node. Think of each node as an SQL query. When generating a report, each node gets processed in a separate thread, and if the nodes are ‘at the same level’ they get processed at the same time.  Here’s a screenshot showing what I mean by nodes at the same level.

A report template with two 'levels' of nodes

The three ‘red’ nodes will be processed at the same time, and then the three ‘green’ nodes will be processed at the same time. The green nodes won’t be processed until the red nodes have been processed. The more nodes being processed at the same time increases the number of simultaneous threads and the amount of CPU being used.

CPU usage when filtering reports

CPU usage is also affected by the number of records being processed from your storage. If you are running a report on your entire storage with no filters, then Vantage will be pushing all records in your storage through the reporting engine. If you run the same report but with a filter for a specific user, then Vantage will seek through the records in the storage until it finds a record for that user, then push that record through the reporting engine. This results in a ‘trickle’ of records being pushed through the reporting engine so it doesn’t get a chance to really push your CPUs.

A filter that excludes a lot of information that exists in your storage is the most common reason for low CPU utilization while running a report.

In Short

The number of logs, report template structure and filters can all have an effect on the way Vantage utilizes your CPUs.

We also have some exciting ideas on our roadmap to ensure Vantage utilizes as many CPUs as you can throw at it. Until then, I hope the above information helps you understand when and why your CPU usage will and won’t be pushed.

This is a great update for Vantage Ultimate users as we’ve introduced a new feature/tab into the Web Module called ‘Dynamic Reports’.

If you’re publishing the same report to the Web Module each day, you can use the Dynamic Reports tab to select a date range and a department (or whatever organizational groups you have defined) and the Web Module will collate all the daily reports that match that filter into one report. This allows you to report on entire week, month or year by simply ‘reporting on reports’, rather than reporting months of raw storage data.

Here’s the full list of changes since the last auto update (2.2.0.32 on the 14th April 2010).

Application Changes

• Added Dynamic Reports feature to the Web Module.
• Rewrote the Web Module transfer protocol. New protocol adds version checking, connection checking, and integrity checking for high latency environments.
• Purge data from storage task no longer prevents importing new hits when all data is removed from an input within a storage.
• IPv6 addresses now show IPv4-mapped addresses as plain IPv4 addresses in summaries.
• IPv6 and IPv4 addresses are now freely interchangable in filter expressions.
• Fixed IPv6 drilldowns on the Summaries screen
• SQL inputs can now be resumed from the previous position. Previously any input that was partially imported would be skipped when importing new hits.
• Template-based analysis has been fixed, no longer results in blank/non-existent analysis.
• Added new string manipulation functions to expression language; Contains, StartsWith, EndsWith, IndexOf.

Loader Changes

• Astaro: Now checks that the ID field is present in a line before attempting to read it.
• Barracuda Web Filter: Added this format to replace Spy Filter.
• BlueCoat Proxy SG W3C: Added support for gmttime, timestamp, x-bluecoat-surfcontrol-is-denied and x-bluecoat-transaction-id.
• ClearSwift: Added a new loader group for ClearSwift that includes the MimeSweeper loaders
• ClearSwift SECURE Web Gatway: Now supported with the Web Appliance loader
• Clearswift Web Appliance: User summary displays Source IP if Username is blank.
• IronPort WSA: Fixed memory usage issues.
• Microsoft FTMG: Added category name lookup to SQL loader.
• Microsoft FTMG: No longer fails to import lines where the rule field contains square brackets.
• Microsoft FTMG: URL Category field is now a string instead of an integer. Added URL Categorization Reason field.
• Microsoft FTMG: Fixed memory usage issues.
• Microsoft IIS W3C: No longer hangs or crashes when loading a file that isn’t IIS W3C.
• NetAsq: Added support for srcname field. The Username summary is populated with user first, and then srcname if user is blank. The User summary is also now populated with Source IPs if the Username summary is blank.

To update WebSpy Vantage, simple select Tools | Check for updates.

To update the Web Module, login to the Web Module server, right-click the WebSpy system tray icon, and select Check for updates.

As always, please contact us if you have any issues or questions.

Unfortunately there are two issues with the Web Module auto update process. Everyone on a 64 bit operating system will encounter issue #1 (Unable to locate installation location), and some of you may encounter issue #2 (System.IO.FileLoadException).

This article describes the errors and how to work around them to successfully update the Web Module.

Issue #1

The usual process to update your Web Module is to log into your Web Module server, right-click the WebSpy system tray icon and select ‘Check for updates’.

If you do this on a 64 bit operating system, you will receive the following error:

Web Module Updater error: Unable to locate installation location

You can fix this issue by clicking Yes, and specifying the Web Module’s installation location, which is usually somewhere under c:\inetpub\wwwroot (or just c:\inetpub\wwwroot if you didn’t specify a virtual directory when installing).

This will allow the updater to continue and you will be prompted to download and install the latest update.

Issue #2

Unfortunately, you may encounter another error during the update installation. The text of the error will be something along the lines of:

System.IO.FileLoadException: Could not load file or assembly ‘ICSharpCode.SharpZipLib, Version=0.84.0.0, Culture=neutral, PublicKeyToken=1b03e6acf1164f73′ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0×80131040)

Work Around

We are currently working on solutions to both of these issues. In the mean time, here is a work around to install the update. On the Web Module server:

1. Download this file:
   http://update.webspy.com/autoupdate/files/vantagewebmodule/vantagewebmodule2.2.0.18.zip
2. Stop IIS (See instructions below).
3. Right-click the WebSpy system tray icon and click Exit.
4. Backup your existing Web Module installation by copying everything in your Web Module’s installation folder (usually under c:\inetpub\wwwroot (or just c:\inetpub\wwwroot if you didn’t specify a virtual directory when installing) into a completely separate location (i.e. don’t keep it in a sub-folder).
5. Extract the downloaded zip file to your web module’s installation folder. Overwrite all the existing files.
6. Start IIS (see instructions below).

Your Web Module will now be updated to the latest version.

Stopping and Starting IIS

In IIS Manager (Start | Control Panel | Administrative Tools | Internet Information Services (IIS)) , right-click the site you want to start or stop, and click Start or Stop

We sincerely apologize for the inconvenience and will hopefully have a solution out soon. If you have any problems with the process above, please contact support at webspy dot com.

For those that don’t know what Soho is all about, check out this video:

Soho is a dashboard application that displays download and upload traffic statistics for each computer in your network. If you haven’t yet tried Soho, please give it a go and download it here.

It is our intention to keep our Alpha testers up to date with our ongoing development. Right now, I’d like to inform you about some issues experienced by a handful of testers and how to go about resolving them.

Learn to restart the Soho Agent

First of all, one of the handiest things we can tell you right now is how to restart the Soho Agent. This single step is resolves 99% of all Soho issues, at least temporarily. If these steps seem too complicated, rebooting your PC also has the same effect.

To restart the Soho Agent on Windows:

1. Launch the Services Console by going to Control Panel | Administrative Tools | Services. Or if you like handy short cuts, try Start | Search (or Run), Type ‘services.msc’ (without the quotes) and press enter.
2. Right-click the “WebSpy Soho Agent” service and select Restart. If you get a ‘time out’ error message or warning, ignore it and right-click the service again and select Start.

To restart the Soho Agent on Mac OS:

1. Open the terminal from /Applications/Utilities/Terminal
2. Type sudo launchctl stop “WebSpy Soho Agent”
3. Enter your user password if requested.
4. Wait about 5 seconds.
5. Type sudo launchctl start “WebSpy Soho Agent”
6. Again, enter your user password if requested

OK, now you have the skills, here are the issues and work-arounds:

100% CPU usage after sleep or hibernate

Some users reported Soho’s impressive ability to consume 100% of their CPU when their computer wakes from sleep or hibernation. A few users experienced a slow and sluggish PC, and uninstalled Soho immediately.

If you’re looking for the Soho process in Windows Task Manager you will not see it until you click the Show processes from all users button (or checkbox in XP). This is because the Soho Agent runs under the System user account in order for it to run, no matter who is logged onto the PC.

From here you can end the WebSpy.Soho.Agent.exe process and everything should return to normal. You can then restart the “WebSpy Soho Agent” to get Soho working again (see steps above).

We believe we have fixed this and are in the middle of some final testing. All going well, we should have a new build ready for you very soon. In the mean time, you may like to disable sleep and hibernation in your PC’s power options.

Soho doesn’t install on Mac OS 10.5??

Our first Alpha release did not install on Mac OS 10.5 (Leopard). This was due to a silly checkbox in our packaging system not being checked. We’ve checked the checkbox and uploaded a new build to our web site. You can download it from here:
http://www.webspy.com.au/products/soho/download.aspx

Note: Soho will only install on Mac OS 10.5 (Leopard) and 10.6 (Snow Leopard). Versions 10.4 (Tiger) and below are not supported.

All computers disappear from the Current Activity chart except the local computer

Sometimes all computers will disappear from the Current Activity chart leaving your local computer all by itself. This happens when the communication between Soho Agents becomes jammed. You can usually resolve the issue by restarting the Soho Agent on your local computer (see steps below). If this doesn’t work, restart the Agents on all other computers running Soho. We are currently working on a fix for this issue.

The Soho User Interface is completely blank

If there is no information in the Total, Current Activity or History chart, this is because the Soho Agent has stopped running. Reasons for this may vary, so please let us know if this is regularly occurring. Restarting your agent usually resolves the issue (see steps above).

Feedback

Thank you to everyone that has submitted feedback so far.

Just a reminder to please let us know if your network card works or doesn’t work with Soho on this page:
http://www.webspy.com.au/products/soho/supportednics.aspx

You may also like to review the current list of features and bugs and vote them up or down at:
http://webspysoho.uservoice.com/

There is also a dedicated Soho Alpha Feedback thread in our forums at:
http://www.webspy.com.au/forums/viewtopic.php?f=8&t=12

We will let you know when fixes are available to the above issues.

When imported into WebSpy Vantage, the result is shown in a new summary called ‘Group’ which you can add to your reports.

We also added support for the custom fields Bytes Sent and Bytes Received. Due to the absence of a header in the IronPort access log, Bytes Received and Bytes Sent fields must both be present to be detected, and the Received field must precede the Sent field.

We also added support for the custom fields Request Body Size and Response Body Size. These fields can be included in your access log by adding %q (Request body size) and %b (Response body size) in the ‘Custom Fields’ edit box. Due to the absence of a header in the IronPort access log, Request Body Size and Response Body Size fields must both be present to be detected, and the Request field must precede the Response field.

We’ve also noticed that the values in the Bytes Sent and Bytes Received fields do not necessarily add up to the value logged for ‘Size’. We’re discussing this issue with our friends at IronPort and we will hopefully post a solution or explanation soon..
The information we first received about these fields indicated they represented Bytes Sent and Bytes Received. This is the way they are represented in the builds below (2.2.0.29). We will release a new build soon, with the field names changed to Request body size and Response body size. Body size is different to bytes sent/received as it does not include bytes from packet headers etc.

We’re yet to issue an automatic update for the Vantage applications, so in the mean time you can download the latest builds here:

Vantage Ultimate:
ftp://ftp.webspy.com/webspy/Builds/VantageUltimate2.2.0.29.zip

Vantage Web Module:
ftp://ftp.webspy.com/webspy/Builds/VantageWebModule2.2.0.8.exe

Vantage Giga:
ftp://ftp.webspy.com/webspy/Builds/VantageGiga2.2.0.29.zip

Vantage Premium:
ftp://ftp.webspy.com/webspy/Builds/VantagePremium2.2.0.29.zip

To apply the Vantage update, close Vantage and extract the downloaded file into Vantage’s installation folder (Usually c:\Program Files\WebSpy\Vantage <flavour> 2.2). Overwrite the existing files.

To apply the Web Module update, uninstall the Vantage Web Module from Add/Remove Programs (Programs and Features in Windows 7/Server 2008), then run the downloaded exe file, making sure you specify the same server, virtual directory and data location that your Web Module was previously using.

We will be releasing this as a public auto-update soon. Let us know if you have any issues.

Check your version in Help | About. If you are running 2.2.0.27 or above, then you already have this update. If not, make sure you update to your software by selecting Tools | Check for updates.

Here’s a quick video outlining some of the differences between TMGs Reporting, and what can be achieved using WebSpy Vantage. The video does not illustrate all the limitations outlined below, so please read on.

Whats is in the Forefront TMG report?

The default TMG report contains the following sections

• Summary
• Web Usage
• Application Usage
• Traffic and Utilization
• Security
• Malware Protection
• URL Filtering
• Network Inspection System

Each section contains overviews such as ‘Top users’ and ‘Top Sites’.

If your reporting requirements can be satisfied with these overviews – that’s great! Unfortunately, when you start thinking about what system administrators and other people in your organization actually need to make informed decisions, this report is quite limiting.

The 8 Limitations of Microsoft Forefront TMG’s Reporting

Here is what I consider to be the 8 main limitations of Microsoft Forefront TMG’s reporting functionality.

1. No Drilldowns

Want to see the sites that the top 5 users accessed? Want to see the users that downloaded the most traffic from youtube? These are fairly standard reporting requirements that simply cannot be achieved using the inbuilt TMG reporting.

WebSpy Vantage lets you either interactively drilldown into a user or site, or produce a regular report that includes further details about what your top users have actually been up to.

2. No Filtering

When you generate a report in TMG, you can only filter the report by a date range. There is no way to filter out anonymous (unauthenticated) traffic or exclude traffic coming from advertising servers (such as doubleclick and 2mdn.net) that tend to dominate most of the top 10 sites.

This can easily be achieved using WebSpy’s software. Check out my video on how to remove clutter from your web reports.

3. No Customization

Customization of each overview in the TMG report is limited to the number of items to show (e.g. top 5 or top 50 users), and the sort order (Incoming Bytes, Outgoing Bytes, Requests and Total Bytes).

What about the time a user spent browsing the web, or the number of users that visited a specific site? There is no way to add custom columns such as total browsing time, average session time, or number of users/sites/IPs to the report tables.

Or say you simply want to change your top users chart from a bar to pie to easily see the percentage used. Nope sorry!

If you do make one of the two available customizations in a TMG report, you then get the annoying Apply / Discard message to save changes to the configuration database.

All of these customizations can be achieved using WebSpy Vantage, and it doesn’t touch your TMG server to apply a change to a report.

4. Limited Report Distribution

When you generate a report, you get the option to email it to a specific email address. What if you would like to create a report for every department, and then email it to the managers of each department? Or better yet, host the report on a secure web server where department managers can log in and view their reports?

WebSpy Vantage Ultimate comes with a secure ‘Web Module’ specifically for this purpose and managers still receive a link to the report via email.

5. Cluttered ‘Top Sites’ List

The ‘Top sites’ list can become particularly cluttered due to the inclusion of sub-domains. I don’t want to mentally add up the size values from farm1.static.flickr.com, farm2.static.flickr.com, and farm3.static.flicr.com – I just want to know how much was downloaded from flickr.com.

This is compounded by the inability to exclude sites that are merely placing advertising banners on the actual sites users are visiting (as mentioned in the ‘No Filtering’ limitation above).

WebSpy Vantage breaks URLs down into separate components and lets you analyze each part separately. Look at the Site Domains summary to remove sub-domains and see only flickr.com. Or perhaps you want to see the keywords a user entered into search engines like Google? Or perhaps the top pages accessed within a website? No problem. Just include the Site Keywords or Site Resource summaries in your Vantage reports.

6. No Grouping or Aliasing

There is no way to group users into departments or locations, or IP addresses into subnets, or extensions such as .html, .pdf or .exe into file types. The ability to group and represent raw log data in more meaningful ways, as offered by WebSpy Vantage, can increase the value of a report tremendously.

7. No Productivity Assessment

One of the major features introduced in TMG since ISA Server 2006 is the included URL categorization technology.

Although the TMG report gives you an overview of the categories that have been visited, the report does not use this information to display a productivity assessment for your users.

WebSpy Vantage not only provides this assessment, but also the ability to customize the categories that are deemed productive as this can vary wildly depending on the industry and organization.

8. Not browser independent

This is a minor limitation that can be a major annoyance. The report that TMG produces is a HTML report that only displays correctly in Internet Explorer. As Forefront TMG is a Microsoft product, this is not exactly surprising, but still very annoying if IE is not your default browser.

How to get awesome reports from Forefront TMG

If you have had personal experience with any of the above limitations, you’ve probably been hunting for an alternative solution. I strongly recommend checking out the WebSpy Vantage range of products, and if you would like secure report distribution via the ‘Web Module’, Vantage Ultimate is what you are after.

If you agree or disagree with anything in this article, I encourage you to leave your thoughts in the comments.

Cheers!

Scott

TMG or UAG? What is the difference?

TMG is an outgoing proxy that protects your internal users from malware, viruses and the like. TMG generates some great web proxy log files to import into WebSpy Vantage allowing you to monitor where your users are going on the Internet, how much they’re downloading etc.  TMG, unlike ISA, now has deep packet inspection for HTTPS traffic, plus a bunch of other new features.

UAG is an incoming proxy that provides employees, partners and vendors secure remote access to corporate resources such as Outlook Web Access (OWA) and Sharepoint (MOSS). It utilizes the TMG engine, but this is mainly just to protect the UAG server (more on this topic here http://technet.microsoft.com/en-us/library/ee522953.aspx).

TMG can also publish your OWA and MOSS sites, but this is no longer recommended by Microsoft. They recommend using a dedicated UAG server to perform this function.

Upgrading to TMG

If you’re thinking about migrating your ISA server (2004 or 2006) to TMG, you may like to check out this migration guidance video with Mohit Saxena (Senior Technical Lead) and Jim Harrison (Program Manager). http://edge.technet.com/Media/ISA-to-TMG-Migration-Guidance/

Microsoft Forefront TMG Migration Video

Reporting on TMG

If you’re using TMG at the moment, we invite you to analyze your web proxy and/or firewall logs using WebSpy Vantage and tell us what you think!  Download your copy of WebSpy Vantage here, and import your logs using the Microsoft FTMG format:

Microsoft Forefront Threat Management Gateway

In the mean time, you can do it by editing an XML file manually.

If you go to the Web Module’s data folder you will find a file called “Vantage Web Module.Reports”.  If you open this in Notepad, you’ll notice chunks of xml for each report:

<WebReport>
<Guid>89208266-42e5-44bc-baa2-157c404c9688</Guid>
<Title>Business Unit Report</Title>
<Date>633945813293343143</Date>
<Type>Analysis</Type>
<Access>
<Everybody>False</Everybody>
<Attributed>True</Attributed>
<Specific>True</Specific>
<SpecificEntities>
<item>person:CN=Luke,OU=Users,OU=Australia,OU=Webspy,DC=wsy,DC=com</item>
</SpecificEntities>
<Managers>True</Managers>
<ManagerLevelRestriction>3</ManagerLevelRestriction>
</Access>
<Attribution>person:CN=Scott,OU=Users,OU=Australia,OU=Webspy,DC=wsy,DC=com
</Attribution>
</WebReport>


Depending on how you published your reports, the unique ID of the person that currently has access to the reports will be mentioned in either the ‘SpecificEntities’ or ‘Attribution’ section.

You just need to find/replace this with the unique ID of the person you would like to transfer these reports to. You can find the unique ID of a person on the Organization screen in Vantage.  It’s called ‘Distinguished Name’:

Finding a user's unique ID in Vantage

A few more things:

• Make a backup of your original “Vantage Web Module.Reports” file before making any change.
• As you can see above, people need to be entered into this XML file using the syntax
  “person:<uniqueID>“
• You will also need to stop IIS before making any change as the web module caches this data in its memory while running.

As mentioned, creating a user interface to do this is a planned feature so follow us on Twitter, or subscribe to our RSS feed for updates!

In the mean time, I thought I’d share a way to re-brand the main elements in the Web Module by editing a few files and replacing a few images.

The only issue with this technique is that any future auto-updates for the Web Module will overwrite your edited files, so you just need to keep a copy of your customized files, so that you can restore them again after the auto-update.

Before you begin

In order to edit anything, you first need to know where your Web Module is located on your web server’s hard drive. This can be found by opening IIS Manager (Start | Control Panel | Administrative Tools | Internet Information Services (IIS) Manager) expanding the left hand server/site tree to find your Web Module.

• In IIS7, select the Web Module and click Basic Settings… in the right hand ‘Actions’ panel. The location is specified in ‘Physical Path’.
  

  Finding the Web Module's physical path in IIS7

• In IIS6, right-click the Web Module and select Properties… then go to the Home Directory tab. The location is specified in ‘Local Path’.

Windows may also prevent you from editing these files directly due to permissions issues. I’ve found a good technique is to copy the files you want to edit to your desktop, edit them, and then copy them back into the Web Module’s physical path. Windows will then prompt you to elevate to administrator and the copy/replace will succeed.

Ready To Go…

There are a few places where the WebSpy logo and WebSpy Text is presented.

• The login page
• The header bar
• The welcome Page
• Report cover pages

The Login page

The logo displayed on the login page can be found at /images/logo.png. Replace this image with your own logo. Then open Default.aspx in the Web Module’s root folder in a text editor such as notepad, and replace the following line

<img runat=”server” alt=”WebSpy” src=”~/Images/Get.ashx?image=Logo” />

with

<img runat=”server” alt=”WebSpy” src=”~/Images/logo.png” />

Before

Web Module's Login Page Before logo.png Change

After

Web Module Login Page After logo.png change

The header bar

The header bar utilizes the image located a /Images/bauble.png. Replace this image with your own custom image.

Then open Navigation.Master in the Web Module’s root folder in a text editor such as notepad, and replace the following line

Also look for the text:

Before

Web Module's Header Bar Before Bauble.png and Text Changes

After

Web Module's Header Bar After Bauble.png and Text Change

The Welcome Page

When you first login to the Web Module, you are presented with a Welcome Page. The first line on this page reads “Welcome to the WebSpy Vantage Web Module. You can change this by editing the first line in the Welcome.aspx file located in the Web Module’s root folder. Edit the section in bold below:
<%@ Page Language=”C#” MasterPageFile=”~/Navigation.Master” AutoEventWireup=”true” CodeBehind=”Welcome.aspx.cs” Inherits=”WebSpy.Vantage.WebModule.Welcome” Title=”Insert Custom Text Here” %>

Before

Web Module's Welcome Page Before Text Change

After

Web Module's Welcome Page After Text Change

The Report Cover Pages

The Image used on the cover page of reports is much easier to change.

1. Login to the Web Module as Administrator
2. Go to the Options Tab
3. Click ‘Report Logo’ under Web Module Options
4. Click Choose File, and select the image or logo you would like displayed on your report cover page
5. Click Upload

Before

Web Module's Report Cover Page Before Report Logo Change

After

Web Module's Report Cover Page After Report Logo Change

Summary

The changes above cover a majority of the areas your users will come into contact with in the Web Module. There may be a few more instances of the word “WebSpy” but for the most part, it should just be a matter of opening the relevant .aspx file and editing the html.

As I mentioned, if you auto-update the Web Module (via the system tray icon on the Web Module server), your edited files will be overwritten. I recommend keeping a copy of your edited files in a safe place outside the Web Module’s physical folder, so that you can copy them back in after the update. If the only changes you make are the ones above, then you’ll need to keep a copy of:

• /Navigation.Master
• /Default.aspx
• /Welcome.aspx
• /Images/logo.png
• /Images/bauble.png

We will also be adding the functionality to make these changes ‘properly’ in a future build, so follow us on Twitter, or subscribe to our RSS feed for updates!

Cheers!

Unfortunately, the WebSpy Twitter account fell victim to a phishing scam, and as a result sent phishing spam to all our Twitter followers. We are embarrassed by the incident and we apologize to all of our followers, especially the ones that clicked the link in the DM and were caught by the phishing scam themselves.

Here’s a rundown of the event in the hope that it will help others know what to look out for.

What Happened?

The phishing scam works like this:

1. You receive a strange yet intriguing Direct Message from someone you follow and likely trust. This is the key element to the scams success.
2. The DM contains a link using a shortened URL such as dwarfurl.com/blah. In our case, most of them were using dwarfurl.com, wapurl.co.uk, and 3.ly
3. You click the link and get taken to what appears to be the Twitter login page. But if you look at the URL it is actually something like blogs.videos.dsfasdc.com or videos.twitter.dsfasdc.com. Checking the URL is the key to making sure the scam doesn’t get you too!
4. You enter your Twitter login details. Reports of what happens after this login page vary. You may see the Twitter fail whale, or a blank page, or a random blog.
5. Now that the phishing site has your login details, the same Direct Messages is sent to all your Twitter contacts.
6. You eventually discover what happened. You feel like a violated idiot and start scrambling to fix everything.

What to do if it happens to you

If the above sounds familiar, you need to login to Twitter right now and change your password to make sure the phishing site can no longer access your account. You also need to go to the Connections tab and disable any third party applications that look suspicious. You’ll then need to update the credentials in all the twitter clients, website/blog plug-ins, and anything else that may be using your old Twitter credentials.

Fortunately, we were still able to login to our Twitter account and change our password and disable third party connections. Thankfully there were not any new suspicious connections that we needed to worry about.

Lessons Learned

Now that we’ve fixed everything and regained control of our Twitter account, it’s good to sit back and reflect on what just happened and how to avoid it in the future.

You’ve probably heard all of this before. We had too. But it takes an incident like this to really think about and address any shortfalls in your own organization. Some of our followers were also caught out by the scam and these are people that are in the tech industry and generally know about these sorts of scams. We were definitely surprised that we fell for it! So take a moment of your time to imagine your own Twitter account was compromised in the same way, then imagine all the possible ways it could have happened. Now go and take every precaution to ensure it doesn’t happen.

Having now been through it, here are some tips to help you avoid the same fate in the future.

1. Just because a Direct Message comes from someone you trust, does not mean it is trustworthy. Always use caution!
2. Educate your employees – especially those that know your company’s Twitter credentials. The main goal you want to achieve here is getting your employees into the habit of glancing at the URL in the address bar of their browser before entering ANY login details. We used our own log analysis software (Vantage) to find out who ended up on the websites in question, and then spoke to them directly to ensure they understood what to look out for.
3. Use a Twitter application that can display the actual URL behind a shortened URL before clicking on the link. For TweetDeck users, go to Settings | General, and check ‘Show preview information for short URLs’. Please note, however that this function only works for a few specific URL shortening services.
4. If you’re using the Twitter web page directly, use a browser and plug-in that can expand shortened URLs such as Mozilla Firefox with Long URL Please.
5. Use a browser with integrated anti-phishing security (such as Firefox or Google Chrome) and keep it up to date, or ensure you have good third party anti-phishing / anti-malware software installed.
6. As always, keep your security software and OS up to date.

Our friends at Sophos also have some good information about the scam that you may like to read: http://www.sophos.com/blogs/sophoslabs/?p=7366

Sorry!

An event like this makes you realize how important Twitter is to the overall public perception of a company. Our followers trust us to deliver relevant and useful content about our key areas of expertise – log file analysis and reporting. We spend a large amount of effort researching and writing content to ensure our tweets provide our followers with a good source of information. Having a breach like this certainly degrades this public perception that we work so hard at trying to maintain.

I would therefore like to thank all our followers who have kept with us and not clicked the ‘Unfollow’ button. Now that everything is under control again we will continue to bring you the best content we can provide about the log analysis and surrounding industries.

Once again, many many apologies to all of our followers, especially those that were affected.

Supported on Windows 7

The following WebSpy products support Windows 7:

• WebSpy Analyzer (Standard, Premium and Giga)
  Analyzer customers will need to download the latest version of their product as support for Windows 7 was uploaded today (9th November 2009).
• WebSpy Vantage (Premium, Giga and Ultimate)
  All products within the Vantage range have officially supported Windows 7 since version 2.2 was released on  24th June 2009.
• WebSpy Insight for Microsoft SBS Premium
  Although Insight for Microsoft SBS Premium does not mention Windows 7 support in its documentation, it will install and run on Windows 7 without issue.

Not ‘Officially’ Supported on Windows 7

WebSpy Live, Sentinel and FlowMonitor are not yet officially supported on Windows 7 and will be updated soon. In the mean time, here are some instructions on how to get these products working on Windows 7 right now.

WebSpy Live

The existing version of Live will install and run on Windows 7 with one minor issue. When you run the application, there will be no Triggers, Aliases, or Profiles available.

To fix this:

1. Install and run WebSpy Live on Windows 7.
2. Shutdown WebSpy Live by right-clicking the Live icon in the Windows system tray and selecting ‘Shutdown’
   

   Shutting down WebSpy Live on Windows 7

3. Download the following file containing WebSpy Live’s default Triggers, Aliases and Profiles:
   WebSpy Live Default Files (zip – 19.7 KB).
4. Extract the zip file to C:\Users\<your user profile>\AppData\Roaming\WebSpy\Live 2.2 and overwrite the existing files.
5. Run WebSpy Live. You will now be able to see a list of Triggers, Profiles and Aliases on the respective Configuration screens in WebSpy Live.

WebSpy Sentinel

WebSpy Sentinel will not yet install on Windows 7 due to an issue with the included version of WinPCap (the packet driver used by Sentinel).

To fix this:

1. Download and install the latest version (4.1.1) of WinPCap from http://www.winpcap.org/install/default.htm
2. Install WebSpy Sentinel. The product should install and run without issue.

FlowMonitor

Unfortunately no work around is available for WebSpy FlowMonitor. We will update the product to work with Windows 7 as soon as possible.

Got a problem?

If you experience an issue running these products on Windows 7, or any other Windows operating system for that matter, please let us know!

Vantage’s Storage Location

If you’re using Vantage Premium or Giga, there’s only one location you need to be aware of. That is where Vantage keeps its storages (Vantage’s custom database that log files are imported into). This setting is easily changed by going to Tools | Options | Paths and double clicking the Storages path. Easy.

Web Module Storage Locations

If you’re using Vantage Ultimate, you also need to be aware of the Storage location mentioned above, but you also may need to adjust where the Web Module stores its data.  There are two locations you need to be aware of here:

1. The Web Module Data Location
   This is where the Web Module permanently keeps it’s storages, reports and settings
2. The Windows temporary folder
   This is where Vantage keeps storages while they’re being processed before uploading them to the Web Module’s data location

The Web Module Data Location

The data location for the Web Module is specified during installation and defaults to C:\Vantage Web Data. If you have already installed the Web Module, you can change this location using the following steps:

1. Find the Web Module’s Web.Config file. The Web.Config file can be found in the Web Module’s physical folder. If you don’t know where the Web Module’s physical folder is:
   1. Open Microsoft IIS (Control Panel | Administrative Tools | Internet Information Services (IIS) Manager)
   2. Select the Web Module in the left hand side (e.g. Server-> Sites ->Default Web Site -> webmodule).
      • If you’re using IIS 6, Right-click the Web Module site and select Properties then go to the Home Directory tab to find the physical folder.
      • If you’re using II7, select your Web Module site and click Basic Settings…
2. Open the Web.Config file in Notepad.
3. Find the line that looks like this:

   <add key="SettingsPath" value="C:\Vantage Web Data"/>

4. Change c:\Vantage Web Data to the location you would like to use and save the file.
5. In Windows Explorer, copy all files and folders from C:\Vantage Web Data (or where ever your original location was) to the new location you specified in step 4
6. Restart IIS by going to Start | Run and type iisreset /restart

The Windows Temporary Folder

The Windows Temporary folder can also be modified, but please note this is a system wide change.

1. Right-click ‘My computer’ and select Properties.
2. Go to Advanced and Click the Environment Variables button
3. Change the location for the ‘TEMP’ and ‘TMP’ environment variables (do not use the same location specified in step 4 above)

Vantage and the Vantage Web Module (Ultimate only) will now use your new locations to temporarily and permanently keep your Storage files.

If you’re considering upgrading your ISA Server to TMG, this means that you can start your deployment using the Release Candidate, and simply switch it to a licensed version with no additional configuration changes once the full release is available. At least, that is what Vladimir Holostov (Lead Program Manager, Release Manager for Forefront TMG 2010) states on the Forefront TMG (ISA Server) Product Team Blog:

“The final product will be released later this year and you can expect it to behave exactly like the Release Candidate. You can install Forefront TMG 2010 RC today and upgrade to a licensed version once available without changing the configuration of your deployment.”

To offer some peace of mind for organizations considering the deployment, Vladimir also mentions that “Forefront TMG 2010 RC is deployed at three major Microsoft sites located around the world in Haifa, Bellevue and Redmond. More than 20,000 employees are already protected by TMG and these deployments have already accumulated more than 5,000 hours of runtime, performing extremely well under heavy load”.

No major features have been added to the Release Candidate since Beta 3, however there have been improvements geared around tightening up security, reliability and performance and telemetry. For more information about the release candidate, please visit the
Forefront TMG (ISA Server) Product Team Blog.

You can also download the release candidate here

I mentioned in my last blog posting that WebSpy has introduced support for reporting on Microsoft Forefront TMG log formats in the Vantage product range. To try it out, please make sure you have installed Vantage 2.2 (any flavour – Premium, Giga or Ultimate), and then select Tools | Check for updates to download build 2.2.0.10 or above. You can then import your TMG log files by selecting the Microsoft FTMG loader in the import wizard.

Importing Microsoft Forefront Threat Management Gateway Log Files

We’re very interested to hear your thoughts on the reporting functionality, so please go ahead and give it a go!

You should be prompted to update your software on startup, but if you’ve turned off that feature, simply go to Tools | Check for Updates.

New Features

This new build sports the following new features:

• Support for Microsoft Forefront Threat Management Gateway (Beta)
  Microsoft Forefront Threat Management Gateway (FTMG) is still currently in Beta, and is due to be released around November 2009. For those that do not know, FTMG is the next version of Microsoft’s popular ISA Server. Information and downloads for FTMG can be found here http://www.microsoft.com/forefront/edgesecurity/isaserver/en/us/tmg-beta.aspx. We have added support for FTMG beta 2 and 3 for both the W3C text logs (recommended) and the internal SQL Server Express Database logs. If you are currently trialling FTMG, we are very interested to hear your feedback. Let us know how you go!
  

  Now Supported - Microsoft Forefront Threat Management Gateway

• Data purge
  You can now purge data from a storage, and schedule this purge to occur on a regular basis using Tasks. Purge options include data between a date range, data before a date, data after a date, data older than a date relative to now, and all data. This feature will let you easily maintain a single storage that only includes data for the last month or day.
  

  Options for Purging data from your storage

• Import Organization from CSV can now be scheduled using Tasks
  If you are importing your organizational structure from CSV, you can now schedule this action using Tasks. This enables you to update your organizational structure before any reports are run.
  

  Import Organization from CSV via Tasks

• Added Support for ExoServer Web
  If you’re running ExoServer Web, you can now analyze it’s logs using WebSpy Vantage.

Fixes

We also fixed some things that may have been bugging you:

• Improved the start time for the application by improving the logic to check for Storage damage.
• Fixed the IronPort loader (Fixed out of range issues on excessive size fields).
• “Having” filters no longer override the sort order of a Report Template node.
• Fixed an issue that may result in duplicated storages after migrating settings from earlier versions.
• Fixed the inability to remove invalid entities from web module permissions list (users that no longer exist).
• Fixed a timeout issue when publishing storages to the web module.

Why are you still reading? Go update now!

Have fun!

To use Windows Authentication, there are just a few things you need to do.

1. Set the Web Module’s Authentication type to IIS Integrated, and add your administrators in the form of domain\username
2. Enable Windows Authentication and disable Anonymous authentication in IIS.
3. Ensure all users in your Organization screen have a login name in the form of domain\username. Use the ‘Prefix’ option to prefix “domain\” (without the quotes) to your usernames names when importing your Organization from LDAP or LDIF.
4. Connect Vantage and the Web Module using the new authentication details and synchronize your Organization.

1. Set the Web Module’s Authentication type to IIS Integrated, and add your Administrators.

When you first install the Web Module, the first screen you see is the ‘Initial Configuration Wizard’ that guides you through the process of selecting your authentication type and specifying your administrator(s). If you have already been through this Wizard and are currently using Vantage In-Built or Client Certificate authentication, you can easily reset this initial configuration wizard. Simply login to the Web Module with your current administrator details and go to Options | Maintenance | Reset Initial Configuration Wizard.

Note: You can also change your authentication and administrator options individually using the Authentication and Administrator options on the Options tab of the Web Module.  However, for ease of demonstration, I’ll use the Initial Configuration Wizard method.

Now that you’re at the Initial Configuration Wizard, proceed through the wizard, selecting IIS Integrated authentication and entering your administrators in the form of domain\username (replace domain with your organization’s AD domain, and username with the sAMAccountName of your administrator.

Initial Configuration Wizard - Welcome Page

Initial Configuration Wizard - Authentication Page

Initial Configuration Wizard - Delegate Administrators Page

Initial Configuration Wizard - Summary Page

Click Finish, and if the authentication was successfully changed, you should get a message saying ‘The specified credentials were not accepted”.

The 'Specified credentials were not accepted' message.

Don’t panic at this point. This message is an indication that the authentication was successfully changed and that the Web Module is now listening for IIS to pass through Windows Usernames. The reason you’re getting this message is because IIS is not yet passing through Windows Usernames to the Web Module. This is configured in the next step.

2. Enable Windows Authentication and disable Anonymous authentication in IIS.

Now that the Web Module is expecting IIS to authenticate your users, you need to set up  IIS  to do this.

1. Open IIS by navigating to Start | Control Panel | Administrative Tools and double-clicking on Internet Information Services (IIS) Manager.
2. Navigate to the Web Module site or virtual directory in the left hand ‘Connections’ Panel. It will be located under <Server Name>\<Sites>. For example, MyServer->Sites->Default Web Site->webmodule.

• If you’re running IIS7 ( Windows Server 2008, Vista or Windows 7)
  1. Select the Web Module site and ensure the ‘Features’ tab is selected at the bottom of the middle pane.
  2. Double-click the ‘Authentication’ feature.
  3. Right-click ‘Anonymous Authentication’ and select Disable
  4. Right-click and ‘Windows Authentication’ and select Enable
  5. Restart IIS by selecting your server in the right hand connections pane, and clicking Restart in the ‘Actions’ pane on the right.

• If you’re running IIS6 or 5.1 (Windows Server 2003, Windows XP)
  1. Right-click Web Module site and select Properties.
  2. Go to the Directory Security tab
  3. Under ‘Authentication and access control’ click the Edit button.
  4. Uncheck ‘Enable anonymous access’ and check ‘Integrated Windows authentication’
  5. Restart IIS by right-clicking the local server, select All Tasks, and then click Restart IIS.

If you added your own Windows login name as an administrator in step 1, you can now test the authentication is working. Go back to the Web Module in your browser and click Refresh. You will be presented with an ‘Authentication Required’ dialog where you can enter your username and password.

Authentication Required Dialog

Again, ensure your username is in the form of domain\username. Click OK, and you should log straight into the Web Module using Windows Authentication.

3. Ensure all users in your Organization screen have a login name in the form of domain\username

Now your administrator account can log into the Web Module using Windows Authentication, but all other users will not be able to log in unless they have their login name specified in the form of domain\username. This is done in Vantage Ultimate on the Organization screen.

Organization Screen showing correct login name for Windows Authentication

If you’re importing your users from LDAP or LDIF, make sure you use the ‘Prefix’ option on the User Details page to prefix domain\ before your imported usernames. For example:

4. Connect Vantage and the Web Module using the new authentication details, and synchronize your Organization.

In order to publish information to the Web Module, you need to add a connection between Vantage and the Web Module. This is done on the Web Module screen in Vantage Ultimate.

1. Click Add Web Module (or if you already had a web module before changing the authentication details, select it and click Properties)
2. Enter the server & virtual directory of the Web module, and enter the correct credentials ensuring domain is specified.
3. Click OK to connect.

Connect to Web Module dialog

Once connected, synchronize Vantage with the Web Module by clicking the Synchronize link in the Web Module task pad. You may also want to provide permissions for your users in the Permissions section on the Web Module screen.

That’s it. You can now test that everything is working by getting one of your users to access the Web Module’s URL. They should sail straight in with no username/password prompt.

I hope this helps! Please let me know your feedback by emailing me, or leaving a comment below.

If you’re using WebSpy Vantage, you are probably interested in filtering your log file imports by date (only import files from the month of June for example). The obvious way to do this is to specify a date filter using the filters page in the Input Wizard. The problem is Vantage will still check every record in every log file being imported to see if it matches the date filter. If you have months or years worth of logs in the folder being imported, that’s a lot of data that Vantage has to pointlessly sift through.

The good news is, if your log files contain the date in their file name, then you can use file masks to instruct Vantage to never touch these unwanted files.

A bit about file masks…

You can specify file masks such as *, *.log, *.gzip, *WEB*.w3c, etc to import logs with specific file extensions, or with specific strings in the file name (such as WEB or FWS to import only Microsoft ISA Web Proxy or Firewall logs respectively).

But if your log file contains the date in the file name, you can also use date modifiers in the file mask to select logs from a particular month, date or year.

Say you have log files that look like this:

• 20090801.log

• 20090802.log

• 20090803.log

and so on..

You can create a simple file mask to only import log files from the month of August very easily using 200908*, or 200908*.log.

Using date modifiers in file masks

But if you’re using Tasks to automatically create a new storage each month, you don’t want to have to worry about manually changing the file mask to 200909*.log when the first day of the next month rolls around.

So intsead, you can use a date modifier in the file mask that will automatically select the logs for the current month, every time your task runs. For the above example, the file mask looks like this:

• %[yyyyMM]* (you can also use %[yyyyMM]*.log)

When the task runs, %[yyyyMM] will be replaced with actual values from the current date. So if the task runs on the 1st of August 2009, the file mask will become 200909* (or 200909*.log).

Dealing with different date formats

You can also use date modifiers for log files that look like this:

• 2009-Aug-01.log

• 2009-Aug-02.log

• 2009-Aug-03.log

In this case the file mask looks like:

• %[yyyy-MMM]* - notice the three MMM's as opposed to two MM's used previously.

Vantage uses the custom date and time format strings available in the .NET framework, so for more information on whether to use m or M or MMM, please refer to this article http://msdn.microsoft.com/en-us/library/8kb3ddd4.aspx

Importing logs from previous months

If you would like to import logs from a previous month, this can also be done by adding an additional element to the date modifier. For example, to import the previous months logs you can use:

• %[-1m,yyyyMM]*

Notice the -1m meaning ‘minus one month’. You can also use -1d (for minus one day), or -1y (for minus one year).

More examples

Here are some more examples to give you an idea of what is possible using date modifiers.  Assuming the date is 14th of August 2009:

• %[-1y,yyyyMM]*.log will create a file mask of 200808*.log

• %[yyyy-MM-dd]*.log will create a file mask of 2009-08-14*.log

• %[-4d,yyyyMMdd]*.log will create a file mask of 20090810*.log

• %[1-m,-4d,yyyyMMdd]*.log will create a file mask of 20090710*.log

• %[-1y,1-m,-4d,yyyyMMdd]*.log will create a file mask of 20080710*.log

• ISALOG_%[-1m,yyyyMM]*_WEB_*.w3c will create a file mask of  ISALOG_200907*_WEB_*.w3c

• *%[-1m,yyyyMM]* will create a file mask of  *200907*

Adding a file mask

File masks are configured on the Input Selection page of the Input Wizard, when you select Add | Folder.

Adding a File Mask

There is also an option to save the literal date into the file mask when the task is run.  For more information on this option, please see my previous blog about this feature.

Other uses for date modifiers

Date Modifiers are a great way to speed up log file imports, but you can also use them when specifying storage names as well as report names. For example, if you specify a storage name of %[yyyyMM]_storage, this will create storages with the names 200907_storage, 200908_storage and so on. When selecting the storages to report on, you can click the Add button on the storage selection toolbar in the Report Wizard, and specify storages such as %[-1m,yyyyMM]_storage, to report on the previous month’s storage.  For more information, please see Automatic Importing and Reporting using Tasks.

I hope this helps someone out there. Let me know how you go!

We’ve got a great deal at the moment where you get 20% off Live and Sentinel if you purchase them online with Analyzer Standard.

If you convert your logs to text using this script, they won’t import into WebSpy Vantage or Analyzer due to an extra line break in the header of the file (after #fields:).

We’ve therefore created a modified version of the script that creates compatible log files for WebSpy software.

Download the modified MSDEToText script:
MSDEToText.zip -26 KB

Also make sure the file names of your output log files contain the word WEB (for Web Proxy logs) or FWS (for Firewall Logs) as Analyzer and Vantage use these strings to automatically detect the type of ISA log file.

Happy converting!

Watching a single video on YouTube will probably generate a list of about three to five sites such as lax-v41.lax.youtube.com, www.youtube.com, img.youtube.com, and so on. Your list of top sites also probably contains hits to ad servers and tracking servers, such as doubleclick.net, google-analytics.com and imrworldwide.com. All this clutter gets in the way of determining what sites were ‘intentionally’ visited.

Fortunately there are a few simple steps you can take to exclude this information from your reports. Watching is much easier than reading, so I thought I’d create a video demo to walk you through the process.

By the way, this is the first video demo of what I hope will be many more to come. I created it using TechSmith’s Camtasia Studio which is by far the best screen recording software I’ve used. All the zooming you see throughout the demonstration is completely auto-magical! It’s a brilliant piece of software that has saved me hours of time. Props to the guys at TechSmith! The one pitfall of Camtasia is that it seems to make me sound like a geek with a raw Aussie accent… I hope they fix that in the next version.

Anyway, I hope you find this useful.

All three products in the Vantage range (Premium, Giga and Ultimate) have been updated with the following features:

• Multi-processing
  Vantage now utilizes the extra processing power on machines with multiple cores or CPUs to import log files and generate reports faster.
  
  

  The new Performance tab in Vantage

• Improved report styles
  HTML (including MHT) reports are now sporting an updated look and feel.
  
  

  New Report Style

• Updated Aliases and Profiles
  Added support for the latest search engines, social networking sites, operating systems and user agents.
  
  

  Profiles Aliases support the latest Internet tools and technologies.

• Storage Repair Utility
  Storages can become damaged if there is a system crash or if WebSpy.Vantage.exe process is ended when an import is in progress. Damaged storages are now detected automatically and you have the option of repairing them. Storages can also be manually checked and repaired on the Storage properties dialog.
  
  

  Storage Repair Utility

• Save / Open Tasks
  Scheduled Tasks can now be saved to an external file (.Tasks) for backup and migration purposes.
  
  

  Save and Open Tasks

Obviously, the multi-processing feature is the big one! Many of you (Vantage customers) have been running Vantage on multi-core or multi-cpu machines and just dying for Vantage to grab hold of the CPUs and make them work. You should be happy with this new build. Import a folder full of log files and watch 6 to 8 logs import simultaneously instead of watching them import one after the other. Run a full analysis or a comprehensive report with no filters and watch your CPU jump into gear to generate the report in 30% – 50% of the time. We’re expecting this to be a very popular update, especially among our larger customers.

Multi-processing is a great new feature, but be aware that there are some circumstances where Vantage will not be able to utilise more CPU power when generating reports. For example, if your report contains filters that significantly cuts down the amount of data in your storage that needs analysis, Vantage’s analysis engine and your CPUs will not be significantly pushed. Also see my earlier blog for more information on how the report structure can also affect the performance.

To get hold of this build, simply download the latest version of Vantage from our website. It will run side by side with your existing Vantage 2.1 installation, which minimizes any downtime, and makes it easy to transfer all your settings across (use the Migration Wizard in the Tools menu). Then you can uninstall Vantage 2.1 when you’re ready.

Let us know how you go by leaving comments below, or by tweeting us.

Cheers!
Scott

Well, it’s now been released and can be downloaded from the Microsoft Download Center:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=e05aecbc-d0eb-4e0f-a5db-8f236995bccd

One of the major improvements in Beta 3 is URL filtering which leverages Microsoft Reputation Services (MRS). With regards to this, Microsoft says:

“At the time of this release, the MRS database content is being populated and updated continuously as part of the initial beta service offering. As this process continues, URL filtering categorization accuracy and comprehensiveness will increase. A telemetry package designed for improving the quality of URL filtering database and collecting your feedback is planned to be released soon. Please check back for updates in August.”

Support for importing Microsoft TMG log files into your favorite WebSpy product is coming soon so stay tuned! Subscribe to the WebSpy blog RSS feed or follow us on twitter if you want to be notified as soon as it’s available.

According to the latest blog from TMG’s Product Unit Manager, David B. Cross, Beta 3 will be released in the next couple of weeks. As for the full release, David says that they are still on track for Q4 this calendar year.

Beta 3 will introduce URL filtering that is ‘fully integrated’ with TMG’s web policy rules, and also utilizes Microsoft Reputation Services.

Microsoft are also introducing Intrusion Prevention and Detection (IPS/IDS) capabilities in TMG. These systems will utilize a technology they’re calling Network Inspection System (NIS) that detects attacks using signatures of known vulnerabilities, downloaded from the Microsoft Malware Protection Center. For more information on NIS see http://blogs.technet.com/isablog/archive/2009/04/12/exercising-nis-with-test-signature.aspx

If you’re currently using ISA 2004 or 2006, upgrading to TMG will consist of exporting rules and settings from ISA, then importing them into a clean installation of TMG. TMG will also only run on Windows Server 2008.

Improving the on-box reporting has not been a focus for the TMG development team, so analyzing TMG’s web proxy and firewall logs is still the best way to go for in depth reporting.

If you’re interested in reporting on your TMG log files stay tuned! We’re currently implementing support for the SQL Express, W3C and Native text logs. WebSpy Vantage is likely to be the first application to include the feature, with Analyzer and Live soon to follow.

All going well, you can expect to see TMG support in your favourite WebSpy app within the next month or so. If you want to be notified once we’ve added support, just leave a comment below.

Cheers!
Scott.

• Windows Vista & Server 2008:
  C:\Users\<user profile>\AppData\Roaming\WebSpy\Vantage <edition> <version>
• Windows XP & Server 2003:
  C:\Documents and Settings\<user profile>\Application Data\WebSpy\Vantage <edition> <version>

When Vantage opens, it loads the information contained in these files.

To move all your settings across, you can simply copy all the files in this location on your original machine, into the same folder on the new machine (make sure Vantage is closed when you do this).

When you open Vantage on the new machine and you’ll have all your settings moved across.

If you’re using scheduled tasks, one thing to note is that copying these files will not recreate the Windows scheduled task jobs. To get your tasks functioning on the new machine:

• Go to the Tasks tab and double-click each scheduled task.
• Proceed through the wizard and make sure all the settings are correct.
• Enter your authentication details on the last page and click OK.

This will create a new Windows scheduled task job which should run as it did on the old machine.

Please note that this process only moves ’settings’ across to the new machine. If Storages and Reports were being written to a location on the old machine, then you will need to move these across to the Storages and Reports locations on the new machine (check Tools | Options | Paths for these locations).

Also take access privileges into account. If you’re old installation was set to create storages in \\server\mystorages, make sure the new machine also has write privileges to this location.

That’s pretty much all there is too it. Happy migrating!

The multi-processing build features a new tab called ‘Performance’ in Tools | Options. Ensure ‘Use multi-processing’ is checked, and set the ‘Maximum Concurrent Threads’ value to twice the number of logical/physical CPUs you have. For example, set it to 4 on a dual core, 8 on a quad-core, or 16 on a machine with two quad-core CPUs. Feel free to also play around with this value, but we have found this formula to be optimal and we’ll automatically set this as default in future builds.

The new Performance tab in Vantage

You’ll notice that if you import a folder of logs, about 6-8 logs will import simultaneously. Simply importing one log file will not show any speed improvement.

Multi-processing will only benefit report generation if your report template contains side-by-side summaries as opposed to drilldown summaries. Here’s what I mean by a template with side-by-side summaries:

A report template that only has side-by-side summaries

And here’s what I mean by a template with drilldown summaries

A report template with only drilldown summaries

Basically, each side-by-side summary will get processed simultaneously, but any drilldowns will be processed sequentially.

Most report templates consist of both side-by-side AND drilldown summaries, such as:

A report template with both side-by-side and drilldown summaries

The amount that these templates benefit from multi-processing will depend on the number of side-by-side summaries.

If this sounds like something you’d like to try, please contact me for a copy of the build!

Of particular note is a fix to the Vantage range:

• Fix: Improved partition matching when filtering by a date range

This will benefit you if you’re running reports with a date range filter such as last week or yesterday (which is pretty much everyone), and are using the default storage partitioning scheme of “Date” (again… pretty much everyone).

There was an issue in Vantage’s partition filtering technology which meant that if you specified a date range filter, Vantage would still analyse an entire storage, instead of just the date partitions you selected in the filter.

So do a Tools | Check for updates to grab build 2.1.2.12 and you should see a significant speed improvement in report generation. Woo hoo!

Here’s more details on the changes:

VANTAGE

Application Changes

• New: Added support for importing event logs from non-domain machines.
• New: Added the ability for the Troubleshoot Alias/Profiles to export results to a file.
• New: Added hash salt and substring functions to the expression language.
• New: Added support for computing a date-modified file mask at import time and storing it back into the storage.
• New: Added a file mask override to the Import new hits to existing storage task action.
• New (Vantage Ulitmate & Giga): Added support for attributes on Organization groups.
• New (Vantage Ultimate): Added custom expression-based split in Web Module publishing (Beta).
• New (Vantage Ultimate): Changed the publish report wizard to allow selection of multiple storages.
• Fix: Improved partition matching when filtering by a date range.
• Fix: Fixed an issue with registrations and trial extensions.
• Fix: Fixed zero-width rectangle exceptions in chart renderer.
• Fix: Fixed overflow exceptions on footer generation in report tables.
• Fix: Fixed XmlException error during settings load.
• Fix: Storage now clears previous data when it is overwritten when using the ‘Import logs into new storage’ task.
• Fix (Vantage Ultimate & Giga): Fixed issues with date range selection when collating reports.
• Fix (Vantage Ultimate): Web Module dock no longer checks for updates to a web module server when ‘Check for updates on startup’ is disabled.

Loader Changes

• New: Added Phion Firewall
• New: Added Watchguard Firebox X Core
• New: Added NetScreen 208
• New: Added IPCop
• New: Added Astaro Mail Gateway
• New: Added iPrism Monitor v4.2
• New: Added Cisco VPN Concentrator
• New: Added Snare for Lotus Notes
• Improved: Added string host name to Kerio Mail Server
• Improved: Improved support for NetIntact PacketLogic
• Fix: Made changes to the IronPort detection method to fix the issue when importing via FTP.
• Fixed: Fixed date format issue in event log import. You can now import event logs in any regional configuration.
• Fix: Postfix loader no longer drops session state after the first recipient line
• Fix: Updated the detection method in CheckPoint Firewall-1 Syslog format.
• Fix: Fixed an issue in iSheriff where a drilldown on the Category field would display no results.
• Fix: Various fixes to Netscreen 10
• Fix: Various fixes to Arkoon PxLog
• Fix: Vaious fixes to Sendmail MTA

VANTAGE WEB MODULE

• FIX: When sorting by key column the chart will now use the “hits” aggregate for value (prevents the chart from going wacky)

ANALYZER

Application Changes

• NEW: Added support for manual configuration of ISA block list settings (Tools | Options | Block Lists)
• Fix: Fixed an issue with registrations and trial extensions
• Fix: Fixed report wizard configuration to restore the sort column from the report template

Loader Changes

• New: Added Webroot
• New: Added Qmail Desknow Mail Server
• New: Added UserGate Proxy Server 2.7
• New: Added Netgear FVX538
• Improved: Added support for importing allowed/blocked status in Sophos Web format
• Fixed: Changed CC Proxy field count check
• Fixed: Changed ISA SQL loaders to use size delta fields instead of cumulative fields
• Fixed: Modified Exchange 2000/2003 loader to discard message cache after import

LIVE

Application changes

• Fix: Fixed an issue with registrations and trial extensions

Loader changes:

• New: Added Webroot
• New: Added Qmail Desknow Mail Server
• New: Added UserGate Proxy Server 2.7
• New: Added Netgear FVX538
• Improved: Added support for importing allowed/blocked status in Sophos Web format
• Fixed: Changed CC Proxy field count check
• Fixed: Modified Exchange 2000/2003 loader to discard message cache after import

I went hunting through the options in the Virtual PC UI but didn’t find anything related to disabling this option. A bit of googling later and I’ve got it disabled by adding a few lines of XML to the .vmc file.

First, make sure your VM is shut down and Virtual PC is closed.

Then find your .vmc file and open it in a text editor such as Notepad.  By default, Virtual PC creates .vmc files in My Documents\My Virtual Machines.

Find the </microsoft> tag and insert the following lines directly above it:

<components>
<host_time_sync>
<enabled type="boolean">false</enabled>
</host_time_sync>
</components>

For example:

Disabling the host time synchoption option in a .vmc file

Then Open Virtual PC, start your VM and you’re all done!

This was tested on Windows Vista 6.0.6001, SP1 using Microsoft Virtual PC 2007 (6.0.156.0)

The fix can be applied to WebSpy Analyzer Giga 2.3, Analyzer Premium 4.3 or Analyzer Standard 4.3. If you’re not running the latest version, download it now!

You can download the new loader build that we created today at either of these locations:
USA West Coast (FTP)
USA East Coast (FTP)

Then extract the zip file into Analyzer’s installation folder (usually C:\Program Files\WebSpy\Analyzer flavour 4.3\) and overwrite the existing file.

Then go to the storages screen and select your Sophos storage(s) and click ‘Reload all hits’. This will re-import your log files using the modified loader and will populated the ‘Blocked’ summary appropriately. To check it out, go to the Summaries screen and run a Full Analysis. Then go to the ‘Blocked’ summary and you should see two items – ‘Blocked’ and ‘Not Blocked’. Drilldown into whichever one you care about to analyze the sites, users, files, browsing times, size downloaded etc. Go nuts!

You can also filter out blocked hits (or Not Blocked hits) from your reports. On the Reports Screen, click Generate a new report and go through the report wizard with this filter (this example shows filtering out blocked hits).

Analyzer Report Wizard - Select Custom Filters

Analyzer Report Wizard - Selecting the 'Blocked' Summary as a Filter

Analyzer Report Wizard - Adding the items that you want to filter

Analyzer Report Wizard - final filter to exclude 'Blocked' hits

Then proceed through the report wizard to generate your report. This filter can be applied to any report as well as analyses on the Summaries screen (using the same options in the Analysis Wizard).

Happy analyzing!

Unfortunately, Exchange 2007 tracking logs are not used to simple questions, and are likely to return a complicated and / or misleading answer.

But the confusion it seems, all comes down to definitions. Once you understand these definintions, things start to make a bit more sense.

What is an Email?

If you send an email to one person, you’ve sent one email. But if you’ve sent that same email to 500 people, have you sent one email, or 500? I will take a guess, and say that a large majority of you will want to see 500 in your reports.

Microsoft Exchange 2007 tracking logs contain an excellent field called ‘Message ID’. If you send an email to someone, that message is uniquely identified by a Message ID that persists though Exchange’s various functions for the lifetime of the message.

At first glance, it seems that counting Message IDs will give us what we want. But if you send the same email to 500 recipients, all those emails get the same unique message ID. So counting message IDs will show us that only one email has been sent. No good.

Then next obvious step is to count the number of recipients that received the email.

What is a Recipient?

The definition of recipient can also get clouded when you start talking about distribution lists. If you send an email to one real person, then that is one recipient. If you send the same email to five real people then that is five recipients. If you send an email to an internal distribution list, the number of recipients is the number of people that are members of that distribution list.

If you send an email to an external distribution list (such as SalesDL@othercompany.com) this will only be recorded as only one recipient, as your Exchange box has no way of knowing how many real people are members of that DL at the other company.

How do I count Recipients?

Again, Exchange Tracking logs contain another excellent field called ‘Recipient Count’. But don’t get carried away as this too can be misleading.

Without going into specifics, Exchange has a bunch of internal functions to deal with an entire message transmission. The tracking logs files contain another excellent field called Internal Message ID that identifies each of these processes per-message.

Unfortunately, each Internal Message ID contains its own value for ‘Recipient Count’. So when you sum the Recipient Count field for a single message, the final result may be much larger than the actual number of real recipients.

To illustrate, WebSpy Vantage imports Recipient Count into a Summary of the same name. Here is a screenshot of the Recipient Count Summary for one message

The Recipient Count Summary for a Single Message

As you can see, there are multiple rows of individual Recipient Counts. The first row, is actually correct. This email was actually sent to 961 people. But there are additional entries where Exchange performed an internal operation with a subset of those messages. Therefore, summing the Recipient Count field for a message is also no good.

Counting recipients “properly”

The best way to count recipients is to use WebSpy Vantage to import your logs, then drilldown into a message to the Recipients summary and look at the total number of recipients at the bottom. Alternatively, add a Count Distinct aggregate for the Recipients summary to any report template.

Here’s a screenshot of the Recipients summary:

The Recipients Summary showing Total 'Real' Recipients

And here’s a screenshot showing how to add the aggregate to a report template:

Adding the Number of Recipients to a report template

Counting Total Number of Emails

The above screenshot will give you a count of all the recipients you have ever sent email to. However, what you really want is a count of recipients per message. You can do this by concatenating the Recipient with the Message ID, and counting the total number of rows. To do this, edit the Number of recipients aggregate column above and enter [Recipient] + [MessageID] in the ‘Custom’ edit box.

Customizing an aggregate column to concatenate Recipient and MessageID

Exchange 2007 Report Templates

You can download a WebSpy Vantage Templates file here that includes three reports (Email Overview, User Email Activity, and Email Trends) that uses columns such as Number of Emails, Number of Unique Messages and Number of Recipients.

Tip: You can convert any email template that has the schema ‘All Mail Schemas’ into an Exchange 2007 template in order to report and filter using all the fields available in Exchange 2007.

To do this:

1. Right click an ‘All Mail Schema’ email template and select Duplicate.
2. Select Microsoft Exchange 2007 from the schema drop down and click OK.
3. When you edit the nodes in your new template, you will have access to all the fields that Exchange 2007 records.

Cheers!

A common setup is to have a task that creates a storage each month, then a separate task that runs at the end of each day – say at 10pm, to imports new hits into the storage.  The issue here is that at the end of the month, hits between 10pm and 12pm will never be imported, as the next time the task runs it will be dealing with next month’s storage.

As long as your log files contain the date in the file name, you can use date modifiers in file masks to import the appropriate log files. Date Modifiers are used to create file masks such as ‘access_200902*.log’, or ‘access_20090203.log’ that will cull down the number of log files to import. This is much faster than using date filters (specified on the filters page).

The Issue:

The issue is that a log file location in a storage can only have one file mask, and this cannot (until now) not be modified by task actions.  So you cannot specify a new task on the first of each month to import yesterday’s log file into last month’s storage, if that storage already has a file mask to only import the current day’s log file.

The two solutions:

1. Save the literal file mask into the storage when the storage gets created.

This option allows you to specify a file mask such as  access_%[yyyyMM]* in the ‘Import logs into new storage’ task, but when each storage is created, the actual file mask that gets saved into the storage includes the literal month and year values such as access_200902*

This way you can specify an ‘Import new hits’ job, and even if that job runs in the next month,  ONLY log files from the desired month will be checked for new hits.

To access this option:

1. Go to the Tasks screen and edit your ‘Import logs into new storage’ action
2. Go to the Input Selection page
3. Select your log folder (not individual files) and click Edit
4. Check the ‘Save literal date into mask’ check box


Save literal date into file mask option

Downside of this option is that in my ‘monthly storage’ example, all 31 log files for the month will be checked for new hits.

2. Override a storage’s file mask using the Import New Hits task action.

The Select Storage dialog that appears when configuring an ‘Import new hits into existing storage’ action now has a new option to override a storage’s file mask when the task runs.



Override the file mask option

This allows you to configure a task that runs on the first day of each month to import only yesterday’s log file into last month’s storage (need to add last month’s storage using the Add button, and specify the appropriate date modifier to access it – e.g. “%[-1m, yyyyMM] My Monthly Storage”).

This will only check one file for new hits and will finish the import job faster than option 1 (using the monthly storage example).

So here it is (you might want to grab a coffee first!).

Unless you have a sophisticated end point security or file auditing solution in place, you’re pretty much limited to the quality of data found in your Windows Security Event log. By default, accesses to your confidential files are not going to trigger any entries to be written to the Event log. You first need to setup file or folder auditing.

WebSpy have written a nice article to help you out with this: Managing Event Logs

Personally, I’m running Windows Vista SP1.  So I first turned on Object Access auditing by going to Control Panel | Administrative Tools | Local Security Policy | Local Policy | Audit Policy and set Audit Object Access for Success and Failure.

Windows Vista Local Security Policy

In Windows Explorer, navigate to the folder or files to audit, then Right-click | Properties | Security | Advanced | Auditing and click Continue when Vista’s User Access Control gets in the way.  Here you get the option to add Users or Groups to the audit policy. So if you only want to know when Joe Bloggs access the file/folder, then only add Joe Bloggs. If you want to know when anyone accesses the file/folder then add your entire company.

Scroll….

Click OK and apply the changes. If applying this to a folder, take note of the setting to ‘apply the auditing entries to containers within this container’ at the bottom and use as required.

Congratulations. That’s the auditing setup. Once people start accessing these files(s), the auditing information will get recorded to the Security Event Log on the machine that hosts the file(s) in question.

The next step is to import the Windows Security log into your flavour of WebSpy Vantage. I’m using Vantage Ultimate, but the steps are the same for Premium and Giga.

1. Run Vantage (as Administrator if on Vista)
2. Go to the Storages tab and click Import Logs
3. Run through the Import Wizard with these settings:

• Storage: New storage
  

  Input Dialog: Storages Page

• Input Type: Windows Event Log
  

  Input Dialog: Input Type Page

• Loader Selection: Microsoft


Input Dialog: Loader Selection

• Input Selection: Add
  Select either local computer, or multiple computers, enter authentication details and Click ‘Filter Event Logs’. Check the ‘Security’ Log and click OK.
  

  Input Dialog: Input Selection Page - Adding Event Logs

• Click OK to start the import.

If there are any issues with the import process, consult these three WebSpy Knowledgebase articles to do with issues importing event logs:

• Event Log Troubleshooting (Known Issues and Fixes)
• Importing Event Logs from machines on a different domain
• Required Services for Event Log Importing

The first article came in handy for me as I’m running on Vista and in order to import from the Local Security log, you need to run Vantage as Administrator. To do this, go to C:\Program Files\WebSpy\Vantage Ultimate 2.1\ right-click the WebSpy.Vantage.exe and select ‘Run as Administrator’.

Once data has been imported into your storage, check it out on the Summaries screen.

To to the Summaries Tab, Run an Analysis on your new storage (ad-hoc analysis will do) , and go to the Category Summary. There should be some ‘File System’ items there assuming the file has been accessed since setting up file auditing. You can then drilldown to Event Type to see ‘Audit Success’ or ‘Audit Failure’. To see who has Successfully accessed a certain file, drilldown into the ‘Audit Success’ item.

Unfortunately the good stuff is buried in the ‘Message’ field, which you can only access in the Individual Records view. This is because the Message field in Event logs is free form and could vary wildly resulting in millions of unique items. A Message Summary has therefore been excluded from a default ad-hoc analysis for very good performance reasons.

Event logs can also be quite verbose, and if you drilldown to Individual Records at this stage, you’ll see lots of messages like ‘A handle to an object was requested’ which probably isn’t of any great value from a reporting perspective. One way to filter out this noise is by Event ID.

I’ve discovered that the events that correspond to ‘An attempt was made to access an object’ have the ID 4663. (One day I’ll create an alias to map Event IDs to their meaningful description. If you come across a good  resource I can use for this, let me know!).  So go to the Event ID summary and drilldown into 4463 to the Individual Records view.

Once you’re at Individual Records, you can hover over the message field to get details. You can also use the find edit box to search for a particular user or file:

Drilldown into Successful File System Accesses (Event ID 4663)

You can export this view To Word Document, HTML, Text or CSV by right-clicking the Individual Records summary and clicking Export.

You can also create a report template to access this same information, but as there is no ‘Message’ summary to choose from, you need to use the Custom expression options, both when adding a column to a node in a Template, and when specifying your filter.

To add a column to a report that displays an Event Message:

1. Go to the Reports Tab and click New Template
2. Create an Analysis template based on the ‘All Windows Event Schemas’ schema
3. Click New Node and click the Advanced button to launch the Advanced editor.
4. On the General page, delete any existing Key columns and select Add | Key. In the Custom Expression section enter [Message] (include the square brackets) and click OK.

To filter the report:

1. Go to the Filters page of the New Node dialog (alternatively you can specify this filter in for all nodes using the Template Properties dialog)
2. Click Add | Field Value Filter. Select Category from the Summary drop down, and click Add. Enter ‘File System’ (without the quotes) and click OK. Click OK to add the filter.
3. Click Add | Field Value Filter. Select Event ID from the Summary drop down and click Add. Enter ‘4463′ (without the quotes) and click OK.
4. To filter on the Message field, Select Add | Manual Filter Expression.
5. Enter the expression:
6. [Message] LIKE “text to filter for”
   Change ‘text to filter for’ to the user or file that you want to search for. If you want to search for multiple strings, repeat the above expression separated by an AND or an OR, and place brackets wherever it makes sense. For example:
   • [Message] LIKE “scottg” AND [Message] LIKE “.avi”
     Will filter for all .avi files that scottg has accessed.
   • [Message] LIKE “scottg” OR [Message] LIKE “.avi”
     Will filter for any file that scottg has accessed and any avi that anyone has accessed.
   • ([Message] LIKE “scottg” AND [Message] LIKE “.avi”) OR [Message] LIKE “andrew”
     Will filter for any all avi files that scottg has accessed and any file that Andrew has accessed.
7. You can add the individual filters using Add | Manual Filter Expression multiple times, and then using the Manual Filter Expression editor at the bottom to change ANDs to Ors and place brackets appropriately, like so:
   

   Filtering for File Access Events by particular users

8. Right-click the Manual Filter Expression edit box and select Validate to make sure everything is good with the expression.
9. Modify chart settings, sorting, etc as appropriate.

Here’s the resulting report template for you, but please note that it includes the filter above (events for the user’s  ‘Asa’ and ‘Scottw’), so you will need to modify the filter and enter the users or files you want to filter on. Just use the user’s windows login name, and/or the name of the file.  Alternatively, remove the filter altogether if you want to see all File Audit events.

That’s it! Now run your report, automate it using the Tasks screen, and your set!