Of note, this release includes full support for Microsoft Exchange 2010 Tracking logs (previously supported with the Exchange 2007 loader, but missing a few fields), as well as JunOS (Juniper) Traffic Logs, IronPort Traffic Monitor Logs and Squid Syslog.

We’ve also included an experimental feature to allow multiple instances of WebSpy Vantage to run on the same operating system. The goal here is to run multiple reports at the same time using multiple instances of the application. To do this, we have also included a second experimental feature to disable storage locking. This allows multiple instances of Vantage to read from the same storage at once. These features can only be enabled by including a config file next to the Vantage’s executable. More on this feature here.

Here’s the full list of changes:

Application Changes

• New: Added suffix option to Import Windows Users wizard in Aliases.
• New: Date modifiers now supports h for hour and n for minute, e.g. %[-2h,yyyyMM - HH].
• New: Added tracing to storage publish task.
• Experimental: Multiple instances of Vantage can now be run simultaneously, by adding the multipleInstance key to the application config file.
• Experimental: Storage locking can be turned off to allow multiple instances of Vantage to run reports on a single storage simultaneously. This is done by adding the storageLocking key to the application config file.
• Fix: Import Organization merge options now appends attributes if keep existing user details is selected, and replaces attributes if update user details from the directory is selected.
• Fix: Import Organization merge no longer replaces user’s passwords.
• Fix: Fixed issue where no results were returned when filtering on time less than one day – such as past n hours.
• Fix: Storages are no longer duplicated in the Import new hits task dialog.
• Fix: Fixed issues where the Site Domain summary included sub-domains for European domains (.fr, .be etc).
• Fix: SQL server inputs now commit correctly if the user edits the input and only changes the port number.

Loader Changes

• New: IronPort Traffic Monitor Logs.
• New: Juniper JunOS Traffic Logs (SRX).
• New: Microsoft Exchange 2010.
• New: Squid Syslog.
• Improved: Astaro Security Gateway: Added support for an additional different syslog header.
• Improved: SonicWall: Split syslog format into Web and Firewall schemas, added support for User field, string-type Category field and split Protocol field.
• Fix: Microsoft FTMG: Changed type of Object Source field in from Int32 to String. Users will need to clear/field select/reload their storages before this change will apply.
• Fix: Astaro Mail Gateway: Improved format detection, fixed negative size issue, and Index out of bounds errors.
• Fix: IronPort WSA: Improved format detection.

How to update

To update your software, simply click Tools | Check for updates. Vantage Ultimate users will also need to update the Web Module in order to use the new loader formats that have been added. To update the Vantage Web Module, right-click the WebSpy system tray icon and select ‘Check for updates’. If you have issues with the Web Module update process, please see: http://www.webspy.com.au/forums/viewtopic.php?f=4&t=29

Let me know if you have any questions or issues!

What’s New?

Clearswift SECURE Web Gateway W3C

Clearswift have just released the latest version of their SECURE Web Gateway, which includes a transaction log export function. This enables you to send transaction logs in W3C format to an off-box FTP server for analysis. If you are updating to the latest Clearswift SECURE Web Gateway, make sure you update your Vantage software to 2.2.0.55 in order to import your W3C Transaction logs. More information on using WebSpy Vantage with Clearswift SECURE Web Gateway.

Cisco Firewall Bandwidth loader

We have also introduced a new Loader for Cisco ASA, PIX and IOS Firewall devices. This new loader imports TCP, UDP, ICMP and GRE ’session close’ events into one schema, allowing you to aggregate size values across these events. This loader is called Cisco Firewall (Bandwidth) and is now available on the Loader Selection page of the Import Wizard. Previously, these events were imported into separate schemas so there was no great way to determine total bandwidth from your Cisco syslog files (without using Netflow and WebSpy FlowMonitor).

Palo Alto Networks and WatchGuard XTM

We’re also very happy to welcome Palo Alto Networks to the WebSpy supported log file list. Vantage now supports both the CSV and syslog file formats from your PA Firewall.

Another new addition is support for the latest WatchGuard XTM devices running firmware version 11.

Full List of Changes

Here’s the full list of changes included in this update:

• New: Clearswift SECURE Web Gateway W3C.
• New: Palo Alto Networks Firewall (CSV/Syslog)
• New: Cisco Firewall (Bandwidth): This new Cisco loader imports TCP, UDP, ICMP and GRE events from ASA, PIX and IOS syslogs into one schema to aggregate size values across these events.
• New: Added WatchGuard XTM: Currently http-proxy, https-proxy, smtp-proxy and firewall lines are supported.
• Fixed: ISA Server: Fixed format detection issues, and issues importing hits with very large size values.
• Fixed: IronPort WSA: Fixed format detection issues, as well as the import issue “Invalid value for DVS Scan Code”
• Fixed: Sophos WSA: Fixed format detection issues and invalid line issues.

How to update

To update your software, simply click Tools | Check for updates. To update the Vantage Web Module, right-click the WebSpy system tray icon and select ‘Check for updates’. If you have issues with the Web Module update process, please see: http://www.webspy.com.au/forums/viewtopic.php?f=4&t=29

Let me know if you have any questions or issues!

An issue was introduced in build 2.2.0.42 when a change was made to the underlying data type of the Url Category field in the Microsoft TMG Web Schema. Unfortunately this change resulted in nulls being imported into the Url Category summary when importing from TMG’s SQL log files. Fortunately, importing W3C text logs were unaffected by this change.

This issue has now been fixed and released as an auto-update. To update your software simply select Tools | Check for updates.

If you have existing storages that have not been importing the URL Category, go to the storages screen and click ‘Reload All’. This will re-import your log files and populate the Url Category summary.

For all the customers affected by this issue, thank you for your patience and understanding.

Here’s the full list of changes:

• New: Juniper SA Series. Vantage can import and report on web traffic and VPN connections.
• New: Microsoft Forefront Protection for Exchange 2010 format.
• New: Avencis SSOx.
• Improved: IronPort WSA: Department and Message fields were sometimes returned as null. Fixed.
• Improved: Microsoft FTMG: Removed usage of deprecated “FilterInfo” field from W3C Web format.
• Improved: Microsoft IAS Radius: Added support for Source/Destination IP and port (field code 5000).

To update your Vantage application simply select Tools | Check for updates.

To update the Vantage Web Module, right-click the WebSpy icon in the Web Module server’s system tray, and select Check for updates. If you have any issues with the Web Module update process, please see my previous blog regarding Web Module Errors and Workarounds.

This release will be welcomed with open arms by many customers for the following reasons:

• General usability improvements in the Web Module
  Multi-select / delete options, Ajax progress indicators to avoid page refreshes, export from Dynamics Report tab and more (see below)
• Fixes to the Microsoft Forefront TMG loader
  See my other post: Microsoft Forefront TMG logs size fields the wrong way around. Also fixed ‘value cannot be null’ error when importing SQL logs.
• Fixes to storage corruption issues
  This build should prevent ‘Normalization Index’ storage corruption issues from occurring. This often occurred after importing data, editing some log inputs and reimporting.
• New loaders and more fixes
  See below for the full list

To update your Vantage application, simply choose Tools | Check for updates. To update the Web Module, right-click the WebSpy icon in your system tray and select ‘Check for updates’. If you have any issues updating the Web Module, please see my previous post Web Module Update Errors and Workarounds.

Web Module Changes:

• New: Task progress is now updated without refreshing the page
• New: Added multi-select / delete functionality to Reports, Analyses and Storages tables.
• New: Added export functionality to Dynamic Reports view.
• New: Added Performance section on the Options tab to enabling multi-processing (improves Analysis speed)
• Fix: Dynamic Reports view now supports Trend reports.
• Fix: Organization selector on Dynamic Reports view now always reflects updated data under IE6/7/8.Fix: Fixed javascript errors in IE when expanding the organization filter.
• Fix: Report template names are no longer truncated on the Dynamic Reports view.
• Fix: Fixed errors that may occur when collating reports on the Dynamic Reports page.
• Fix: Authentication errors are now logged with stack trace.

Vantage Changes

• Fixed: ‘Normalization index’ storage corruption problems.
• Fix: Report collation: Added support for collation of Min/Max aggregates on DateTime columns (time of first hit etc). Also added support for arrayed fields (for example, category fields with a comma separated list of categories)
• Fix: Import windows wizard now remembers settings for Import all or selected users
• Fix: Organization: Filtered LDIFs may now be imported when references to some users are missing (for example, if a user’s manager does not exist in the LDIF)
• Fix: Improved connection and error handling between Vantage and the Web Module.

Loader Changes

• New: BlueReef Sonar Total Management Module
• New: Microsoft Sharepoint 2007
• New: SmoothWall Guardian 7.0 format
• New: Sun One Proxy (Supported under Sun One Webserver)
• Fixed: Astaro: Improved format detection
• Fixed: Cisco: Strings in the IP fields of 113019 lines are now imported
• Fixed: IronPort WSA: Improved log format detection
• Fixed: Microsoft Exchange 2007: No longer raises issues regarding total-bytes or internal-message-id fields
• Fixed: Micorosft FTMG (Web) SQL: No longer encounters value could not be null errors
• Fixed: Microsoft FTMG: Added option to reverse bytes received/sent fields. See Microsoft Forefront TMG logs size fields the wrong way around
• Fixed: Microsoft IIS W3C: Now imports cs-method and connection ID
• Fixed: Sophos Web Appliance: Switched the outgoing and ingoing sizes so that they are now the correct way around
• Fixed: Fixed import new hits issue associated with W3C formats. You must reload your logs before this change will take affect. Formats affected include: BlueCoat, Clearswift, Microsoft Exchange 2007, Microsoft FTMG, Microsoft Windows Media Services, WebSpy Live Tracking Log

Enjoy!

This is a great update for Vantage Ultimate users as we’ve introduced a new feature/tab into the Web Module called ‘Dynamic Reports’.

If you’re publishing the same report to the Web Module each day, you can use the Dynamic Reports tab to select a date range and a department (or whatever organizational groups you have defined) and the Web Module will collate all the daily reports that match that filter into one report. This allows you to report on entire week, month or year by simply ‘reporting on reports’, rather than reporting months of raw storage data.

Here’s the full list of changes since the last auto update (2.2.0.32 on the 14th April 2010).

Application Changes

• Added Dynamic Reports feature to the Web Module.
• Rewrote the Web Module transfer protocol. New protocol adds version checking, connection checking, and integrity checking for high latency environments.
• Purge data from storage task no longer prevents importing new hits when all data is removed from an input within a storage.
• IPv6 addresses now show IPv4-mapped addresses as plain IPv4 addresses in summaries.
• IPv6 and IPv4 addresses are now freely interchangable in filter expressions.
• Fixed IPv6 drilldowns on the Summaries screen
• SQL inputs can now be resumed from the previous position. Previously any input that was partially imported would be skipped when importing new hits.
• Template-based analysis has been fixed, no longer results in blank/non-existent analysis.
• Added new string manipulation functions to expression language; Contains, StartsWith, EndsWith, IndexOf.

Loader Changes

• Astaro: Now checks that the ID field is present in a line before attempting to read it.
• Barracuda Web Filter: Added this format to replace Spy Filter.
• BlueCoat Proxy SG W3C: Added support for gmttime, timestamp, x-bluecoat-surfcontrol-is-denied and x-bluecoat-transaction-id.
• ClearSwift: Added a new loader group for ClearSwift that includes the MimeSweeper loaders
• ClearSwift SECURE Web Gatway: Now supported with the Web Appliance loader
• Clearswift Web Appliance: User summary displays Source IP if Username is blank.
• IronPort WSA: Fixed memory usage issues.
• Microsoft FTMG: Added category name lookup to SQL loader.
• Microsoft FTMG: No longer fails to import lines where the rule field contains square brackets.
• Microsoft FTMG: URL Category field is now a string instead of an integer. Added URL Categorization Reason field.
• Microsoft FTMG: Fixed memory usage issues.
• Microsoft IIS W3C: No longer hangs or crashes when loading a file that isn’t IIS W3C.
• NetAsq: Added support for srcname field. The Username summary is populated with user first, and then srcname if user is blank. The User summary is also now populated with Source IPs if the Username summary is blank.

To update WebSpy Vantage, simple select Tools | Check for updates.

To update the Web Module, login to the Web Module server, right-click the WebSpy system tray icon, and select Check for updates.

As always, please contact us if you have any issues or questions.

Unfortunately there are two issues with the Web Module auto update process. Everyone on a 64 bit operating system will encounter issue #1 (Unable to locate installation location), and some of you may encounter issue #2 (System.IO.FileLoadException).

This article describes the errors and how to work around them to successfully update the Web Module.

Issue #1

The usual process to update your Web Module is to log into your Web Module server, right-click the WebSpy system tray icon and select ‘Check for updates’.

If you do this on a 64 bit operating system, you will receive the following error:

Web Module Updater error: Unable to locate installation location

You can fix this issue by clicking Yes, and specifying the Web Module’s installation location, which is usually somewhere under c:\inetpub\wwwroot (or just c:\inetpub\wwwroot if you didn’t specify a virtual directory when installing).

This will allow the updater to continue and you will be prompted to download and install the latest update.

Issue #2

Unfortunately, you may encounter another error during the update installation. The text of the error will be something along the lines of:

System.IO.FileLoadException: Could not load file or assembly ‘ICSharpCode.SharpZipLib, Version=0.84.0.0, Culture=neutral, PublicKeyToken=1b03e6acf1164f73′ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0×80131040)

Work Around

We are currently working on solutions to both of these issues. In the mean time, here is a work around to install the update. On the Web Module server:

1. Download this file:
   http://update.webspy.com/autoupdate/files/vantagewebmodule/vantagewebmodule2.2.0.18.zip
2. Stop IIS (See instructions below).
3. Right-click the WebSpy system tray icon and click Exit.
4. Backup your existing Web Module installation by copying everything in your Web Module’s installation folder (usually under c:\inetpub\wwwroot (or just c:\inetpub\wwwroot if you didn’t specify a virtual directory when installing) into a completely separate location (i.e. don’t keep it in a sub-folder).
5. Extract the downloaded zip file to your web module’s installation folder. Overwrite all the existing files.
6. Start IIS (see instructions below).

Your Web Module will now be updated to the latest version.

Stopping and Starting IIS

In IIS Manager (Start | Control Panel | Administrative Tools | Internet Information Services (IIS)) , right-click the site you want to start or stop, and click Start or Stop

We sincerely apologize for the inconvenience and will hopefully have a solution out soon. If you have any problems with the process above, please contact support at webspy dot com.

For those that don’t know what Soho is all about, check out this video:

Soho is a dashboard application that displays download and upload traffic statistics for each computer in your network. If you haven’t yet tried Soho, please give it a go and download it here.

It is our intention to keep our Alpha testers up to date with our ongoing development. Right now, I’d like to inform you about some issues experienced by a handful of testers and how to go about resolving them.

Learn to restart the Soho Agent

First of all, one of the handiest things we can tell you right now is how to restart the Soho Agent. This single step is resolves 99% of all Soho issues, at least temporarily. If these steps seem too complicated, rebooting your PC also has the same effect.

To restart the Soho Agent on Windows:

1. Launch the Services Console by going to Control Panel | Administrative Tools | Services. Or if you like handy short cuts, try Start | Search (or Run), Type ‘services.msc’ (without the quotes) and press enter.
2. Right-click the “WebSpy Soho Agent” service and select Restart. If you get a ‘time out’ error message or warning, ignore it and right-click the service again and select Start.

To restart the Soho Agent on Mac OS:

1. Open the terminal from /Applications/Utilities/Terminal
2. Type sudo launchctl stop “WebSpy Soho Agent”
3. Enter your user password if requested.
4. Wait about 5 seconds.
5. Type sudo launchctl start “WebSpy Soho Agent”
6. Again, enter your user password if requested

OK, now you have the skills, here are the issues and work-arounds:

100% CPU usage after sleep or hibernate

Some users reported Soho’s impressive ability to consume 100% of their CPU when their computer wakes from sleep or hibernation. A few users experienced a slow and sluggish PC, and uninstalled Soho immediately.

If you’re looking for the Soho process in Windows Task Manager you will not see it until you click the Show processes from all users button (or checkbox in XP). This is because the Soho Agent runs under the System user account in order for it to run, no matter who is logged onto the PC.

From here you can end the WebSpy.Soho.Agent.exe process and everything should return to normal. You can then restart the “WebSpy Soho Agent” to get Soho working again (see steps above).

We believe we have fixed this and are in the middle of some final testing. All going well, we should have a new build ready for you very soon. In the mean time, you may like to disable sleep and hibernation in your PC’s power options.

Soho doesn’t install on Mac OS 10.5??

Our first Alpha release did not install on Mac OS 10.5 (Leopard). This was due to a silly checkbox in our packaging system not being checked. We’ve checked the checkbox and uploaded a new build to our web site. You can download it from here:
http://www.webspy.com.au/products/soho/download.aspx

Note: Soho will only install on Mac OS 10.5 (Leopard) and 10.6 (Snow Leopard). Versions 10.4 (Tiger) and below are not supported.

All computers disappear from the Current Activity chart except the local computer

Sometimes all computers will disappear from the Current Activity chart leaving your local computer all by itself. This happens when the communication between Soho Agents becomes jammed. You can usually resolve the issue by restarting the Soho Agent on your local computer (see steps below). If this doesn’t work, restart the Agents on all other computers running Soho. We are currently working on a fix for this issue.

The Soho User Interface is completely blank

If there is no information in the Total, Current Activity or History chart, this is because the Soho Agent has stopped running. Reasons for this may vary, so please let us know if this is regularly occurring. Restarting your agent usually resolves the issue (see steps above).

Feedback

Thank you to everyone that has submitted feedback so far.

Just a reminder to please let us know if your network card works or doesn’t work with Soho on this page:
http://www.webspy.com.au/products/soho/supportednics.aspx

You may also like to review the current list of features and bugs and vote them up or down at:
http://webspysoho.uservoice.com/

There is also a dedicated Soho Alpha Feedback thread in our forums at:
http://www.webspy.com.au/forums/viewtopic.php?f=8&t=12

We will let you know when fixes are available to the above issues.

When imported into WebSpy Vantage, the result is shown in a new summary called ‘Group’ which you can add to your reports.

We also added support for the custom fields Bytes Sent and Bytes Received. Due to the absence of a header in the IronPort access log, Bytes Received and Bytes Sent fields must both be present to be detected, and the Received field must precede the Sent field.

We also added support for the custom fields Request Body Size and Response Body Size. These fields can be included in your access log by adding %q (Request body size) and %b (Response body size) in the ‘Custom Fields’ edit box. Due to the absence of a header in the IronPort access log, Request Body Size and Response Body Size fields must both be present to be detected, and the Request field must precede the Response field.

We’ve also noticed that the values in the Bytes Sent and Bytes Received fields do not necessarily add up to the value logged for ‘Size’. We’re discussing this issue with our friends at IronPort and we will hopefully post a solution or explanation soon..
The information we first received about these fields indicated they represented Bytes Sent and Bytes Received. This is the way they are represented in the builds below (2.2.0.29). We will release a new build soon, with the field names changed to Request body size and Response body size. Body size is different to bytes sent/received as it does not include bytes from packet headers etc.

We’re yet to issue an automatic update for the Vantage applications, so in the mean time you can download the latest builds here:

Vantage Ultimate:
ftp://ftp.webspy.com/webspy/Builds/VantageUltimate2.2.0.29.zip

Vantage Web Module:
ftp://ftp.webspy.com/webspy/Builds/VantageWebModule2.2.0.8.exe

Vantage Giga:
ftp://ftp.webspy.com/webspy/Builds/VantageGiga2.2.0.29.zip

Vantage Premium:
ftp://ftp.webspy.com/webspy/Builds/VantagePremium2.2.0.29.zip

To apply the Vantage update, close Vantage and extract the downloaded file into Vantage’s installation folder (Usually c:\Program Files\WebSpy\Vantage <flavour> 2.2). Overwrite the existing files.

To apply the Web Module update, uninstall the Vantage Web Module from Add/Remove Programs (Programs and Features in Windows 7/Server 2008), then run the downloaded exe file, making sure you specify the same server, virtual directory and data location that your Web Module was previously using.

We will be releasing this as a public auto-update soon. Let us know if you have any issues.

Check your version in Help | About. If you are running 2.2.0.27 or above, then you already have this update. If not, make sure you update to your software by selecting Tools | Check for updates.

About WebSpy Soho

WebSpy Soho is our soon-to-be-launched dashboard application for anyone who wants to know the full story on their computer and network bandwidth usage and Internet connection speed. WebSpy Soho is perfect for small offices, home offices, families, shared student households and even a single computer home user.

No longer do you need to be a network administrator to monitor your computer or network, Soho makes it easy and user-friendly for everyone.

WebSpy Soho will help you answer some of these questions:

• How much is each computer on my network uploading and downloading?
• Which computer is slowing down my internet connection speed?
• Am I getting close to reach my ISP internet quota?
• Is Malware eating up large chunck of my bandwidth?
• Is someone freeloading on my wireless connection?

Jason Calacanis and Jack Andrys talking about Soho in TWiST ep #31

What makes WebSpy Soho so special?

There are already many of network and bandwidth monitor solutions out there, both free and paid versions. However, all of these seem to miss the mark when it comes to assisting the new generation of people who want to use them. Gone are the days when only Network Administrators wanted to monitor bandwidth usage and internet connection. Today every other household (and small office) has a network of computers. And guess what? Majority of these users are not Network Administrators or super technical. More often than not it is just your average person who wants to find out more about their network usage.

This is where WebSpy Soho comes in – WebSpy Soho has been specifically designed with these users in mind. Don’t get me wrong, a Network Administrator will be thrilled with WebSpy Soho’s functionalities as well. He/she just won’t have to use any technical skills to set up and use Soho.

WebSpy Soho is different from competing products because:

• WebSpy Soho lets you monitor 1- 15 computers on your network. Majority of free bandwidth monitoring solutions only lets you monitor your local computer.
• Compared to many paid network bandwidth monitoring software, WebSpy Soho is very easy to set up and start using.
• WebSpy Soho is interactive and customizable. You can view real time data as well as specify historical date ranges you want to investigate. WebSpy Soho also lets enter your ISP billing periods and quota allowance to easily keep track of your usage. Majority of other bandwidth monitoring software only gives you a static overview you cannot customize.
• WebSpy Soho distinguishes between internal and external network traffic. Soho will only display actual internet uploads and downloads to prevent skewing your result.
• WebSpy Soho provides you with alerts when your internet connection becomes overloaded, when you are approaching your ISP quota limit, and when an unknown machine is using your network.

Find out more and register your interest for the chance to win one year FREE subscription of WebSpy Soho at http://www.webspy.com/products/soho/default.aspx

TMG or UAG? What is the difference?

TMG is an outgoing proxy that protects your internal users from malware, viruses and the like. TMG generates some great web proxy log files to import into WebSpy Vantage allowing you to monitor where your users are going on the Internet, how much they’re downloading etc.  TMG, unlike ISA, now has deep packet inspection for HTTPS traffic, plus a bunch of other new features.

UAG is an incoming proxy that provides employees, partners and vendors secure remote access to corporate resources such as Outlook Web Access (OWA) and Sharepoint (MOSS). It utilizes the TMG engine, but this is mainly just to protect the UAG server (more on this topic here http://technet.microsoft.com/en-us/library/ee522953.aspx).

TMG can also publish your OWA and MOSS sites, but this is no longer recommended by Microsoft. They recommend using a dedicated UAG server to perform this function.

Upgrading to TMG

If you’re thinking about migrating your ISA server (2004 or 2006) to TMG, you may like to check out this migration guidance video with Mohit Saxena (Senior Technical Lead) and Jim Harrison (Program Manager). http://edge.technet.com/Media/ISA-to-TMG-Migration-Guidance/

Microsoft Forefront TMG Migration Video

Reporting on TMG

If you’re using TMG at the moment, we invite you to analyze your web proxy and/or firewall logs using WebSpy Vantage and tell us what you think!  Download your copy of WebSpy Vantage here, and import your logs using the Microsoft FTMG format:

Microsoft Forefront Threat Management Gateway

You should be prompted to update your software on startup, but if you’ve turned off that feature, simply go to Tools | Check for Updates.

New Features

This new build sports the following new features:

• Support for Microsoft Forefront Threat Management Gateway (Beta)
  Microsoft Forefront Threat Management Gateway (FTMG) is still currently in Beta, and is due to be released around November 2009. For those that do not know, FTMG is the next version of Microsoft’s popular ISA Server. Information and downloads for FTMG can be found here http://www.microsoft.com/forefront/edgesecurity/isaserver/en/us/tmg-beta.aspx. We have added support for FTMG beta 2 and 3 for both the W3C text logs (recommended) and the internal SQL Server Express Database logs. If you are currently trialling FTMG, we are very interested to hear your feedback. Let us know how you go!
  

  Now Supported - Microsoft Forefront Threat Management Gateway

• Data purge
  You can now purge data from a storage, and schedule this purge to occur on a regular basis using Tasks. Purge options include data between a date range, data before a date, data after a date, data older than a date relative to now, and all data. This feature will let you easily maintain a single storage that only includes data for the last month or day.
  

  Options for Purging data from your storage

• Import Organization from CSV can now be scheduled using Tasks
  If you are importing your organizational structure from CSV, you can now schedule this action using Tasks. This enables you to update your organizational structure before any reports are run.
  

  Import Organization from CSV via Tasks

• Added Support for ExoServer Web
  If you’re running ExoServer Web, you can now analyze it’s logs using WebSpy Vantage.

Fixes

We also fixed some things that may have been bugging you:

• Improved the start time for the application by improving the logic to check for Storage damage.
• Fixed the IronPort loader (Fixed out of range issues on excessive size fields).
• “Having” filters no longer override the sort order of a Report Template node.
• Fixed an issue that may result in duplicated storages after migrating settings from earlier versions.
• Fixed the inability to remove invalid entities from web module permissions list (users that no longer exist).
• Fixed a timeout issue when publishing storages to the web module.

Why are you still reading? Go update now!

Have fun!

The survey only consisted of six questions and to prevent influencing answers and encouraging unique thoughts and opinions, the majority of questions were open ended. Those answers applicable for categorization have now been categorized, compared and correlated. The survey was conducted for internal purposes but I would like to share some of the findings and comments we received.

Vendor devices and log formats

The top ten vendor devices client used WebSpy’s software to analyze and report on:

1. Microsoft ISA Server (a whopping 48.6%)
2. Microsoft Exchange
3. Novell – BorderManager
4. Squid – Proxy Server
5. Microsoft Proxy Server
6. Microsoft – Internet Information Services (IIS)
7. Blue Coat – Proxy SG
8. Check Point – Next Generation
9. IronPort – Web Security Appliance
10. Microsoft – Windows Event Logs

Business Issues

The most common reason why clients started to monitor and report on their log files:

1. To achieve transparency, higher visibility, on overall internet and network resources. (No actual issue quoted)
2. To reduce issues related to lost productivity, inappropriate and illegal internet usage
3. To be able to perform detailed investigations on specific user and site levels
4. To reduce bandwidth cost and increase speed
5. To achieve monitoring requirements not satisfied by vendor device
6. To reduce viruses and other security issues
7. To produce reports required by management
8. To trend and forecast Internet and network usage
9. To investigate email usage
10. To immediately address critical issues using real-time monitoring and alerts

Most valued WebSpy software features

1. Dynamic Drilldowns
2. Ad-hoc analysis and summaries screen
3. Real-time monitoring and alerts
4. Comprehensive and detailed reports and analysis options
5. Predefined and customizable reporting and analysis templates
6. Aliases – Logical grouping of data to represent it more meaningfully
7. Profiles – Categorize web site URLs, email subject lines, instant message chats and any other logged data using customizable keyword profiling technology.
8. Extensive log file compatibility and support
9. Ease of use
10. Automated task scheduling

Suggested Improvements

The majority of answers related to suggested improvements and added software features were very specific to individual user experiences and quite difficult to categorize. However, here a list of suggested improvements that our development has taken to heart and decided to prioritize:

Increase and Improve Number of Default Report and Analysis Templates
Major changes and improvements to our reporting engine and interfaces are currently being planned. A short term goal is to provide the ability to create reports in PDF format. This should hopefully be available in the coming months.

Longer term, we will be incorporating a visual report designer (drag / drop charts and tables etc), as well as the ability to create reports that collate information from different sources. For example, using the information from Event logs, Web proxy logs and email server logs, you could produce a report that shows when a user logged on to their PC, what sites they browsed, what files they accessed, and the emails they sent – all sorted chronologically in the one table!

We currently have a large range of report templates available, but choosing the one you want can be a time consuming process, and you will most likely want to customize the report once you’ve found one that suits. We are planning to make the whole process of creating the report YOU want much, much easier.

Further Expand Profiles (Website Categorization)
When it comes to website categorization, WebSpy applications utilize keyword profiling in addition to importing category fields from log files (if available). There are upsides and downsides to keyword profiling. A simple keyword can instantly categorize thousands of sites which is much more efficient than maintaining a URL Category database. Being able to categorizing hits to your own organization’s website or intranet is also a very useful feature.

We are committed to distributing more frequent profile updates, without overwriting any customizations you have made. Longer term, we plan to provide an option for collaborative keyword customization, so that you benefit from the customization that other customers are making around the world. We also plan to integrate third party categorization services, as there are many great organizations that focus purely on URL categorization.

Improve Software Updates Correspondence
Our main mass communication channels include our newsletter, press releases blog and twitter. Here is where we inform existing and prospective clients about our software, updates, offers and much more. Starting September 2009 we will also send out emails, on a monthly basis, informing existing clients about the specific software updates implemented and are ready to download.

Further Improve Reporting Speed
Vantage 2.2 ranges (Premium, Giga and Ultimate) now use multithreading techniques to utilize the extra processing power on machines with multiple cores or CPUs to import log files and generate reports faster. In addition to the multi-processing improvements recently made in Vantage 2.2, the development team are planning to implement even more multi threading improvements in the next major version. These improvements will be available for users of both the Analyzer and Vantage range.

Thanks again for your participation!

The winner of the USB Camera Pen has been picked using a random draw and will be contacted personally via email.

All three products in the Vantage range (Premium, Giga and Ultimate) have been updated with the following features:

• Multi-processing
  Vantage now utilizes the extra processing power on machines with multiple cores or CPUs to import log files and generate reports faster.
  
  

  The new Performance tab in Vantage

• Improved report styles
  HTML (including MHT) reports are now sporting an updated look and feel.
  
  

  New Report Style

• Updated Aliases and Profiles
  Added support for the latest search engines, social networking sites, operating systems and user agents.
  
  

  Profiles Aliases support the latest Internet tools and technologies.

• Storage Repair Utility
  Storages can become damaged if there is a system crash or if WebSpy.Vantage.exe process is ended when an import is in progress. Damaged storages are now detected automatically and you have the option of repairing them. Storages can also be manually checked and repaired on the Storage properties dialog.
  
  

  Storage Repair Utility

• Save / Open Tasks
  Scheduled Tasks can now be saved to an external file (.Tasks) for backup and migration purposes.
  
  

  Save and Open Tasks

Obviously, the multi-processing feature is the big one! Many of you (Vantage customers) have been running Vantage on multi-core or multi-cpu machines and just dying for Vantage to grab hold of the CPUs and make them work. You should be happy with this new build. Import a folder full of log files and watch 6 to 8 logs import simultaneously instead of watching them import one after the other. Run a full analysis or a comprehensive report with no filters and watch your CPU jump into gear to generate the report in 30% – 50% of the time. We’re expecting this to be a very popular update, especially among our larger customers.

Multi-processing is a great new feature, but be aware that there are some circumstances where Vantage will not be able to utilise more CPU power when generating reports. For example, if your report contains filters that significantly cuts down the amount of data in your storage that needs analysis, Vantage’s analysis engine and your CPUs will not be significantly pushed. Also see my earlier blog for more information on how the report structure can also affect the performance.

To get hold of this build, simply download the latest version of Vantage from our website. It will run side by side with your existing Vantage 2.1 installation, which minimizes any downtime, and makes it easy to transfer all your settings across (use the Migration Wizard in the Tools menu). Then you can uninstall Vantage 2.1 when you’re ready.

Let us know how you go by leaving comments below, or by tweeting us.

Cheers!
Scott

Well, it’s now been released and can be downloaded from the Microsoft Download Center:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=e05aecbc-d0eb-4e0f-a5db-8f236995bccd

One of the major improvements in Beta 3 is URL filtering which leverages Microsoft Reputation Services (MRS). With regards to this, Microsoft says:

“At the time of this release, the MRS database content is being populated and updated continuously as part of the initial beta service offering. As this process continues, URL filtering categorization accuracy and comprehensiveness will increase. A telemetry package designed for improving the quality of URL filtering database and collecting your feedback is planned to be released soon. Please check back for updates in August.”

Support for importing Microsoft TMG log files into your favorite WebSpy product is coming soon so stay tuned! Subscribe to the WebSpy blog RSS feed or follow us on twitter if you want to be notified as soon as it’s available.

According to the latest blog from TMG’s Product Unit Manager, David B. Cross, Beta 3 will be released in the next couple of weeks. As for the full release, David says that they are still on track for Q4 this calendar year.

Beta 3 will introduce URL filtering that is ‘fully integrated’ with TMG’s web policy rules, and also utilizes Microsoft Reputation Services.

Microsoft are also introducing Intrusion Prevention and Detection (IPS/IDS) capabilities in TMG. These systems will utilize a technology they’re calling Network Inspection System (NIS) that detects attacks using signatures of known vulnerabilities, downloaded from the Microsoft Malware Protection Center. For more information on NIS see http://blogs.technet.com/isablog/archive/2009/04/12/exercising-nis-with-test-signature.aspx

If you’re currently using ISA 2004 or 2006, upgrading to TMG will consist of exporting rules and settings from ISA, then importing them into a clean installation of TMG. TMG will also only run on Windows Server 2008.

Improving the on-box reporting has not been a focus for the TMG development team, so analyzing TMG’s web proxy and firewall logs is still the best way to go for in depth reporting.

If you’re interested in reporting on your TMG log files stay tuned! We’re currently implementing support for the SQL Express, W3C and Native text logs. WebSpy Vantage is likely to be the first application to include the feature, with Analyzer and Live soon to follow.

All going well, you can expect to see TMG support in your favourite WebSpy app within the next month or so. If you want to be notified once we’ve added support, just leave a comment below.

Cheers!
Scott.

The multi-processing build features a new tab called ‘Performance’ in Tools | Options. Ensure ‘Use multi-processing’ is checked, and set the ‘Maximum Concurrent Threads’ value to twice the number of logical/physical CPUs you have. For example, set it to 4 on a dual core, 8 on a quad-core, or 16 on a machine with two quad-core CPUs. Feel free to also play around with this value, but we have found this formula to be optimal and we’ll automatically set this as default in future builds.

The new Performance tab in Vantage

You’ll notice that if you import a folder of logs, about 6-8 logs will import simultaneously. Simply importing one log file will not show any speed improvement.

Multi-processing will only benefit report generation if your report template contains side-by-side summaries as opposed to drilldown summaries. Here’s what I mean by a template with side-by-side summaries:

A report template that only has side-by-side summaries

And here’s what I mean by a template with drilldown summaries

A report template with only drilldown summaries

Basically, each side-by-side summary will get processed simultaneously, but any drilldowns will be processed sequentially.

Most report templates consist of both side-by-side AND drilldown summaries, such as:

A report template with both side-by-side and drilldown summaries

The amount that these templates benefit from multi-processing will depend on the number of side-by-side summaries.

If this sounds like something you’d like to try, please contact me for a copy of the build!

Of particular note is a fix to the Vantage range:

• Fix: Improved partition matching when filtering by a date range

This will benefit you if you’re running reports with a date range filter such as last week or yesterday (which is pretty much everyone), and are using the default storage partitioning scheme of “Date” (again… pretty much everyone).

There was an issue in Vantage’s partition filtering technology which meant that if you specified a date range filter, Vantage would still analyse an entire storage, instead of just the date partitions you selected in the filter.

So do a Tools | Check for updates to grab build 2.1.2.12 and you should see a significant speed improvement in report generation. Woo hoo!

Here’s more details on the changes:

VANTAGE

Application Changes

• New: Added support for importing event logs from non-domain machines.
• New: Added the ability for the Troubleshoot Alias/Profiles to export results to a file.
• New: Added hash salt and substring functions to the expression language.
• New: Added support for computing a date-modified file mask at import time and storing it back into the storage.
• New: Added a file mask override to the Import new hits to existing storage task action.
• New (Vantage Ulitmate & Giga): Added support for attributes on Organization groups.
• New (Vantage Ultimate): Added custom expression-based split in Web Module publishing (Beta).
• New (Vantage Ultimate): Changed the publish report wizard to allow selection of multiple storages.
• Fix: Improved partition matching when filtering by a date range.
• Fix: Fixed an issue with registrations and trial extensions.
• Fix: Fixed zero-width rectangle exceptions in chart renderer.
• Fix: Fixed overflow exceptions on footer generation in report tables.
• Fix: Fixed XmlException error during settings load.
• Fix: Storage now clears previous data when it is overwritten when using the ‘Import logs into new storage’ task.
• Fix (Vantage Ultimate & Giga): Fixed issues with date range selection when collating reports.
• Fix (Vantage Ultimate): Web Module dock no longer checks for updates to a web module server when ‘Check for updates on startup’ is disabled.

Loader Changes

• New: Added Phion Firewall
• New: Added Watchguard Firebox X Core
• New: Added NetScreen 208
• New: Added IPCop
• New: Added Astaro Mail Gateway
• New: Added iPrism Monitor v4.2
• New: Added Cisco VPN Concentrator
• New: Added Snare for Lotus Notes
• Improved: Added string host name to Kerio Mail Server
• Improved: Improved support for NetIntact PacketLogic
• Fix: Made changes to the IronPort detection method to fix the issue when importing via FTP.
• Fixed: Fixed date format issue in event log import. You can now import event logs in any regional configuration.
• Fix: Postfix loader no longer drops session state after the first recipient line
• Fix: Updated the detection method in CheckPoint Firewall-1 Syslog format.
• Fix: Fixed an issue in iSheriff where a drilldown on the Category field would display no results.
• Fix: Various fixes to Netscreen 10
• Fix: Various fixes to Arkoon PxLog
• Fix: Vaious fixes to Sendmail MTA

VANTAGE WEB MODULE

• FIX: When sorting by key column the chart will now use the “hits” aggregate for value (prevents the chart from going wacky)

ANALYZER

Application Changes

• NEW: Added support for manual configuration of ISA block list settings (Tools | Options | Block Lists)
• Fix: Fixed an issue with registrations and trial extensions
• Fix: Fixed report wizard configuration to restore the sort column from the report template

Loader Changes

• New: Added Webroot
• New: Added Qmail Desknow Mail Server
• New: Added UserGate Proxy Server 2.7
• New: Added Netgear FVX538
• Improved: Added support for importing allowed/blocked status in Sophos Web format
• Fixed: Changed CC Proxy field count check
• Fixed: Changed ISA SQL loaders to use size delta fields instead of cumulative fields
• Fixed: Modified Exchange 2000/2003 loader to discard message cache after import

LIVE

Application changes

• Fix: Fixed an issue with registrations and trial extensions

Loader changes:

• New: Added Webroot
• New: Added Qmail Desknow Mail Server
• New: Added UserGate Proxy Server 2.7
• New: Added Netgear FVX538
• Improved: Added support for importing allowed/blocked status in Sophos Web format
• Fixed: Changed CC Proxy field count check
• Fixed: Modified Exchange 2000/2003 loader to discard message cache after import