Of note, this release includes full support for Microsoft Exchange 2010 Tracking logs (previously supported with the Exchange 2007 loader, but missing a few fields), as well as JunOS (Juniper) Traffic Logs, IronPort Traffic Monitor Logs and Squid Syslog.

We’ve also included an experimental feature to allow multiple instances of WebSpy Vantage to run on the same operating system. The goal here is to run multiple reports at the same time using multiple instances of the application. To do this, we have also included a second experimental feature to disable storage locking. This allows multiple instances of Vantage to read from the same storage at once. These features can only be enabled by including a config file next to the Vantage’s executable. More on this feature here.

Here’s the full list of changes:

Application Changes

• New: Added suffix option to Import Windows Users wizard in Aliases.
• New: Date modifiers now supports h for hour and n for minute, e.g. %[-2h,yyyyMM - HH].
• New: Added tracing to storage publish task.
• Experimental: Multiple instances of Vantage can now be run simultaneously, by adding the multipleInstance key to the application config file.
• Experimental: Storage locking can be turned off to allow multiple instances of Vantage to run reports on a single storage simultaneously. This is done by adding the storageLocking key to the application config file.
• Fix: Import Organization merge options now appends attributes if keep existing user details is selected, and replaces attributes if update user details from the directory is selected.
• Fix: Import Organization merge no longer replaces user’s passwords.
• Fix: Fixed issue where no results were returned when filtering on time less than one day – such as past n hours.
• Fix: Storages are no longer duplicated in the Import new hits task dialog.
• Fix: Fixed issues where the Site Domain summary included sub-domains for European domains (.fr, .be etc).
• Fix: SQL server inputs now commit correctly if the user edits the input and only changes the port number.

Loader Changes

• New: IronPort Traffic Monitor Logs.
• New: Juniper JunOS Traffic Logs (SRX).
• New: Microsoft Exchange 2010.
• New: Squid Syslog.
• Improved: Astaro Security Gateway: Added support for an additional different syslog header.
• Improved: SonicWall: Split syslog format into Web and Firewall schemas, added support for User field, string-type Category field and split Protocol field.
• Fix: Microsoft FTMG: Changed type of Object Source field in from Int32 to String. Users will need to clear/field select/reload their storages before this change will apply.
• Fix: Astaro Mail Gateway: Improved format detection, fixed negative size issue, and Index out of bounds errors.
• Fix: IronPort WSA: Improved format detection.

How to update

To update your software, simply click Tools | Check for updates. Vantage Ultimate users will also need to update the Web Module in order to use the new loader formats that have been added. To update the Vantage Web Module, right-click the WebSpy system tray icon and select ‘Check for updates’. If you have issues with the Web Module update process, please see: http://www.webspy.com.au/forums/viewtopic.php?f=4&t=29

Let me know if you have any questions or issues!

What’s New?

Clearswift SECURE Web Gateway W3C

Clearswift have just released the latest version of their SECURE Web Gateway, which includes a transaction log export function. This enables you to send transaction logs in W3C format to an off-box FTP server for analysis. If you are updating to the latest Clearswift SECURE Web Gateway, make sure you update your Vantage software to 2.2.0.55 in order to import your W3C Transaction logs. More information on using WebSpy Vantage with Clearswift SECURE Web Gateway.

Cisco Firewall Bandwidth loader

We have also introduced a new Loader for Cisco ASA, PIX and IOS Firewall devices. This new loader imports TCP, UDP, ICMP and GRE ’session close’ events into one schema, allowing you to aggregate size values across these events. This loader is called Cisco Firewall (Bandwidth) and is now available on the Loader Selection page of the Import Wizard. Previously, these events were imported into separate schemas so there was no great way to determine total bandwidth from your Cisco syslog files (without using Netflow and WebSpy FlowMonitor).

Palo Alto Networks and WatchGuard XTM

We’re also very happy to welcome Palo Alto Networks to the WebSpy supported log file list. Vantage now supports both the CSV and syslog file formats from your PA Firewall.

Another new addition is support for the latest WatchGuard XTM devices running firmware version 11.

Full List of Changes

Here’s the full list of changes included in this update:

• New: Clearswift SECURE Web Gateway W3C.
• New: Palo Alto Networks Firewall (CSV/Syslog)
• New: Cisco Firewall (Bandwidth): This new Cisco loader imports TCP, UDP, ICMP and GRE events from ASA, PIX and IOS syslogs into one schema to aggregate size values across these events.
• New: Added WatchGuard XTM: Currently http-proxy, https-proxy, smtp-proxy and firewall lines are supported.
• Fixed: ISA Server: Fixed format detection issues, and issues importing hits with very large size values.
• Fixed: IronPort WSA: Fixed format detection issues, as well as the import issue “Invalid value for DVS Scan Code”
• Fixed: Sophos WSA: Fixed format detection issues and invalid line issues.

How to update

To update your software, simply click Tools | Check for updates. To update the Vantage Web Module, right-click the WebSpy system tray icon and select ‘Check for updates’. If you have issues with the Web Module update process, please see: http://www.webspy.com.au/forums/viewtopic.php?f=4&t=29

Let me know if you have any questions or issues!

Here’s the full list of changes:

• New: Juniper SA Series. Vantage can import and report on web traffic and VPN connections.
• New: Microsoft Forefront Protection for Exchange 2010 format.
• New: Avencis SSOx.
• Improved: IronPort WSA: Department and Message fields were sometimes returned as null. Fixed.
• Improved: Microsoft FTMG: Removed usage of deprecated “FilterInfo” field from W3C Web format.
• Improved: Microsoft IAS Radius: Added support for Source/Destination IP and port (field code 5000).

To update your Vantage application simply select Tools | Check for updates.

To update the Vantage Web Module, right-click the WebSpy icon in the Web Module server’s system tray, and select Check for updates. If you have any issues with the Web Module update process, please see my previous blog regarding Web Module Errors and Workarounds.

This release will be welcomed with open arms by many customers for the following reasons:

• General usability improvements in the Web Module
  Multi-select / delete options, Ajax progress indicators to avoid page refreshes, export from Dynamics Report tab and more (see below)
• Fixes to the Microsoft Forefront TMG loader
  See my other post: Microsoft Forefront TMG logs size fields the wrong way around. Also fixed ‘value cannot be null’ error when importing SQL logs.
• Fixes to storage corruption issues
  This build should prevent ‘Normalization Index’ storage corruption issues from occurring. This often occurred after importing data, editing some log inputs and reimporting.
• New loaders and more fixes
  See below for the full list

To update your Vantage application, simply choose Tools | Check for updates. To update the Web Module, right-click the WebSpy icon in your system tray and select ‘Check for updates’. If you have any issues updating the Web Module, please see my previous post Web Module Update Errors and Workarounds.

Web Module Changes:

• New: Task progress is now updated without refreshing the page
• New: Added multi-select / delete functionality to Reports, Analyses and Storages tables.
• New: Added export functionality to Dynamic Reports view.
• New: Added Performance section on the Options tab to enabling multi-processing (improves Analysis speed)
• Fix: Dynamic Reports view now supports Trend reports.
• Fix: Organization selector on Dynamic Reports view now always reflects updated data under IE6/7/8.Fix: Fixed javascript errors in IE when expanding the organization filter.
• Fix: Report template names are no longer truncated on the Dynamic Reports view.
• Fix: Fixed errors that may occur when collating reports on the Dynamic Reports page.
• Fix: Authentication errors are now logged with stack trace.

Vantage Changes

• Fixed: ‘Normalization index’ storage corruption problems.
• Fix: Report collation: Added support for collation of Min/Max aggregates on DateTime columns (time of first hit etc). Also added support for arrayed fields (for example, category fields with a comma separated list of categories)
• Fix: Import windows wizard now remembers settings for Import all or selected users
• Fix: Organization: Filtered LDIFs may now be imported when references to some users are missing (for example, if a user’s manager does not exist in the LDIF)
• Fix: Improved connection and error handling between Vantage and the Web Module.

Loader Changes

• New: BlueReef Sonar Total Management Module
• New: Microsoft Sharepoint 2007
• New: SmoothWall Guardian 7.0 format
• New: Sun One Proxy (Supported under Sun One Webserver)
• Fixed: Astaro: Improved format detection
• Fixed: Cisco: Strings in the IP fields of 113019 lines are now imported
• Fixed: IronPort WSA: Improved log format detection
• Fixed: Microsoft Exchange 2007: No longer raises issues regarding total-bytes or internal-message-id fields
• Fixed: Micorosft FTMG (Web) SQL: No longer encounters value could not be null errors
• Fixed: Microsoft FTMG: Added option to reverse bytes received/sent fields. See Microsoft Forefront TMG logs size fields the wrong way around
• Fixed: Microsoft IIS W3C: Now imports cs-method and connection ID
• Fixed: Sophos Web Appliance: Switched the outgoing and ingoing sizes so that they are now the correct way around
• Fixed: Fixed import new hits issue associated with W3C formats. You must reload your logs before this change will take affect. Formats affected include: BlueCoat, Clearswift, Microsoft Exchange 2007, Microsoft FTMG, Microsoft Windows Media Services, WebSpy Live Tracking Log

Enjoy!

This is a great update for Vantage Ultimate users as we’ve introduced a new feature/tab into the Web Module called ‘Dynamic Reports’.

If you’re publishing the same report to the Web Module each day, you can use the Dynamic Reports tab to select a date range and a department (or whatever organizational groups you have defined) and the Web Module will collate all the daily reports that match that filter into one report. This allows you to report on entire week, month or year by simply ‘reporting on reports’, rather than reporting months of raw storage data.

Here’s the full list of changes since the last auto update (2.2.0.32 on the 14th April 2010).

Application Changes

• Added Dynamic Reports feature to the Web Module.
• Rewrote the Web Module transfer protocol. New protocol adds version checking, connection checking, and integrity checking for high latency environments.
• Purge data from storage task no longer prevents importing new hits when all data is removed from an input within a storage.
• IPv6 addresses now show IPv4-mapped addresses as plain IPv4 addresses in summaries.
• IPv6 and IPv4 addresses are now freely interchangable in filter expressions.
• Fixed IPv6 drilldowns on the Summaries screen
• SQL inputs can now be resumed from the previous position. Previously any input that was partially imported would be skipped when importing new hits.
• Template-based analysis has been fixed, no longer results in blank/non-existent analysis.
• Added new string manipulation functions to expression language; Contains, StartsWith, EndsWith, IndexOf.

Loader Changes

• Astaro: Now checks that the ID field is present in a line before attempting to read it.
• Barracuda Web Filter: Added this format to replace Spy Filter.
• BlueCoat Proxy SG W3C: Added support for gmttime, timestamp, x-bluecoat-surfcontrol-is-denied and x-bluecoat-transaction-id.
• ClearSwift: Added a new loader group for ClearSwift that includes the MimeSweeper loaders
• ClearSwift SECURE Web Gatway: Now supported with the Web Appliance loader
• Clearswift Web Appliance: User summary displays Source IP if Username is blank.
• IronPort WSA: Fixed memory usage issues.
• Microsoft FTMG: Added category name lookup to SQL loader.
• Microsoft FTMG: No longer fails to import lines where the rule field contains square brackets.
• Microsoft FTMG: URL Category field is now a string instead of an integer. Added URL Categorization Reason field.
• Microsoft FTMG: Fixed memory usage issues.
• Microsoft IIS W3C: No longer hangs or crashes when loading a file that isn’t IIS W3C.
• NetAsq: Added support for srcname field. The Username summary is populated with user first, and then srcname if user is blank. The User summary is also now populated with Source IPs if the Username summary is blank.

To update WebSpy Vantage, simple select Tools | Check for updates.

To update the Web Module, login to the Web Module server, right-click the WebSpy system tray icon, and select Check for updates.

As always, please contact us if you have any issues or questions.

Unfortunately there are two issues with the Web Module auto update process. Everyone on a 64 bit operating system will encounter issue #1 (Unable to locate installation location), and some of you may encounter issue #2 (System.IO.FileLoadException).

This article describes the errors and how to work around them to successfully update the Web Module.

Issue #1

The usual process to update your Web Module is to log into your Web Module server, right-click the WebSpy system tray icon and select ‘Check for updates’.

If you do this on a 64 bit operating system, you will receive the following error:

Web Module Updater error: Unable to locate installation location

You can fix this issue by clicking Yes, and specifying the Web Module’s installation location, which is usually somewhere under c:\inetpub\wwwroot (or just c:\inetpub\wwwroot if you didn’t specify a virtual directory when installing).

This will allow the updater to continue and you will be prompted to download and install the latest update.

Issue #2

Unfortunately, you may encounter another error during the update installation. The text of the error will be something along the lines of:

System.IO.FileLoadException: Could not load file or assembly ‘ICSharpCode.SharpZipLib, Version=0.84.0.0, Culture=neutral, PublicKeyToken=1b03e6acf1164f73′ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0×80131040)

Work Around

We are currently working on solutions to both of these issues. In the mean time, here is a work around to install the update. On the Web Module server:

1. Download this file:
   http://update.webspy.com/autoupdate/files/vantagewebmodule/vantagewebmodule2.2.0.18.zip
2. Stop IIS (See instructions below).
3. Right-click the WebSpy system tray icon and click Exit.
4. Backup your existing Web Module installation by copying everything in your Web Module’s installation folder (usually under c:\inetpub\wwwroot (or just c:\inetpub\wwwroot if you didn’t specify a virtual directory when installing) into a completely separate location (i.e. don’t keep it in a sub-folder).
5. Extract the downloaded zip file to your web module’s installation folder. Overwrite all the existing files.
6. Start IIS (see instructions below).

Your Web Module will now be updated to the latest version.

Stopping and Starting IIS

In IIS Manager (Start | Control Panel | Administrative Tools | Internet Information Services (IIS)) , right-click the site you want to start or stop, and click Start or Stop

We sincerely apologize for the inconvenience and will hopefully have a solution out soon. If you have any problems with the process above, please contact support at webspy dot com.

When imported into WebSpy Vantage, the result is shown in a new summary called ‘Group’ which you can add to your reports.

We also added support for the custom fields Bytes Sent and Bytes Received. Due to the absence of a header in the IronPort access log, Bytes Received and Bytes Sent fields must both be present to be detected, and the Received field must precede the Sent field.

We also added support for the custom fields Request Body Size and Response Body Size. These fields can be included in your access log by adding %q (Request body size) and %b (Response body size) in the ‘Custom Fields’ edit box. Due to the absence of a header in the IronPort access log, Request Body Size and Response Body Size fields must both be present to be detected, and the Request field must precede the Response field.

We’ve also noticed that the values in the Bytes Sent and Bytes Received fields do not necessarily add up to the value logged for ‘Size’. We’re discussing this issue with our friends at IronPort and we will hopefully post a solution or explanation soon..
The information we first received about these fields indicated they represented Bytes Sent and Bytes Received. This is the way they are represented in the builds below (2.2.0.29). We will release a new build soon, with the field names changed to Request body size and Response body size. Body size is different to bytes sent/received as it does not include bytes from packet headers etc.

We’re yet to issue an automatic update for the Vantage applications, so in the mean time you can download the latest builds here:

Vantage Ultimate:
ftp://ftp.webspy.com/webspy/Builds/VantageUltimate2.2.0.29.zip

Vantage Web Module:
ftp://ftp.webspy.com/webspy/Builds/VantageWebModule2.2.0.8.exe

Vantage Giga:
ftp://ftp.webspy.com/webspy/Builds/VantageGiga2.2.0.29.zip

Vantage Premium:
ftp://ftp.webspy.com/webspy/Builds/VantagePremium2.2.0.29.zip

To apply the Vantage update, close Vantage and extract the downloaded file into Vantage’s installation folder (Usually c:\Program Files\WebSpy\Vantage <flavour> 2.2). Overwrite the existing files.

To apply the Web Module update, uninstall the Vantage Web Module from Add/Remove Programs (Programs and Features in Windows 7/Server 2008), then run the downloaded exe file, making sure you specify the same server, virtual directory and data location that your Web Module was previously using.

We will be releasing this as a public auto-update soon. Let us know if you have any issues.

Check your version in Help | About. If you are running 2.2.0.27 or above, then you already have this update. If not, make sure you update to your software by selecting Tools | Check for updates.