This release will be welcomed with open arms by many customers for the following reasons:

• General usability improvements in the Web Module
  Multi-select / delete options, Ajax progress indicators to avoid page refreshes, export from Dynamics Report tab and more (see below)
• Fixes to the Microsoft Forefront TMG loader
  See my other post: Microsoft Forefront TMG logs size fields the wrong way around. Also fixed ‘value cannot be null’ error when importing SQL logs.
• Fixes to storage corruption issues
  This build should prevent ‘Normalization Index’ storage corruption issues from occurring. This often occurred after importing data, editing some log inputs and reimporting.
• New loaders and more fixes
  See below for the full list

To update your Vantage application, simply choose Tools | Check for updates. To update the Web Module, right-click the WebSpy icon in your system tray and select ‘Check for updates’. If you have any issues updating the Web Module, please see my previous post Web Module Update Errors and Workarounds.

Web Module Changes:

• New: Task progress is now updated without refreshing the page
• New: Added multi-select / delete functionality to Reports, Analyses and Storages tables.
• New: Added export functionality to Dynamic Reports view.
• New: Added Performance section on the Options tab to enabling multi-processing (improves Analysis speed)
• Fix: Dynamic Reports view now supports Trend reports.
• Fix: Organization selector on Dynamic Reports view now always reflects updated data under IE6/7/8.Fix: Fixed javascript errors in IE when expanding the organization filter.
• Fix: Report template names are no longer truncated on the Dynamic Reports view.
• Fix: Fixed errors that may occur when collating reports on the Dynamic Reports page.
• Fix: Authentication errors are now logged with stack trace.

Vantage Changes

• Fixed: ‘Normalization index’ storage corruption problems.
• Fix: Report collation: Added support for collation of Min/Max aggregates on DateTime columns (time of first hit etc). Also added support for arrayed fields (for example, category fields with a comma separated list of categories)
• Fix: Import windows wizard now remembers settings for Import all or selected users
• Fix: Organization: Filtered LDIFs may now be imported when references to some users are missing (for example, if a user’s manager does not exist in the LDIF)
• Fix: Improved connection and error handling between Vantage and the Web Module.

Loader Changes

• New: BlueReef Sonar Total Management Module
• New: Microsoft Sharepoint 2007
• New: SmoothWall Guardian 7.0 format
• New: Sun One Proxy (Supported under Sun One Webserver)
• Fixed: Astaro: Improved format detection
• Fixed: Cisco: Strings in the IP fields of 113019 lines are now imported
• Fixed: IronPort WSA: Improved log format detection
• Fixed: Microsoft Exchange 2007: No longer raises issues regarding total-bytes or internal-message-id fields
• Fixed: Micorosft FTMG (Web) SQL: No longer encounters value could not be null errors
• Fixed: Microsoft FTMG: Added option to reverse bytes received/sent fields. See Microsoft Forefront TMG logs size fields the wrong way around
• Fixed: Microsoft IIS W3C: Now imports cs-method and connection ID
• Fixed: Sophos Web Appliance: Switched the outgoing and ingoing sizes so that they are now the correct way around
• Fixed: Fixed import new hits issue associated with W3C formats. You must reload your logs before this change will take affect. Formats affected include: BlueCoat, Clearswift, Microsoft Exchange 2007, Microsoft FTMG, Microsoft Windows Media Services, WebSpy Live Tracking Log

Enjoy!

The survey only consisted of six questions and to prevent influencing answers and encouraging unique thoughts and opinions, the majority of questions were open ended. Those answers applicable for categorization have now been categorized, compared and correlated. The survey was conducted for internal purposes but I would like to share some of the findings and comments we received.

Vendor devices and log formats

The top ten vendor devices client used WebSpy’s software to analyze and report on:

1. Microsoft ISA Server (a whopping 48.6%)
2. Microsoft Exchange
3. Novell – BorderManager
4. Squid – Proxy Server
5. Microsoft Proxy Server
6. Microsoft – Internet Information Services (IIS)
7. Blue Coat – Proxy SG
8. Check Point – Next Generation
9. IronPort – Web Security Appliance
10. Microsoft – Windows Event Logs

Business Issues

The most common reason why clients started to monitor and report on their log files:

1. To achieve transparency, higher visibility, on overall internet and network resources. (No actual issue quoted)
2. To reduce issues related to lost productivity, inappropriate and illegal internet usage
3. To be able to perform detailed investigations on specific user and site levels
4. To reduce bandwidth cost and increase speed
5. To achieve monitoring requirements not satisfied by vendor device
6. To reduce viruses and other security issues
7. To produce reports required by management
8. To trend and forecast Internet and network usage
9. To investigate email usage
10. To immediately address critical issues using real-time monitoring and alerts

Most valued WebSpy software features

1. Dynamic Drilldowns
2. Ad-hoc analysis and summaries screen
3. Real-time monitoring and alerts
4. Comprehensive and detailed reports and analysis options
5. Predefined and customizable reporting and analysis templates
6. Aliases – Logical grouping of data to represent it more meaningfully
7. Profiles – Categorize web site URLs, email subject lines, instant message chats and any other logged data using customizable keyword profiling technology.
8. Extensive log file compatibility and support
9. Ease of use
10. Automated task scheduling

Suggested Improvements

The majority of answers related to suggested improvements and added software features were very specific to individual user experiences and quite difficult to categorize. However, here a list of suggested improvements that our development has taken to heart and decided to prioritize:

Increase and Improve Number of Default Report and Analysis Templates
Major changes and improvements to our reporting engine and interfaces are currently being planned. A short term goal is to provide the ability to create reports in PDF format. This should hopefully be available in the coming months.

Longer term, we will be incorporating a visual report designer (drag / drop charts and tables etc), as well as the ability to create reports that collate information from different sources. For example, using the information from Event logs, Web proxy logs and email server logs, you could produce a report that shows when a user logged on to their PC, what sites they browsed, what files they accessed, and the emails they sent – all sorted chronologically in the one table!

We currently have a large range of report templates available, but choosing the one you want can be a time consuming process, and you will most likely want to customize the report once you’ve found one that suits. We are planning to make the whole process of creating the report YOU want much, much easier.

Further Expand Profiles (Website Categorization)
When it comes to website categorization, WebSpy applications utilize keyword profiling in addition to importing category fields from log files (if available). There are upsides and downsides to keyword profiling. A simple keyword can instantly categorize thousands of sites which is much more efficient than maintaining a URL Category database. Being able to categorizing hits to your own organization’s website or intranet is also a very useful feature.

We are committed to distributing more frequent profile updates, without overwriting any customizations you have made. Longer term, we plan to provide an option for collaborative keyword customization, so that you benefit from the customization that other customers are making around the world. We also plan to integrate third party categorization services, as there are many great organizations that focus purely on URL categorization.

Improve Software Updates Correspondence
Our main mass communication channels include our newsletter, press releases blog and twitter. Here is where we inform existing and prospective clients about our software, updates, offers and much more. Starting September 2009 we will also send out emails, on a monthly basis, informing existing clients about the specific software updates implemented and are ready to download.

Further Improve Reporting Speed
Vantage 2.2 ranges (Premium, Giga and Ultimate) now use multithreading techniques to utilize the extra processing power on machines with multiple cores or CPUs to import log files and generate reports faster. In addition to the multi-processing improvements recently made in Vantage 2.2, the development team are planning to implement even more multi threading improvements in the next major version. These improvements will be available for users of both the Analyzer and Vantage range.

Thanks again for your participation!

The winner of the USB Camera Pen has been picked using a random draw and will be contacted personally via email.

All three products in the Vantage range (Premium, Giga and Ultimate) have been updated with the following features:

• Multi-processing
  Vantage now utilizes the extra processing power on machines with multiple cores or CPUs to import log files and generate reports faster.
  
  

  The new Performance tab in Vantage

• Improved report styles
  HTML (including MHT) reports are now sporting an updated look and feel.
  
  

  New Report Style

• Updated Aliases and Profiles
  Added support for the latest search engines, social networking sites, operating systems and user agents.
  
  

  Profiles Aliases support the latest Internet tools and technologies.

• Storage Repair Utility
  Storages can become damaged if there is a system crash or if WebSpy.Vantage.exe process is ended when an import is in progress. Damaged storages are now detected automatically and you have the option of repairing them. Storages can also be manually checked and repaired on the Storage properties dialog.
  
  

  Storage Repair Utility

• Save / Open Tasks
  Scheduled Tasks can now be saved to an external file (.Tasks) for backup and migration purposes.
  
  

  Save and Open Tasks

Obviously, the multi-processing feature is the big one! Many of you (Vantage customers) have been running Vantage on multi-core or multi-cpu machines and just dying for Vantage to grab hold of the CPUs and make them work. You should be happy with this new build. Import a folder full of log files and watch 6 to 8 logs import simultaneously instead of watching them import one after the other. Run a full analysis or a comprehensive report with no filters and watch your CPU jump into gear to generate the report in 30% – 50% of the time. We’re expecting this to be a very popular update, especially among our larger customers.

Multi-processing is a great new feature, but be aware that there are some circumstances where Vantage will not be able to utilise more CPU power when generating reports. For example, if your report contains filters that significantly cuts down the amount of data in your storage that needs analysis, Vantage’s analysis engine and your CPUs will not be significantly pushed. Also see my earlier blog for more information on how the report structure can also affect the performance.

To get hold of this build, simply download the latest version of Vantage from our website. It will run side by side with your existing Vantage 2.1 installation, which minimizes any downtime, and makes it easy to transfer all your settings across (use the Migration Wizard in the Tools menu). Then you can uninstall Vantage 2.1 when you’re ready.

Let us know how you go by leaving comments below, or by tweeting us.

Cheers!
Scott

Of particular note is a fix to the Vantage range:

• Fix: Improved partition matching when filtering by a date range

This will benefit you if you’re running reports with a date range filter such as last week or yesterday (which is pretty much everyone), and are using the default storage partitioning scheme of “Date” (again… pretty much everyone).

There was an issue in Vantage’s partition filtering technology which meant that if you specified a date range filter, Vantage would still analyse an entire storage, instead of just the date partitions you selected in the filter.

So do a Tools | Check for updates to grab build 2.1.2.12 and you should see a significant speed improvement in report generation. Woo hoo!

Here’s more details on the changes:

VANTAGE

Application Changes

• New: Added support for importing event logs from non-domain machines.
• New: Added the ability for the Troubleshoot Alias/Profiles to export results to a file.
• New: Added hash salt and substring functions to the expression language.
• New: Added support for computing a date-modified file mask at import time and storing it back into the storage.
• New: Added a file mask override to the Import new hits to existing storage task action.
• New (Vantage Ulitmate & Giga): Added support for attributes on Organization groups.
• New (Vantage Ultimate): Added custom expression-based split in Web Module publishing (Beta).
• New (Vantage Ultimate): Changed the publish report wizard to allow selection of multiple storages.
• Fix: Improved partition matching when filtering by a date range.
• Fix: Fixed an issue with registrations and trial extensions.
• Fix: Fixed zero-width rectangle exceptions in chart renderer.
• Fix: Fixed overflow exceptions on footer generation in report tables.
• Fix: Fixed XmlException error during settings load.
• Fix: Storage now clears previous data when it is overwritten when using the ‘Import logs into new storage’ task.
• Fix (Vantage Ultimate & Giga): Fixed issues with date range selection when collating reports.
• Fix (Vantage Ultimate): Web Module dock no longer checks for updates to a web module server when ‘Check for updates on startup’ is disabled.

Loader Changes

• New: Added Phion Firewall
• New: Added Watchguard Firebox X Core
• New: Added NetScreen 208
• New: Added IPCop
• New: Added Astaro Mail Gateway
• New: Added iPrism Monitor v4.2
• New: Added Cisco VPN Concentrator
• New: Added Snare for Lotus Notes
• Improved: Added string host name to Kerio Mail Server
• Improved: Improved support for NetIntact PacketLogic
• Fix: Made changes to the IronPort detection method to fix the issue when importing via FTP.
• Fixed: Fixed date format issue in event log import. You can now import event logs in any regional configuration.
• Fix: Postfix loader no longer drops session state after the first recipient line
• Fix: Updated the detection method in CheckPoint Firewall-1 Syslog format.
• Fix: Fixed an issue in iSheriff where a drilldown on the Category field would display no results.
• Fix: Various fixes to Netscreen 10
• Fix: Various fixes to Arkoon PxLog
• Fix: Vaious fixes to Sendmail MTA

VANTAGE WEB MODULE

• FIX: When sorting by key column the chart will now use the “hits” aggregate for value (prevents the chart from going wacky)

ANALYZER

Application Changes

• NEW: Added support for manual configuration of ISA block list settings (Tools | Options | Block Lists)
• Fix: Fixed an issue with registrations and trial extensions
• Fix: Fixed report wizard configuration to restore the sort column from the report template

Loader Changes

• New: Added Webroot
• New: Added Qmail Desknow Mail Server
• New: Added UserGate Proxy Server 2.7
• New: Added Netgear FVX538
• Improved: Added support for importing allowed/blocked status in Sophos Web format
• Fixed: Changed CC Proxy field count check
• Fixed: Changed ISA SQL loaders to use size delta fields instead of cumulative fields
• Fixed: Modified Exchange 2000/2003 loader to discard message cache after import

LIVE

Application changes

• Fix: Fixed an issue with registrations and trial extensions

Loader changes:

• New: Added Webroot
• New: Added Qmail Desknow Mail Server
• New: Added UserGate Proxy Server 2.7
• New: Added Netgear FVX538
• Improved: Added support for importing allowed/blocked status in Sophos Web format
• Fixed: Changed CC Proxy field count check
• Fixed: Modified Exchange 2000/2003 loader to discard message cache after import