An issue was introduced in build 2.2.0.42 when a change was made to the underlying data type of the Url Category field in the Microsoft TMG Web Schema. Unfortunately this change resulted in nulls being imported into the Url Category summary when importing from TMG’s SQL log files. Fortunately, importing W3C text logs were unaffected by this change.

This issue has now been fixed and released as an auto-update. To update your software simply select Tools | Check for updates.

If you have existing storages that have not been importing the URL Category, go to the storages screen and click ‘Reload All’. This will re-import your log files and populate the Url Category summary.

For all the customers affected by this issue, thank you for your patience and understanding.

This release will be welcomed with open arms by many customers for the following reasons:

• General usability improvements in the Web Module
  Multi-select / delete options, Ajax progress indicators to avoid page refreshes, export from Dynamics Report tab and more (see below)
• Fixes to the Microsoft Forefront TMG loader
  See my other post: Microsoft Forefront TMG logs size fields the wrong way around. Also fixed ‘value cannot be null’ error when importing SQL logs.
• Fixes to storage corruption issues
  This build should prevent ‘Normalization Index’ storage corruption issues from occurring. This often occurred after importing data, editing some log inputs and reimporting.
• New loaders and more fixes
  See below for the full list

To update your Vantage application, simply choose Tools | Check for updates. To update the Web Module, right-click the WebSpy icon in your system tray and select ‘Check for updates’. If you have any issues updating the Web Module, please see my previous post Web Module Update Errors and Workarounds.

Web Module Changes:

• New: Task progress is now updated without refreshing the page
• New: Added multi-select / delete functionality to Reports, Analyses and Storages tables.
• New: Added export functionality to Dynamic Reports view.
• New: Added Performance section on the Options tab to enabling multi-processing (improves Analysis speed)
• Fix: Dynamic Reports view now supports Trend reports.
• Fix: Organization selector on Dynamic Reports view now always reflects updated data under IE6/7/8.Fix: Fixed javascript errors in IE when expanding the organization filter.
• Fix: Report template names are no longer truncated on the Dynamic Reports view.
• Fix: Fixed errors that may occur when collating reports on the Dynamic Reports page.
• Fix: Authentication errors are now logged with stack trace.

Vantage Changes

• Fixed: ‘Normalization index’ storage corruption problems.
• Fix: Report collation: Added support for collation of Min/Max aggregates on DateTime columns (time of first hit etc). Also added support for arrayed fields (for example, category fields with a comma separated list of categories)
• Fix: Import windows wizard now remembers settings for Import all or selected users
• Fix: Organization: Filtered LDIFs may now be imported when references to some users are missing (for example, if a user’s manager does not exist in the LDIF)
• Fix: Improved connection and error handling between Vantage and the Web Module.

Loader Changes

• New: BlueReef Sonar Total Management Module
• New: Microsoft Sharepoint 2007
• New: SmoothWall Guardian 7.0 format
• New: Sun One Proxy (Supported under Sun One Webserver)
• Fixed: Astaro: Improved format detection
• Fixed: Cisco: Strings in the IP fields of 113019 lines are now imported
• Fixed: IronPort WSA: Improved log format detection
• Fixed: Microsoft Exchange 2007: No longer raises issues regarding total-bytes or internal-message-id fields
• Fixed: Micorosft FTMG (Web) SQL: No longer encounters value could not be null errors
• Fixed: Microsoft FTMG: Added option to reverse bytes received/sent fields. See Microsoft Forefront TMG logs size fields the wrong way around
• Fixed: Microsoft IIS W3C: Now imports cs-method and connection ID
• Fixed: Sophos Web Appliance: Switched the outgoing and ingoing sizes so that they are now the correct way around
• Fixed: Fixed import new hits issue associated with W3C formats. You must reload your logs before this change will take affect. Formats affected include: BlueCoat, Clearswift, Microsoft Exchange 2007, Microsoft FTMG, Microsoft Windows Media Services, WebSpy Live Tracking Log

Enjoy!

The log databases are stored in an SQL Express instance named MSFW. By default these databases cannot be accessed by a remote computer. I’d first like to say that we recommend changing TMG’s logging to W3C text files, as these logs are about 5-6 times faster to import, and you don’t need to worry about the steps below.

But if you need to stick with the SQL Express logging, here are the basic steps to enable access to the logs from a remote computer:

Enable TCP access to the MSFW instance

To do this:

1. Log into your Forefront TMG server using administrator credentials.
2. Select Start | All Programs | Microsoft SQL Server 2008 | Configuration Tools | SQL Server Configuration Manager.
3. Expand SQL Server Network Configuration and select Protocols for MSFW
4. Right-click TCP/IP and select Enable
5. Click OK on the Warning dialog informing you that “changes will not take effect until the service is stopped and restarted.”

Enabling TCP/IP on the MSFW instance

Set the listening Port on the MSFW instance

Once TCP/IP is enabled on the MSFW instance, you need to set it to listen on port 1433

1. Select Protocols for MSFW under SQL Server Network Configuration
2. Right-click TCP/IP and select Properties.
3. Click the IP Addresses tab and scroll to the IPAll section at the bottom of the list.
4. Change the TCP Port to 1433 and ensure nothing is entered in TCP Dynamic Ports (Delete the ‘0′ value  if present). Click OK and click OK on the Warning dialog.

Setting the Port on the MSFW instance

Change the listening port on the ISARS instance

The ISARS SQL instance also listens on port 1433 and this can cause connection issues. Change this instance to use port 1434:

1. Still in SQL Server Configuration Manager, select Protocols for ISARS under SQL Server Network Configuration
2. Right-click TCP/IP and select Properties.
3. Click the IP Addresses tab and scroll to the IPAll section at the bottom of the list.
4. Change the TCP Port to 1434 and ensure nothing is entered in TCP Dynamic Ports. Click OK and click OK on the Warning dialog.

Changing the port on the ISARS instance

Restart the Services

For the above changes to take effect, you need to restart the SQL Server (ISARS) and then the SQL Server (MSFW) services in that order.

1. Go to Start | Administrative Tools | Services
2. Right-click the SQL Server (ISARS) service and select Restart.
3. Right-click the SQL Server (MSFW) service and select Restart.

Test the connection from the WebSpy machine

You should now be able to connect to the MSFW databases from a remote computer. To test the connection, we recommend that you install SQL Management Studio on the machine running WebSpy and try to connect to <TMGservername>\MSFW, 1433 (replace <TMGservername> with your actual server name or IP address). For example TMGServer\MSFW, 1433 or 192.168.0.10\MSFW, 1433.

As long as you are logged into Windows with a user account that is a local administrator on the TMG server, you should be able to connect without issue.

Importing the TMG Log files into WebSpy Vantage

Once you have established a connection, you can import your logs using WebSpy Vantage like so:

Create a new Storage

Select Database Connection

Select the Microsoft FTMG Loader

Click Add

Enter TMGServer\MSFW and port 1433

Successfully Imported WebProxy Logs

The screenshots above also illustrate using a database mask of *WEB* to only import the WebProxy logs. If you only want to import the Firewall logs, set the database mask to *FWS*. If you want to import both the WebProxy and Firewall logs, leave the database and table masks set to *.

Now that you have your log files imported, you can run a quick ad-hoc analysis on the Summaries screen or generate any of Vantage’s default web of firewall reports. M

Make sure you also download our Forefront TMG specific Aliases and report template. For more information, see our Forefront TMG How To page.

If you have any questions or encounter any hurdles, please leave a comment below.

If you’re considering upgrading your ISA Server to TMG, this means that you can start your deployment using the Release Candidate, and simply switch it to a licensed version with no additional configuration changes once the full release is available. At least, that is what Vladimir Holostov (Lead Program Manager, Release Manager for Forefront TMG 2010) states on the Forefront TMG (ISA Server) Product Team Blog:

“The final product will be released later this year and you can expect it to behave exactly like the Release Candidate. You can install Forefront TMG 2010 RC today and upgrade to a licensed version once available without changing the configuration of your deployment.”

To offer some peace of mind for organizations considering the deployment, Vladimir also mentions that “Forefront TMG 2010 RC is deployed at three major Microsoft sites located around the world in Haifa, Bellevue and Redmond. More than 20,000 employees are already protected by TMG and these deployments have already accumulated more than 5,000 hours of runtime, performing extremely well under heavy load”.

No major features have been added to the Release Candidate since Beta 3, however there have been improvements geared around tightening up security, reliability and performance and telemetry. For more information about the release candidate, please visit the
Forefront TMG (ISA Server) Product Team Blog.

You can also download the release candidate here

I mentioned in my last blog posting that WebSpy has introduced support for reporting on Microsoft Forefront TMG log formats in the Vantage product range. To try it out, please make sure you have installed Vantage 2.2 (any flavour – Premium, Giga or Ultimate), and then select Tools | Check for updates to download build 2.2.0.10 or above. You can then import your TMG log files by selecting the Microsoft FTMG loader in the import wizard.

Importing Microsoft Forefront Threat Management Gateway Log Files

We’re very interested to hear your thoughts on the reporting functionality, so please go ahead and give it a go!

You should be prompted to update your software on startup, but if you’ve turned off that feature, simply go to Tools | Check for Updates.

New Features

This new build sports the following new features:

• Support for Microsoft Forefront Threat Management Gateway (Beta)
  Microsoft Forefront Threat Management Gateway (FTMG) is still currently in Beta, and is due to be released around November 2009. For those that do not know, FTMG is the next version of Microsoft’s popular ISA Server. Information and downloads for FTMG can be found here http://www.microsoft.com/forefront/edgesecurity/isaserver/en/us/tmg-beta.aspx. We have added support for FTMG beta 2 and 3 for both the W3C text logs (recommended) and the internal SQL Server Express Database logs. If you are currently trialling FTMG, we are very interested to hear your feedback. Let us know how you go!
  

  Now Supported - Microsoft Forefront Threat Management Gateway

• Data purge
  You can now purge data from a storage, and schedule this purge to occur on a regular basis using Tasks. Purge options include data between a date range, data before a date, data after a date, data older than a date relative to now, and all data. This feature will let you easily maintain a single storage that only includes data for the last month or day.
  

  Options for Purging data from your storage

• Import Organization from CSV can now be scheduled using Tasks
  If you are importing your organizational structure from CSV, you can now schedule this action using Tasks. This enables you to update your organizational structure before any reports are run.
  

  Import Organization from CSV via Tasks

• Added Support for ExoServer Web
  If you’re running ExoServer Web, you can now analyze it’s logs using WebSpy Vantage.

Fixes

We also fixed some things that may have been bugging you:

• Improved the start time for the application by improving the logic to check for Storage damage.
• Fixed the IronPort loader (Fixed out of range issues on excessive size fields).
• “Having” filters no longer override the sort order of a Report Template node.
• Fixed an issue that may result in duplicated storages after migrating settings from earlier versions.
• Fixed the inability to remove invalid entities from web module permissions list (users that no longer exist).
• Fixed a timeout issue when publishing storages to the web module.

Why are you still reading? Go update now!

Have fun!

Well, it’s now been released and can be downloaded from the Microsoft Download Center:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=e05aecbc-d0eb-4e0f-a5db-8f236995bccd

One of the major improvements in Beta 3 is URL filtering which leverages Microsoft Reputation Services (MRS). With regards to this, Microsoft says:

“At the time of this release, the MRS database content is being populated and updated continuously as part of the initial beta service offering. As this process continues, URL filtering categorization accuracy and comprehensiveness will increase. A telemetry package designed for improving the quality of URL filtering database and collecting your feedback is planned to be released soon. Please check back for updates in August.”

Support for importing Microsoft TMG log files into your favorite WebSpy product is coming soon so stay tuned! Subscribe to the WebSpy blog RSS feed or follow us on twitter if you want to be notified as soon as it’s available.

According to the latest blog from TMG’s Product Unit Manager, David B. Cross, Beta 3 will be released in the next couple of weeks. As for the full release, David says that they are still on track for Q4 this calendar year.

Beta 3 will introduce URL filtering that is ‘fully integrated’ with TMG’s web policy rules, and also utilizes Microsoft Reputation Services.

Microsoft are also introducing Intrusion Prevention and Detection (IPS/IDS) capabilities in TMG. These systems will utilize a technology they’re calling Network Inspection System (NIS) that detects attacks using signatures of known vulnerabilities, downloaded from the Microsoft Malware Protection Center. For more information on NIS see http://blogs.technet.com/isablog/archive/2009/04/12/exercising-nis-with-test-signature.aspx

If you’re currently using ISA 2004 or 2006, upgrading to TMG will consist of exporting rules and settings from ISA, then importing them into a clean installation of TMG. TMG will also only run on Windows Server 2008.

Improving the on-box reporting has not been a focus for the TMG development team, so analyzing TMG’s web proxy and firewall logs is still the best way to go for in depth reporting.

If you’re interested in reporting on your TMG log files stay tuned! We’re currently implementing support for the SQL Express, W3C and Native text logs. WebSpy Vantage is likely to be the first application to include the feature, with Analyzer and Live soon to follow.

All going well, you can expect to see TMG support in your favourite WebSpy app within the next month or so. If you want to be notified once we’ve added support, just leave a comment below.

Cheers!
Scott.