Richard has been working with Forefront Threat Management Gateway (TMG) 2010 and its predecessors for more than 12 years. He has designed and deployed network security solutions using TMG and ISA for SMB’s, military and defense organizations, and Fortune 500 companies around the world.

In addition to his isaserver.org blogs, Richard has his own ISA/TMG blog where he recently posted some useful tips on changing WebSpy Vantage’s scheduled task recurrence interval using the schtasks.exe command line tool. Adding more frequent import options (i.e. hourly) is on the product roadmap but until then, using the command line tool is a great alternative.

We do recommending visiting tmgblog.richardhicks.com – brimming with ISA Server and TMG information and tips, here’s just some of the latest blogs:

• Changing WebSpy Vantage Scheduled Task Recurrence Interval
• Websense Integration Support for Forefront Threat Management Gateway (TMG) 2010
• Load Balancing and Forefront TMG Firewall Clients
• How to Slipstream Service Pack 1 for TMG

You should be prompted to update your software on startup, but if you’ve turned off that feature, simply go to Tools | Check for Updates.

New Features

This new build sports the following new features:

• Support for Microsoft Forefront Threat Management Gateway (Beta)
  Microsoft Forefront Threat Management Gateway (FTMG) is still currently in Beta, and is due to be released around November 2009. For those that do not know, FTMG is the next version of Microsoft’s popular ISA Server. Information and downloads for FTMG can be found here http://www.microsoft.com/forefront/edgesecurity/isaserver/en/us/tmg-beta.aspx. We have added support for FTMG beta 2 and 3 for both the W3C text logs (recommended) and the internal SQL Server Express Database logs. If you are currently trialling FTMG, we are very interested to hear your feedback. Let us know how you go!
  

  Now Supported - Microsoft Forefront Threat Management Gateway

• Data purge
  You can now purge data from a storage, and schedule this purge to occur on a regular basis using Tasks. Purge options include data between a date range, data before a date, data after a date, data older than a date relative to now, and all data. This feature will let you easily maintain a single storage that only includes data for the last month or day.
  

  Options for Purging data from your storage

• Import Organization from CSV can now be scheduled using Tasks
  If you are importing your organizational structure from CSV, you can now schedule this action using Tasks. This enables you to update your organizational structure before any reports are run.
  

  Import Organization from CSV via Tasks

• Added Support for ExoServer Web
  If you’re running ExoServer Web, you can now analyze it’s logs using WebSpy Vantage.

Fixes

We also fixed some things that may have been bugging you:

• Improved the start time for the application by improving the logic to check for Storage damage.
• Fixed the IronPort loader (Fixed out of range issues on excessive size fields).
• “Having” filters no longer override the sort order of a Report Template node.
• Fixed an issue that may result in duplicated storages after migrating settings from earlier versions.
• Fixed the inability to remove invalid entities from web module permissions list (users that no longer exist).
• Fixed a timeout issue when publishing storages to the web module.

Why are you still reading? Go update now!

Have fun!

If you’re using WebSpy Vantage, you are probably interested in filtering your log file imports by date (only import files from the month of June for example). The obvious way to do this is to specify a date filter using the filters page in the Input Wizard. The problem is Vantage will still check every record in every log file being imported to see if it matches the date filter. If you have months or years worth of logs in the folder being imported, that’s a lot of data that Vantage has to pointlessly sift through.

The good news is, if your log files contain the date in their file name, then you can use file masks to instruct Vantage to never touch these unwanted files.

A bit about file masks…

You can specify file masks such as *, *.log, *.gzip, *WEB*.w3c, etc to import logs with specific file extensions, or with specific strings in the file name (such as WEB or FWS to import only Microsoft ISA Web Proxy or Firewall logs respectively).

But if your log file contains the date in the file name, you can also use date modifiers in the file mask to select logs from a particular month, date or year.

Say you have log files that look like this:

• 20090801.log

• 20090802.log

• 20090803.log

and so on..

You can create a simple file mask to only import log files from the month of August very easily using 200908*, or 200908*.log.

Using date modifiers in file masks

But if you’re using Tasks to automatically create a new storage each month, you don’t want to have to worry about manually changing the file mask to 200909*.log when the first day of the next month rolls around.

So intsead, you can use a date modifier in the file mask that will automatically select the logs for the current month, every time your task runs. For the above example, the file mask looks like this:

• %[yyyyMM]* (you can also use %[yyyyMM]*.log)

When the task runs, %[yyyyMM] will be replaced with actual values from the current date. So if the task runs on the 1st of August 2009, the file mask will become 200909* (or 200909*.log).

Dealing with different date formats

You can also use date modifiers for log files that look like this:

• 2009-Aug-01.log

• 2009-Aug-02.log

• 2009-Aug-03.log

In this case the file mask looks like:

• %[yyyy-MMM]* - notice the three MMM's as opposed to two MM's used previously.

Vantage uses the custom date and time format strings available in the .NET framework, so for more information on whether to use m or M or MMM, please refer to this article http://msdn.microsoft.com/en-us/library/8kb3ddd4.aspx

Importing logs from previous months

If you would like to import logs from a previous month, this can also be done by adding an additional element to the date modifier. For example, to import the previous months logs you can use:

• %[-1m,yyyyMM]*

Notice the -1m meaning ‘minus one month’. You can also use -1d (for minus one day), or -1y (for minus one year).

More examples

Here are some more examples to give you an idea of what is possible using date modifiers.  Assuming the date is 14th of August 2009:

• %[-1y,yyyyMM]*.log will create a file mask of 200808*.log

• %[yyyy-MM-dd]*.log will create a file mask of 2009-08-14*.log

• %[-4d,yyyyMMdd]*.log will create a file mask of 20090810*.log

• %[1-m,-4d,yyyyMMdd]*.log will create a file mask of 20090710*.log

• %[-1y,1-m,-4d,yyyyMMdd]*.log will create a file mask of 20080710*.log

• ISALOG_%[-1m,yyyyMM]*_WEB_*.w3c will create a file mask of  ISALOG_200907*_WEB_*.w3c

• *%[-1m,yyyyMM]* will create a file mask of  *200907*

Adding a file mask

File masks are configured on the Input Selection page of the Input Wizard, when you select Add | Folder.

Adding a File Mask

There is also an option to save the literal date into the file mask when the task is run.  For more information on this option, please see my previous blog about this feature.

Other uses for date modifiers

Date Modifiers are a great way to speed up log file imports, but you can also use them when specifying storage names as well as report names. For example, if you specify a storage name of %[yyyyMM]_storage, this will create storages with the names 200907_storage, 200908_storage and so on. When selecting the storages to report on, you can click the Add button on the storage selection toolbar in the Report Wizard, and specify storages such as %[-1m,yyyyMM]_storage, to report on the previous month’s storage.  For more information, please see Automatic Importing and Reporting using Tasks.

I hope this helps someone out there. Let me know how you go!

A common setup is to have a task that creates a storage each month, then a separate task that runs at the end of each day – say at 10pm, to imports new hits into the storage.  The issue here is that at the end of the month, hits between 10pm and 12pm will never be imported, as the next time the task runs it will be dealing with next month’s storage.

As long as your log files contain the date in the file name, you can use date modifiers in file masks to import the appropriate log files. Date Modifiers are used to create file masks such as ‘access_200902*.log’, or ‘access_20090203.log’ that will cull down the number of log files to import. This is much faster than using date filters (specified on the filters page).

The Issue:

The issue is that a log file location in a storage can only have one file mask, and this cannot (until now) not be modified by task actions.  So you cannot specify a new task on the first of each month to import yesterday’s log file into last month’s storage, if that storage already has a file mask to only import the current day’s log file.

The two solutions:

1. Save the literal file mask into the storage when the storage gets created.

This option allows you to specify a file mask such as  access_%[yyyyMM]* in the ‘Import logs into new storage’ task, but when each storage is created, the actual file mask that gets saved into the storage includes the literal month and year values such as access_200902*

This way you can specify an ‘Import new hits’ job, and even if that job runs in the next month,  ONLY log files from the desired month will be checked for new hits.

To access this option:

1. Go to the Tasks screen and edit your ‘Import logs into new storage’ action
2. Go to the Input Selection page
3. Select your log folder (not individual files) and click Edit
4. Check the ‘Save literal date into mask’ check box


Save literal date into file mask option

Downside of this option is that in my ‘monthly storage’ example, all 31 log files for the month will be checked for new hits.

2. Override a storage’s file mask using the Import New Hits task action.

The Select Storage dialog that appears when configuring an ‘Import new hits into existing storage’ action now has a new option to override a storage’s file mask when the task runs.



Override the file mask option

This allows you to configure a task that runs on the first day of each month to import only yesterday’s log file into last month’s storage (need to add last month’s storage using the Add button, and specify the appropriate date modifier to access it – e.g. “%[-1m, yyyyMM] My Monthly Storage”).

This will only check one file for new hits and will finish the import job faster than option 1 (using the monthly storage example).