This video will cover the Organization section of the software, showing you how to import your users and groups from Active Directory / LDAP.

This video follows on from #3, and will cover the Profiles section of the software, and how to use site categorization.

In this video we’ll look at how to import some data into a storage, use the Summaries section, and start customizing Aliases.

This video will take you through the system requirements, and the installation of Vantage and the Web Module.

This video is a high level overview, giving you a general insight into the software and its different parts.

WARNING: With storage locking disabled, it is possible to import into a storage while running a report, and doing this may cause storage corruption. It is therefore very important if you decide to enable these features to ensure that a storage is not written to while running reports.

Due to the experimental nature of these features, they can only be enabled by including a config file next to Vantage’s executable. To enable multi-instance capabilities and disable storage locking:

1. Download the following config file:
   WebSpy.Vantage.exe.config
2. Close Vantage
3. Extract downloaded zip file into Vantage’s installation folder (usually c:\Program Files (x86)\WebSpy\Vantage <flavour> 2.2). If you already have a file of the same name in that location, make a backup of it before overwriting it with the  new file.
4. Run Vantage.

You can now run Vantage again to launch another instance of the application.

Be aware of:

Simultaneous reading and writing, and multiple writes

I just want to be very clear that if you run reports while importing, or import into the same storage simultaneously, storage corruption can occur. Storages are not designed to be unlocked for these reasons. The only reason we’ve provided this ability is so that you can READ from the a single storage  simultaneously (i.e. run two or more reports). Reading and writing, and multiple writing is NOT supported, but Vantage will attempt to do it if you ask it to, with undefined behavior.  Check your Tasks configuration and note when any import jobs are likely to occur to avoid running reports at these times.

Configuration Changes

When Vantage closes it writes all of it’s state to a series of files under c:\users\<user profile>\AppData\Roaming\WebSpy\Vantage <flavour> 2.2). When Vantage opens, it loads these files into memory. When  running multiple instances, these instances will be reading and writing the same files. So if you open two instances of Vantage, make a change to a report template in one instance, then close the application, the Vantage.Templates file will be updated. But when you close the second instance of the application, the Vantage.Templates file will be overwritten with a version that doesn’t include the change.

When making configuration changes (templates, tasks, aliases, organization etc), make sure only one instance is running (check Task Manager for the WebSpy.Vantage.exe process).

It’s Experimental!

There may be other undefined behaviors that we are yet unaware of, so we advise running this configuration in a test environment.

We’re providing these feature on an “as-is” basis, meaning we will not be providing technical support for issues that arise as a result. That said, we are certainly interested to hear about any issue to help us improve the feature.

Let us know how you go!

You can also read through these steps on this page: Analyzing SonicWALL log files with WebSpy.

Creating and Importing SonicWALL log files

Analyzing SonicWALL log files

We intend to make some SonicWALL specific report templates available on our SonicWALL how to page soon.

Until then, feel free to create your own templates, or modify our existing web reports to include the extra goodies contained in the SonicWALL logs.

TIP: To modify an existing web report, right-click the report and choose ‘Duplicate template’. Then choose the “SonicWall Web” schema. You’ll then have a report template that you can modify to include all the SonicWALL summaries, such as Categories, and Source and Destination Interface.

If you need some assistance getting the report(s) you need, feel free to contact me, or support@webspy.com.

The next morning I woke to find several more messages from my VPS host, each with a higher and more significant excess transfer than the last. At this point it occurred to me that it was unusual for my VPS to reach its quota, let alone exceed it. The excess transfer was now enough that it was going to incur significant cost, so I set about investigating the cause.

I downloaded some firewall logs for the previous few days from the server and imported them into Vantage. The first place I looked was in an analysis at the “Source Address” summary, to see where the activity was coming from. What I found was a single host with a disproportionately larger amount of transferred data than the other addresses listed, so I drilled down to the “Destination Port” summary for this source address to see what services it was accessing. I found that all the traffic was going to port 53 – my DNS. More accurately, the large amount of data was going from my DNS to the source address. Drilling down to the “Individual records” view then showed that my server was providing a large response to a small DNS request from that source address – about 20 times per second.

Curious about why this single machine somewhere on the Internet was bombarding my server with small DNS requests at such a high rate, I set my server’s firewall to deny packets from that address and began searching around online for any information.

I quickly found out that I hadn’t configured my DNS properly, and it was set to allow recursive requests, meaning that if a request came in for a domain my server wasn’t authoritative for, it would then forward the request to another DNS that could answer, or given a blank request it would respond with the full list of root servers. Running tcpdump on the VPS revealed that every request coming in was blank, and my server was responding with the full list of root servers for each request.

It still seemed odd that a server would be constantly sending small requests to my server and receiving large responses. Then it dawned on me; I was looking at a Distributed Reflected Denial of Service (DRDoS) attack. The source address in all the requests I had looked at was forged by the attackers, so that my server – and many other servers out there also receiving the requests – would send their responses to the forged source address in an attempt to flood its connection. The source address in my firewall logs was the target of the attack. I found more information about this specific type of attack here.

Having disabled recursion on my DNS, my server’s contribution to the attack was significantly reduced. However, my server was now responding with a much smaller “request denied” packet for each incoming request. I wanted some way of preventing my DNS from responding at all, so again I headed out to the Internet to see what I could find.

I discovered a package called “fail2ban”, which dynamically updates your firewall rules to block addresses that are abusing your server’s services. I installed it using this guide, and immediately my bandwidth usage dropped off as it blocked further DNS requests. Even now the requests are still flooding in, but now my VPS contributes only a handful of packets towards the attack instead of the previous millions per day.

Take a look at our dedicated Astaro pages to get an idea of what can be achieved when analyzing Astaro Web Gateway log files with WebSpy Vantage.

I’ve created some quick videos to show you how to enable the correct logging options on the Astaro Security Gateway appliance, how to import these log files into Vantage, and analyze the data on the Summaries screen.

Configure Logging

The best way to configure logging is to setup a 3rd party syslog server (such as Kiwi Syslog) on a machine in your network, then configure the Astaro Security Gateway to send syslog messages to that server. The syslog server then creates log files that can be imported into WebSpy Vantage. This video takes you through that process.

Importing and Analyzing Astaro logs

Once you have successfully configured syslogging on your Astaro Security Gateway, you can import the log files into WebSpy Vantage and analyze activity on the Summaries screen. This video takes you through that process.

We intend to make some Astaro specific report templates available on our Astaro How To page soon.

Until then, feel free to create your own templates, or modify our existing web reports to include the extra goodies contained in the Astaro logs.

TIP: To modify an existing web report, right-click the report and choose ‘Duplicate template’. Then choose the “Astaro Security Gateway – Filter with category” schema. You’ll then have a report template that you can modify to include all the Astaro summaries, such as Actions and Categories.

If you need some assistance getting the report(s) you need, feel free to contact me, or support@webspy.com.

Even though the webinars are channel focused I thought I’d share the most recent one with all of you. It includes very interesting product demos from Mark Maciw, product manager at Clearswift, and Scott Glew, Product Operations Manager at WebSpy. By seeing the products in action you’ll get a greater understanding of their capabilities and complementing aspects and hopefully learn how to:

• Maximize Internet investment, employee productivity and enjoy the benefits of a web-enabled environment;
• Reduce security vulnerabilities, to successfully protect organizational assets and employees

Enjoy!

Download Clearswift’s Research Report: Web 2.0 in the Workplace Today

Cyber bullying is legally defined as repeated harassment online, although in popular use, it can describe even a sharp-elbowed, unwarranted swipe online. We all know kids can be cruel because they often lack the maturity and empathy to understand the emotional ramifications their words or actions can have on others. Adding the anonymity of the Internet, cyber bullying can be more psychologically savage than schoolyard bullying. The Internet erases inhibitions and adolescents often take the bullying much further online than in person.

The article describes a number of bullying cases played out in the US over the last few years. It also goes into detail on how the bullies, victims, parents, schools, and the authority responded in each instance.

D.C. and the forged Facebook profile

The main story focus on Marie and her son D.C. The kids at school started to avoid D.C. as he allegedly was posting horrible comments about other kids on his Facebook profile. As a matter of fact D.C. didn’t even have a Facebook account and it turned out someone had forged his identity on Facebook, and was bullying others in his name. Marie was desperate to make it stop as the ongoing online bullying had detrimental effect on, not only her son, but also the kids targeted on D.C’s forged Facebook account.

When D.C’s mum contacted school officials to help track down the students who was making her son miserable she was told there was nothing they could do. It was an off-campus matter.

Finally, after months and months of continued harassment, the police was able to subpoena Facebook for the address of the computer linked to the forged profile and much later subpoena Comcast, the Internet service provider, for the home address of the computer’s owner. Three boys were identified to be behind the scheme.

The Common Thread

The lawlessness of the Internet, its potential for casual, breathtaking cruelty, and its capacity to cloak a bully’s identity all present slippery new challenges for kids and parents. This is a dark, vicious side of adolescence, enabled and magnified by technology.

The article covered a variety of online bullying scenarios and the common thread running through all of them is the parents’ helplessness and frustration about the school’s inability and reluctance to intervene and proactively protect the students.

Yes, a large chunk of the responsibility lies with the parents. It even starts outside the borders of technology by simply raising mindful and responsible children. However, even if parents got that part right, in addition to restricting, monitoring, discussing, and educating themselves and their children about safe and responsible Internet use, the schools also have responsibility to ensure their internet resources are not used for cyber bullying. The negligent approach of passing off D.C’s case as an off-campus matter is simply unacceptable. If even one of the malicious Facebook updates happened during school hours, on the schools network, identifying the culprits could have taken a few minutes, as opposed to months!

What the school should have done

Majority of schools require students to use individual login details to access the Internet. This means an IT administrator could have easily match students’ Facebook activity with the timing of the malicious Facebook updates on D.C’s public profile and quickly narrowed down the list of suspects. Even better, as Facebook generates a specific URL every time someone updates their Facebook profile, a report filtering out all other Facebook activity would have been an even more efficient option.

Here’s how you filter based on Facebook updates using Vantage:

Let’s say you want to track down the source of malicious Facebook updates made during a specific time period.

• Make sure you have access to the log files created during the period you which to investigate. You either need to import them to a new storage or apply a date filter when running a new analysis.
• In the ‘Summaries’ screen click ‘New Analysis’ and make sure your storage is selected.
• Click through the wizard and add a date filter if your storage include dates you are not interested in investigating.
• In the ‘Filters’ section click ‘Add’ and select ‘Field value filter’.
• In the ‘Summary’ drop down select ‘Site URL’.
• Click ‘Add’ and enter value: http://www.facebook.com/ajax/updatestatus.php (This is the URL recorded every time someone updates their facebook profile).

   

  

   

• Click OK in all wizard windows until the new analysis starts running.
• Once the new analysis has been completed click ‘Users’ and you will be able to see which users/student posted Facebook updates during your selected time period and drilldown (right click) into each user and select ‘Individual Record’ to get the exact time of the Facebook updates.

   

  

   

  

  Click to enlarge

• Alternatively you can start by looking at the dates in question, drilldown (right click) into each user for a specific date and drilldown again into ‘Individual Records’ for the exact time of the specific user’s update

The Claim

According to network security firm, Arbor Networks, traffic to Google sites broke a new record this month, and now accounts for 6.4% of all Internet traffic around the world.

The 6.4% includes all sites owned by Google, including Google’s search engine, YouTube, GMail, Google Maps, AdWords and Google’s office suite of products like Google Docs and Spreadsheets. The data was obtained through more than 110 ISPs in 17 countries.

Our Test

Using Vantage I ran an analysis on web proxy traffic for July – September 2010. I am aware the Google data is for September 2010 but wanted to test a larger set of data since I’m reporting on traffic from a smaller amount of users, compared to 110 ISPs in 17 countries.

How much of your traffic is going to Google sites?

To find out simply create an Alias and add all Google related sites.

• Click on the ‘Alias’ tab in Vantage.
• Double-click the ‘Web sites’ alias on the left-hand side and make sure ‘Use wildcard matching’ is selected. Click ‘OK’.
• Click ‘Add Group’ in the Groups task pad.
• Name the group something intuitive…I named mine ‘GOOGLE SITES’.
• Click ‘Add’ and type ‘*google*, youtube.com, ytimg.com, gmail.com and adwords.com’. The *google* will add all domains with the string google in it. ytimg.com is YouTube’s DNS and also needs to be added (see previous blog post for more information).
  
• Click ‘OK’
• Click on ‘Summaries’ tab
• Click ‘Site Domain’ to get a listing of sites
• Enable the ‘Web site’ Alias on the left hand side.
  
• Click the top of the column called ‘Size’ to sort domains in descending order
• Click the pie chart icon on your right-hand side
  
• Have a look at the pie chart to identify what percentage of your traffic is going to Google related sites.
  

  % of WebSpy traffic going to Google related sites

It is apparent that WebSpy traffic to Google sites makes up more than twice than the reported average. I’m also fortunate enough to have access to one of our larger client’s (approximately 25,000 users) most visited websites. The below pie chart is from one of their reports on site domain traffic for a random period in October.

% of traffic to Google sites from Enterprise size organization

How much of your traffic goes to Google websites?

We really encourage you to add a Google sites alias and have a look at your Google traffic. Please comment below on the percentage of your traffic going to Google sites, approximately how many users you are reporting on and the time period (about a month or more) you are reporting on.

Many thanks.

As the term ‘hit’ can sometimes be confusing, let’s start off by properly define hits.

What is a hit?

WebSpy classifies a hit as an individual file or item that has passed through your logging device and been recorded in a log file.

The actual content and size of a hit can vary widely. A hit may consist of a small picture or text file, a large zip file or executable, or any other individual file. A hit can be a file downloaded from an Internet site, an email that’s been sent or received, a file downloaded from an FTP site and so on.

When dealing with web log files, one web page can be made up of many hits – the main page, the pictures on the page, the files on the page and so on. In some situations, a user cannot control the content of the hits they access, such as in the case of advertising pop-up messages.

How many hits on one page?

Here’s just an example of number of hits when browsing to some of the more popular sites out there:

• facebook.com – 75 hits
• apple.com/iphone – 69 hits
• youtube.com – 29 hits
• google.com – 4 hits
• yahoo.com – 40 hits
• twitter.com – 67 hits

These hit counts are just from one user (me) visiting a few sites. Imagine the amount of hits/data you’ll get for an organization with 100, 1,000 or 10,000 users!

Also interesting is the fact that out of those hits, only a small proportion actually displays in your log files as a hit to facebook.com or youtube.com. I logged into my Facebook account and at the same time ran a Firefox add-on called Firebug. Firebug allows you to see all the resources on a specific page and how long they took to download. As you can see below only one of the hits are attributed to facebook.com. The rest are from fbcdn.net. The same test on a YouTube page also yielded one out of 29 hits attributed to youtube.com. The rest was from ytimg.com, doubleclick.net, googlesyndication.com and gstatic.com.

Click to Enlarge

Using Aliases to merge your hits

fbcdn.net and ytimg.com are Facebook’s and YouTube’s CDNs (Content Delivery Network). Basically a network of servers hosting images and resources, spread out across the globe, to enable local delivery and thus improve performance. Unfortunately it doesn’t improve the accuracy of your reports, but there’s an easy way to fix this using Vantage. Simply merge the hits from the main domain (facebook.com, youtube.com, or any other site) and their CDNs by creating an Alias in Vantage’s Summaries screen.

To do this simply:

• Click on the ‘Summaries” tab in Vantage
• Click ‘Site Domain’ to get a listing of sites
• Ctrl + click the sites you want to add to the Alias (in this case facebook.com and fbcdn.net), right click and click ‘Add to Alias”
  
  

  Click to Enlarge

   

• Pick the Alias called ‘Web sites’ and name it something intuitive…why not facebook.com

   

• Now enable the ‘Web site’ Alias on the left hand side

 

• You’re good to go, from now on all your Facebook traffic will be reported on more accurately by merging actual facebook.com hit with hits on Facebook’s CDN (fbcdn.net).

If you want to find out more about using Aliases you can read about it in our User Guide (starting at page 46).

The default LDAP query when you first run through the Import Organization wizard should filter these computers objects out. The query is:
(&(objectCategory=person)(objectClass=user))

In Active Directory, computers do not generally have an objectCategory equal to Person. Computers usually have the objectCategory ‘Computer’.

If by chance your computers are not being excluded by this filter, you could exclude all objects without an email address. This of course assumes that all users you want to import have an email address populated in Active Directory. To exclude objects without email addresses, the filter becomes:

(&(objectCategory=person)(mail=*)(objectClass=user))

Another useful addition to the query is to exclude users that have been disabled in Active Directory. You usually disable an account when a person leaves the organization, but you still need their user profile in Active Directory for whatever reason. This query is slightly less obvious:

(&(objectCategory=person)(mail=*)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))

For information on what the numbers mean in the query above, see How to query Active Directory using a bitwise Filter

Another question I’m often asked is how to exclude specific OUs from a query. Unfortunately LDAP does not support this concept and the only way to do this is to run multiple queries on different root level DNs. This means running through the Import Organization wizard multiple times with a different Root Distinguished Name each time, and the ‘Merge’ options set to ‘Keep users that are no longer in the directory’ and ‘Keep existing user details’.

If you have other helpful LDAP queries, please leave a comment below.

Event Log Management

Every event that occurs across a network can be recorded in an event log file. The list of events that are recorded by default can be modified to reflect the needs of the organizations system. Information stored in event log files is extremely useful to organizations as it provides real-time indications of network incidents as well as an audit trail of user activity. However extracting useful information can be challenging as it is very difficult to manage and filter the vast amount of data generated.

An organizations’ event log management is only as effective as the amount of data they are including from their networks activity. To be able to provide an accurate report on any particular part of the system, data needs to be generated for that part. For example, you cannot compile a report on who accessed a confidential file if you do not set up the file to raise an event (and have the event logged) when the file is accessed.

As the required level of monitoring depends on the organization and there are many event categories in security auditing, the first step is determining which event categories need to be audited. The following are a list of available categories:

• Account Logon Events
  Track users logon and logoff events.
• Account Management
  Tracks attempts to create users or groups, rename users or groups, enable user accounts, disable user accounts or change account passwords.
• Directory Service Access
  Used with auditing tasks on domain controllers.
• Logon Events
  Records creation and destruction of logon sessions (including remote sessions)
• Object Access
  Used to record user access of objects such as files.
• Policy Change
  Records changes to user rights assignment policies such as Windows Firewall Policy.
• Privilege Use
  Records when users exercise a user privilege.
• Process Tracking
  Tracks process information such as program activation/exit.
• System Events
  Records system events such as shutting down a computer.

Each of these categories contains many subcategories and events which can be used to create a complete audit trail of system activity. It is recommended that only essential events are setup for auditing as generating a large number of events can severely affect system performance.

To enable audit log and specify the files/folders to audit in your operating system please refer to http://support.microsoft.com/

Vantage and Event Logs

After file auditing settings have been implemented on the system, it is a simple process to start managing event logs and extracting information. Although the MS provided interface for event logging and tracing has improved dramatically from the original, Vantage simply does a much better job at it. Hey, don’t take my word for it. Try out both and see for yourself.

WebSpy Vantage’s ability to translate event log data into manageable information will, among other things, enable organizations to:

• Monitor failed authentication attempts
  Identify users trying to access files and folders they are not authorized to access, or the system failing to provide legitimate user access.
• Prevent data loss and leakage
  Identify the access, modification or printing of confidential files to prevent information leakage or identify the person behind accidental or deliberate data loss.
• Ensure employees adhere to specified work schedules
  Monitor event logs that record when an employee’s computer has been powered on or shut down.

Importing Event Logs into Vantage

The first step is to import Windows Event Logs into a storage in Vantage. This process can be added to run automatically at appropriate intervals using Tasks.

After creating a storage for Windows Event Logs, reports can be generated and analysis run. This will allow useful information to be extracted from Event Log data.

Vantage uses aliases for the creation of more meaningful information, for example, event ID’s are translated to an event category to enhance readability of generated reports and analysis. A list of event ID’s and their categories has been included at the bottom of this post for reference purposes.

Importing event logs into a storage:

1. Open Vantage and click the Storages tab
2. In the left pane, click Import Logs This will start the import dialog wizard
3. Enter a name for the storage in the Create a new storage dialog box, then click Next
4. Select the Windows Event Log radio button, then click Next
5. Select the Microsoft format (description: Windows Event Log), then click Next
6. Click Add, enter the name the computer in the Server dialog box, click OK and then click Next
7. Continue through the wizard and select any filter, field or partitioning options to include, then click OK The event log data will now be imported into the storage

Generating a Report

1. Click the Reports tab
2. Select the type of Report to generate Note: Vantage includes many default templates for Windows Event Logs such as Failed Events, Application Errors and Failure Audit Trends.
3. In the left pane, click Generate Report This will launch the Generate Report wizard
4. Select the storage to report on Note: This should be the storage created previously for Windows Event Logs
5. Select the document format(s) for the report
6. Enter the report name in the Document Name dialog box
7. Continue through the wizard and select any splitting, filtering or email options, then click OK The report will now be generated

Running an Analysis

1. Click the Summaries tab
2. In the left pane, click New Analysis This will launch the Create Analysis wizard
3. Enter a name for the analysis in the Name dialog box, select the storage, and check that the schema is set to All Windows Event Schemas, then click Next
4. Select the type of Analysis to run, then click Next
5. Continue through the wizard and select any filtering or summaries options, then click OK The summary will now be generated

The summary allows interactive drilldowns to any level for data mining and information exploration.

Also see previous blog ‘File Access Reporting – How to report on who accessed a file or a folder‘.

If you have any questions about reporting on event logs don’t hesitate to get in touch with our support team.

Event ID’s and Categories

Richard has been working with Forefront Threat Management Gateway (TMG) 2010 and its predecessors for more than 12 years. He has designed and deployed network security solutions using TMG and ISA for SMB’s, military and defense organizations, and Fortune 500 companies around the world.

In addition to his isaserver.org blogs, Richard has his own ISA/TMG blog where he recently posted some useful tips on changing WebSpy Vantage’s scheduled task recurrence interval using the schtasks.exe command line tool. Adding more frequent import options (i.e. hourly) is on the product roadmap but until then, using the command line tool is a great alternative.

We do recommending visiting tmgblog.richardhicks.com – brimming with ISA Server and TMG information and tips, here’s just some of the latest blogs:

• Changing WebSpy Vantage Scheduled Task Recurrence Interval
• Websense Integration Support for Forefront Threat Management Gateway (TMG) 2010
• Load Balancing and Forefront TMG Firewall Clients
• How to Slipstream Service Pack 1 for TMG

So let’s use WebSpy Vantage to drill into that Anonymous user and find out what is going on.

One way to do this is to run an Ad-hoc analysis on the Summaries screen and drilldown into the Anonymous user to view all the information about that user. However, TMG and ISA tend to log a lot of information that may not be relevant to this particular investigation, so I’ve created some report templates (one for ISA and one for TMG) and a set of Aliases that pull out some relevant information.

Download our Anonymous Traffic Investigation Report

If you’re running WebSpy Vantage download the Anonymous Traffic Report Templates & Aliases

Then open the .Templates file on the Reports tab, and the .Aliases file on the Aliases tab. Once you have both files opened, go to the Reports tab and click either the ‘Anonymous Traffic Investigation (ISA)’ or the ‘Anonymous Traffic Investigation (TMG)’ report. Then click the ‘Generate report’ link and run the report template on your ISA or TMG storage.

The report gives you the ability to drill into the Allowed, Denied and Failed traffic to see a list of the unauthenticated IPs, Sites, Rules responsible for blocking or allowing the traffic, unauthenticated Applications and Result Codes.

Main causes of anonymous traffic

What you will probably find is that most of the Anonymous traffic is being denied by your TMG or ISA firewall. When a client first requests a web page, the proxy will challenge the client for authentication. These events are often logged with the result code 12209 meaning ‘authorization is required to fulfill the request’. These requests are therefore denied by the proxy until the client’s credentials are authenticated.

Have a look at the amount of traffic being denied and then checkout the Result Codes associated with the denied traffic. Chances are you’ll see ‘proxy authentication required’ appear predominantly.

If you also look at the Applications section you may also find that Windows Updates are sailing through your TMG or ISA firewall unauthenticated.

Filter out unauthenticated traffic from Reports

The most logical next step is to filter out the information you do not want in your reports. You’ll probably still want to include Windows Update traffic in your reports, but you’re probably not so interested in the ‘proxy authentication required’ information. So let’s filter that out.

To do this:

1. Go to the Reports tab and select the report you want to filter (such as your Organization report)
2. Click ‘Edit Template’, then click ‘Template Properties’.
3. In the filter section at the bottom of the dialog, click Add | Field value filter.
4. Select the ‘Result Code’ summary and select the Status Code Names (ISA-FTMG) alias.
5. On the toolbar, search for Authorization, and check the following two items:
   • The server requires authorization to fulfill the request. Access to the Web Proxy filter is denied.
   • The server requires authorization to fulfill the request. Access to the Web server is denied. Contact the server administrator.
6. Ensure the ‘Exclude’ radio button is selected and click OK.

If you decide that you don’t care about seeing ANY unauthenticated traffic in your reports, you can always simply filter out the Anonymous user from your reports.

To do this:

1. Go to the Reports tab and select the report you want to filter (such as your Organization report)
2. Click ‘Edit Template’, then click ‘Template Properties’.
3. In the filter section at the bottom of the dialog, click Add | Field value filter.
4. Select the ‘Username’ summary.
5. On the toolbar, click Add and type ‘anonymous’. Click OK.
6. Ensure the Exclude radio button is selected and click OK.

Hopefully this article improves your understanding of the ‘anonymous’ user, and gives you some actions to take for your specific reporting situation.

If you have any questions, please leave a comment below.

Majority of security vendors will give you a high level overview of the categories, such as Sports, Shopping, Online Community, Streaming Media, Employment and Gambling, but rarely provides intuitive ways to further investigate the traffic going to the sites within these categories. The nifty thing about WebSpy’s solutions is that, as long as categories are logged, you can use WebSpy to analyze web browsing in relation to these categories and get a much clearer overview of your organization’s web usage.

Classify Productive & Unproductive Categories

Assessing productivity in relation to predefined categories is what I would like to focus on today. I have imported and run an analysis on TMG logs using WebSpy Vantage. As previously mentioned, you can import logs from any security device we support – if the information is in the log file WebSpy can report on it.

TMG logs contain information whether traffic has been ‘Allowed’, ‘Denied’ or ‘Failed’. Using WebSpy Vantage you can easily drill down further into this information. For example, let’s say I’m interested in having a look what categories have been allowed, i.e. not blocked, I simple expand the ‘Allowed’ node and click ‘URL category’.

Allowed Categories - Click to Enlarge

This information is great but it doesn’t tell us anything about productivity. WebSpy Vantage not only provides this assessment for your entire organization, specific department and individual users, but also gives you the ability to customize the categories that are deemed productive as this can vary wildly depending on the industry and organization.

How?

You use WebSpy’s Aliases feature to sort categories in relation to your organization’s view of their productiveness. Our software comes with a default list of aliases so you can either edit these or set up new aliases. I’ll take you through the process of setting up an Alias from scratch.

1. Creating a New Alias

• Click on the Alias tab and select ‘New Alias’ in the top left corner
• Name your Alias something appropriate and provide a short description. I’ll name mine ‘Productivity’.
• Make sure ‘Apply alias to selected summaries’ option is checked
• Click ‘Schema’ to specify the log file type and scroll down to the bottom of the list to locate and select ‘URL Category’.
• Tick the ‘Group unresolved into a single name’ box and name it something appropriate. Let’s go with ‘Uncertain’.

 

2. Add Alias Groups

Once an alias has been added, you need to add alias groups. You can have as many alias groups as you want but for this purpose it makes sense to have only two, ‘Productive’ and ‘Unproductive’. There might be certain categories, such as ‘Education/Reference’ or ‘Blogs/Wiki’, that might be difficult to correctly deem as productive or unproductive and you’d rather not specify. If this is the case you don’t need to add an alias group as it will automatically be created for any category that hasn’t been grouped under the other alias groups. Remember how we ticket ‘Group unresolved into a single name’ and called it ‘Uncertain’ before.

• Click the Add Group button in the Groups task pad.
• Enter the desired alias group name (Productive) in the ‘Key’ edit box and click OK. Repeat steps for the ‘Unproductive’ group.
• At this stage you could also add items (categories) to your group but I’m going to show you another way of adding categories.

   

  

3. Add Categories to your ‘Productive’ and ‘Unproductive’ Alias Groups

This is where customization really works its charm. What is deemed as unproductive at one company might be completely legit and considered productive at another. For example, in a recruitment company one could assume it would perfectly normal for employees to visit other employment sites but this could be considered personal and unproductive at a hospital or real estate agent.

There’s a few different ways of adding items to an Alias group. While still in the Alias screen you can click ‘Refresh Unassigned’ in the top right part of your screen. Because you haven’t assigned anything yet all categories will be displayed. From here you can simply highlight the category group, for example ‘Unproductive’ and Ctrl + click all categories you want to place in that group. Once you’ve selected your categories right click and select ‘Add to selected group’. Repeat the process to add categories to your ‘Productive’ group.

Alternatively, you can go back to the ‘URL Category’ listings in the ‘Summaries’ tab and Ctrl + click selected categories, right click and select ‘Add to alias’, select your ‘Productivity’ alias from the drop down menu and select the ‘Productive’ or ‘Unproductive’ group.

4. Assess Productivity

With aliases, groups and items set up you’re ready to assess productive and unproductive browsing. In the ‘Summaries’ screen, left hand side under ‘Aliases’, simple select your ‘Productivity’ alias and the URL categories will be sorted in accordance with your view of their productiveness.

Productive vs Unproductive Browsing - Click to Enlarge

You can also investigate further by, for example, drilling down to determine what unproductive categories are most popular, what are the most popular unproductive websites within those categories, what hours during the day majority of unproductive sites are accessed (you might have a policy that allows personal web browsing during lunch hours), and of course who spends the most time on unproductive websites within your organization.

Top Unproductive Websites - Click to Enlarge

• How to import your log files and explore the information recorded by IronPort using the Summaries screen
• How to open the customized IronPort Report Templates and Aliases
• How to generate reports
• How to import your organizational structure and report on departments
• How to setup the Web Module and publish reports

PART 1: Importing log files & exploring your IronPort summaries

Once you have exported your IronPort access logs (see http://www.webspy.com.au/vendors/ironport/howto.aspx#ftp), this video takes you through importing your logs into WebSpy Vantage and analyzing data on the Summaries screen.

PART 2: Opening the customized IronPort Templates & Aliases, and running reports

This video takes you through opening the IronPort-specific report templates and aliases and generating a report that provides an overview of your organization’s Internet usage.

PART 3: Importing your Organization structure & generating department reports

This video shows you how to import your organizational structure into WebSpy Vantage from a directory server (such as Active Directory) using LDAP, and then generating a report that contains information on your newly imported departments.

PART 4: Using the Web Module.

This video takes you through configuring and using the WebSpy Vantage Web Module. Specifically, it takes you through the following tasks:

• Configuring the Web Module for Windows Authentication
• Adding a Web Module to Vantage
• Publishing reports to the Web Module
• Adding permissions for a user
• Synchronizing the Web Module
• Using the Dynamic Reports tab

PART 5: A quick word about tasks & conclusion

This video summarizes the actions taken in the previous four videos and also briefly discusses how to automate the reporting processing using scheduled tasks.

I hope this helps! Let me know if you have any questions by leaving a comment below.

The log databases are stored in an SQL Express instance named MSFW. By default these databases cannot be accessed by a remote computer. I’d first like to say that we recommend changing TMG’s logging to W3C text files, as these logs are about 5-6 times faster to import, and you don’t need to worry about the steps below.

But if you need to stick with the SQL Express logging, here are the basic steps to enable access to the logs from a remote computer:

Enable TCP access to the MSFW instance

To do this:

1. Log into your Forefront TMG server using administrator credentials.
2. Select Start | All Programs | Microsoft SQL Server 2008 | Configuration Tools | SQL Server Configuration Manager.
3. Expand SQL Server Network Configuration and select Protocols for MSFW
4. Right-click TCP/IP and select Enable
5. Click OK on the Warning dialog informing you that “changes will not take effect until the service is stopped and restarted.”

Enabling TCP/IP on the MSFW instance

Set the listening Port on the MSFW instance

Once TCP/IP is enabled on the MSFW instance, you need to set it to listen on port 1433

1. Select Protocols for MSFW under SQL Server Network Configuration
2. Right-click TCP/IP and select Properties.
3. Click the IP Addresses tab and scroll to the IPAll section at the bottom of the list.
4. Change the TCP Port to 1433 and ensure nothing is entered in TCP Dynamic Ports (Delete the ‘0′ value  if present). Click OK and click OK on the Warning dialog.

Setting the Port on the MSFW instance

Change the listening port on the ISARS instance

The ISARS SQL instance also listens on port 1433 and this can cause connection issues. Change this instance to use port 1434:

1. Still in SQL Server Configuration Manager, select Protocols for ISARS under SQL Server Network Configuration
2. Right-click TCP/IP and select Properties.
3. Click the IP Addresses tab and scroll to the IPAll section at the bottom of the list.
4. Change the TCP Port to 1434 and ensure nothing is entered in TCP Dynamic Ports. Click OK and click OK on the Warning dialog.

Changing the port on the ISARS instance

Restart the Services

For the above changes to take effect, you need to restart the SQL Server (ISARS) and then the SQL Server (MSFW) services in that order.

1. Go to Start | Administrative Tools | Services
2. Right-click the SQL Server (ISARS) service and select Restart.
3. Right-click the SQL Server (MSFW) service and select Restart.

Test the connection from the WebSpy machine

You should now be able to connect to the MSFW databases from a remote computer. To test the connection, we recommend that you install SQL Management Studio on the machine running WebSpy and try to connect to <TMGservername>\MSFW, 1433 (replace <TMGservername> with your actual server name or IP address). For example TMGServer\MSFW, 1433 or 192.168.0.10\MSFW, 1433.

As long as you are logged into Windows with a user account that is a local administrator on the TMG server, you should be able to connect without issue.

Importing the TMG Log files into WebSpy Vantage

Once you have established a connection, you can import your logs using WebSpy Vantage like so:

Create a new Storage

Select Database Connection

Select the Microsoft FTMG Loader

Click Add

Enter TMGServer\MSFW and port 1433

Successfully Imported WebProxy Logs

The screenshots above also illustrate using a database mask of *WEB* to only import the WebProxy logs. If you only want to import the Firewall logs, set the database mask to *FWS*. If you want to import both the WebProxy and Firewall logs, leave the database and table masks set to *.

Now that you have your log files imported, you can run a quick ad-hoc analysis on the Summaries screen or generate any of Vantage’s default web of firewall reports. M

Make sure you also download our Forefront TMG specific Aliases and report template. For more information, see our Forefront TMG How To page.

If you have any questions or encounter any hurdles, please leave a comment below.

“I would like to know how much of my bandwidth is being eaten by each protocol. I will then use this information to determine if circuit may need to be increased due to increased traffic”.

This customer was collecting syslog messages from a Cisco Firewall, then using WebSpy Vantage to generate reports. In theory, this sounds like a fair plan. Unfortunately, the Cisco Firewall logs many different types of messages. Some to do with denied packets, some to do with authentication, some for vpn and so on. The information contained within each message changes. Some events include the size information that is required for any type of bandwidth assessment and some don’t. Correlating the required events to get any sort of accurate ‘bandwidth’ representation is a bit of a nightmare.

Fortunately, there’s a simpler method. If you search the Cisco website or the Internet for bandwidth utilization reporting, you’ll no doubt be pointed in the direction of NetFlow.

NetFlow is a network protocol developed by Cisco Systems to run on Cisco IOS-enabled equipment for collecting IP traffic information [Source Wikipedia http://en.wikipedia.org/wiki/Netflow]

There are a couple of commands to enter on your router to turn NetFlow on, and then you just need a NetFlow collector to receive the Netflow information and generate reports.

Fortunately WebSpy has developed a little tool called FlowMonitor that collects the Netflow information and writes a log file that can then be imported into WebSpy Vantage and reported on.

The FlowMonitor Management Console

Once your FlowMonitor logs are imported into WebSpy Vantage, you can run the default FlowMonitor report to see the size of traffic flowing between IP addresses, subnets, router interfaces or protocols. Alternatively you can create your own custom reports to see exactly what you want to see.

NetFlow doesn’t record usernames or URLs so it’s not great for reporting on the web sites your users are visiting, but it is great for network administration and trouble shooting. Identify chatty IP addresses, protocols that are chewing too much bandwidth, the times throughout the day when incoming or outgoing links become heavily utilized and so on.

For information on how to configure your router and deploy FlowMonitor, see the FlowMonitor Installation and User Guide. You can also download a free trial here.

FlowMonitor is a handy little tool. Ask your friendly WebSpy account manager about it today!

One of Forefront TMG’s major strengths is obviously its URL categorization and filtering abilities. Since TMG now takes care of the threat management aspects, clients converting from other solutions, such as ISA Server, no longer need a third party filtering solution and will most likely save a considerable amount of money.

However, the reporting functionality included in Forefront TMG are not much different from ISA Server 2006, i.e. very little flexibility or customization for those with reporting requirements beyond general overviews cluttered with irrelevant information.

We’ve blogged a lot about TMG reporting in the past and have now uploaded new and dedicated WebSpy Vantage and Microsoft Forefront TMG pages outlining:

• 10 Reasons to Use WebSpy Vantage to Report on Forefront TMG
• How to:
• Set-up TMG Logging for WebSpy
• Import TMG Logs into WebSpy Vantage
• Forefront TMG Report Templates and Aliases (created to make your life a lot easier)
• Run Reports
• Analyze and Drilldown into Data

Have a look at WebSpy Vantage and Microsoft Forefront TMG.

Hopefully it can assist you in your quest for sophisticated Forefront TMG reporting.

A large chunk of our clients use WebSpy Vantage to report on a wide spectrum of devices (web proxy server, email server, firewalls, routers, switches etc) from more than 150 different vendors and not solely as a web analytics tool. Saying that, many WebSpy Vantage users do take advantage of its web server reporting abilities and have experienced first hand the benefits of log file analysis over GA’s JavaScript tagging. Clients generally like the way they can work with the data in GA, but not completely happy with the data GA provides and therefore use WebSpy Vantage to complement and provide detailed drilldowns into certain areas.

Log File Analysis vs. JavaScript tagging

In order to enable Google Analytics to report on incoming traffic you need to copy and paste a custom JavaScript to every page within your site or site template. Since Google Analytics is relying on JavaScript the following issues are very likely to affect the accuracy of your data:

• Not reporting on visitors who disable JavaScripts in their browsers
• Overestimating visitors who regularly clear their cookies
• Limited or no reporting for non-standard page extensions

Limited or No Reporting for Non-Standard Page Extensions

This is the area I want to expand on today. GA won’t give you any information on resources accessed within the site that is not an actual page containing the custom JavaScript. You won’t be able to see how many times visitors downloaded your documents, such as white paper, product catalog or PowerPoint presentation, in file types such as .doc, .pdf, .txt, .pps, .zip, .xls, etc.

Alright, I’m telling lies, there are roundabout ways of tracking these downloads but it includes tagging all document links with a _trackPageview() JavaScript. This piece of JavaScript assigns a pageview to any click on a tagged link. Not only does this involve a pretty cumbersome process of adding scripts to all your website document links, you also need to take into account the error in reported document downloads that will occur since you are not always in control, and can tag, all instances of your document links appearing on the inter-web. For example, if other websites are linking straight to your documents, or if people are arriving at those resources from other external sources (e.g., an email containing an url to the pdf).

Reporting on your web server log files is a much more reliable way of getting the accurate information you need. It’s pretty darn easy as well.

Using Vantage to Analyze Non-Standard Page Extensions

Below is just a brief example of how easy it is to use WebSpy Vantage to get the reliable information on different file type downloads.

After importing my web server logs, for a randomly selected period in April, into Vantage I can get an overview of all the different file types accessed by simply clicking ‘Site Extension’.

Let’s say I am curious to find out more about the .pdf documents accessed by visitors during this period. I click on the .pdf site extension and am immediately presented with a variety of options to further investigate .pdf downloads.

For example, I can drilldown into .pdf ‘Site Resource’ to get a list of all the pdf documents accessed. To locate the most popular pdf I simply sort the list by number of hits.

It seems like the most popular .pdf document on this particular day one was our ‘5 Reasons to recommend WebSpy Reporting with 44 hits. Again, I can now drilldown further into any site extension to find out relevant information on referring sites, search engines used, the keywords used search engines etc.

Referrer Keywords

Unfortunately there are two issues with the Web Module auto update process. Everyone on a 64 bit operating system will encounter issue #1 (Unable to locate installation location), and some of you may encounter issue #2 (System.IO.FileLoadException).

This article describes the errors and how to work around them to successfully update the Web Module.

Issue #1

The usual process to update your Web Module is to log into your Web Module server, right-click the WebSpy system tray icon and select ‘Check for updates’.

If you do this on a 64 bit operating system, you will receive the following error:

Web Module Updater error: Unable to locate installation location

You can fix this issue by clicking Yes, and specifying the Web Module’s installation location, which is usually somewhere under c:\inetpub\wwwroot (or just c:\inetpub\wwwroot if you didn’t specify a virtual directory when installing).

This will allow the updater to continue and you will be prompted to download and install the latest update.

Issue #2

Unfortunately, you may encounter another error during the update installation. The text of the error will be something along the lines of:

System.IO.FileLoadException: Could not load file or assembly ‘ICSharpCode.SharpZipLib, Version=0.84.0.0, Culture=neutral, PublicKeyToken=1b03e6acf1164f73′ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0×80131040)

Work Around

We are currently working on solutions to both of these issues. In the mean time, here is a work around to install the update. On the Web Module server:

1. Download this file:
   http://update.webspy.com/autoupdate/files/vantagewebmodule/vantagewebmodule2.2.0.18.zip
2. Stop IIS (See instructions below).
3. Right-click the WebSpy system tray icon and click Exit.
4. Backup your existing Web Module installation by copying everything in your Web Module’s installation folder (usually under c:\inetpub\wwwroot (or just c:\inetpub\wwwroot if you didn’t specify a virtual directory when installing) into a completely separate location (i.e. don’t keep it in a sub-folder).
5. Extract the downloaded zip file to your web module’s installation folder. Overwrite all the existing files.
6. Start IIS (see instructions below).

Your Web Module will now be updated to the latest version.

Stopping and Starting IIS

In IIS Manager (Start | Control Panel | Administrative Tools | Internet Information Services (IIS)) , right-click the site you want to start or stop, and click Start or Stop

We sincerely apologize for the inconvenience and will hopefully have a solution out soon. If you have any problems with the process above, please contact support at webspy dot com.

For those that don’t know what Soho is all about, check out this video:

Soho is a dashboard application that displays download and upload traffic statistics for each computer in your network. If you haven’t yet tried Soho, please give it a go and download it here.

It is our intention to keep our Alpha testers up to date with our ongoing development. Right now, I’d like to inform you about some issues experienced by a handful of testers and how to go about resolving them.

Learn to restart the Soho Agent

First of all, one of the handiest things we can tell you right now is how to restart the Soho Agent. This single step is resolves 99% of all Soho issues, at least temporarily. If these steps seem too complicated, rebooting your PC also has the same effect.

To restart the Soho Agent on Windows:

1. Launch the Services Console by going to Control Panel | Administrative Tools | Services. Or if you like handy short cuts, try Start | Search (or Run), Type ‘services.msc’ (without the quotes) and press enter.
2. Right-click the “WebSpy Soho Agent” service and select Restart. If you get a ‘time out’ error message or warning, ignore it and right-click the service again and select Start.

To restart the Soho Agent on Mac OS:

1. Open the terminal from /Applications/Utilities/Terminal
2. Type sudo launchctl stop “WebSpy Soho Agent”
3. Enter your user password if requested.
4. Wait about 5 seconds.
5. Type sudo launchctl start “WebSpy Soho Agent”
6. Again, enter your user password if requested

OK, now you have the skills, here are the issues and work-arounds:

100% CPU usage after sleep or hibernate

Some users reported Soho’s impressive ability to consume 100% of their CPU when their computer wakes from sleep or hibernation. A few users experienced a slow and sluggish PC, and uninstalled Soho immediately.

If you’re looking for the Soho process in Windows Task Manager you will not see it until you click the Show processes from all users button (or checkbox in XP). This is because the Soho Agent runs under the System user account in order for it to run, no matter who is logged onto the PC.

From here you can end the WebSpy.Soho.Agent.exe process and everything should return to normal. You can then restart the “WebSpy Soho Agent” to get Soho working again (see steps above).

We believe we have fixed this and are in the middle of some final testing. All going well, we should have a new build ready for you very soon. In the mean time, you may like to disable sleep and hibernation in your PC’s power options.

Soho doesn’t install on Mac OS 10.5??

Our first Alpha release did not install on Mac OS 10.5 (Leopard). This was due to a silly checkbox in our packaging system not being checked. We’ve checked the checkbox and uploaded a new build to our web site. You can download it from here:
http://www.webspy.com.au/products/soho/download.aspx

Note: Soho will only install on Mac OS 10.5 (Leopard) and 10.6 (Snow Leopard). Versions 10.4 (Tiger) and below are not supported.

All computers disappear from the Current Activity chart except the local computer

Sometimes all computers will disappear from the Current Activity chart leaving your local computer all by itself. This happens when the communication between Soho Agents becomes jammed. You can usually resolve the issue by restarting the Soho Agent on your local computer (see steps below). If this doesn’t work, restart the Agents on all other computers running Soho. We are currently working on a fix for this issue.

The Soho User Interface is completely blank

If there is no information in the Total, Current Activity or History chart, this is because the Soho Agent has stopped running. Reasons for this may vary, so please let us know if this is regularly occurring. Restarting your agent usually resolves the issue (see steps above).

Feedback

Thank you to everyone that has submitted feedback so far.

Just a reminder to please let us know if your network card works or doesn’t work with Soho on this page:
http://www.webspy.com.au/products/soho/supportednics.aspx

You may also like to review the current list of features and bugs and vote them up or down at:
http://webspysoho.uservoice.com/

There is also a dedicated Soho Alpha Feedback thread in our forums at:
http://www.webspy.com.au/forums/viewtopic.php?f=8&t=12

We will let you know when fixes are available to the above issues.

Here’s a quick video outlining some of the differences between TMGs Reporting, and what can be achieved using WebSpy Vantage. The video does not illustrate all the limitations outlined below, so please read on.

Whats is in the Forefront TMG report?

The default TMG report contains the following sections

• Summary
• Web Usage
• Application Usage
• Traffic and Utilization
• Security
• Malware Protection
• URL Filtering
• Network Inspection System

Each section contains overviews such as ‘Top users’ and ‘Top Sites’.

If your reporting requirements can be satisfied with these overviews – that’s great! Unfortunately, when you start thinking about what system administrators and other people in your organization actually need to make informed decisions, this report is quite limiting.

The 8 Limitations of Microsoft Forefront TMG’s Reporting

Here is what I consider to be the 8 main limitations of Microsoft Forefront TMG’s reporting functionality.

1. No Drilldowns

Want to see the sites that the top 5 users accessed? Want to see the users that downloaded the most traffic from youtube? These are fairly standard reporting requirements that simply cannot be achieved using the inbuilt TMG reporting.

WebSpy Vantage lets you either interactively drilldown into a user or site, or produce a regular report that includes further details about what your top users have actually been up to.

2. No Filtering

When you generate a report in TMG, you can only filter the report by a date range. There is no way to filter out anonymous (unauthenticated) traffic or exclude traffic coming from advertising servers (such as doubleclick and 2mdn.net) that tend to dominate most of the top 10 sites.

This can easily be achieved using WebSpy’s software. Check out my video on how to remove clutter from your web reports.

3. No Customization

Customization of each overview in the TMG report is limited to the number of items to show (e.g. top 5 or top 50 users), and the sort order (Incoming Bytes, Outgoing Bytes, Requests and Total Bytes).

What about the time a user spent browsing the web, or the number of users that visited a specific site? There is no way to add custom columns such as total browsing time, average session time, or number of users/sites/IPs to the report tables.

Or say you simply want to change your top users chart from a bar to pie to easily see the percentage used. Nope sorry!

If you do make one of the two available customizations in a TMG report, you then get the annoying Apply / Discard message to save changes to the configuration database.

All of these customizations can be achieved using WebSpy Vantage, and it doesn’t touch your TMG server to apply a change to a report.

4. Limited Report Distribution

When you generate a report, you get the option to email it to a specific email address. What if you would like to create a report for every department, and then email it to the managers of each department? Or better yet, host the report on a secure web server where department managers can log in and view their reports?

WebSpy Vantage Ultimate comes with a secure ‘Web Module’ specifically for this purpose and managers still receive a link to the report via email.

5. Cluttered ‘Top Sites’ List

The ‘Top sites’ list can become particularly cluttered due to the inclusion of sub-domains. I don’t want to mentally add up the size values from farm1.static.flickr.com, farm2.static.flickr.com, and farm3.static.flicr.com – I just want to know how much was downloaded from flickr.com.

This is compounded by the inability to exclude sites that are merely placing advertising banners on the actual sites users are visiting (as mentioned in the ‘No Filtering’ limitation above).

WebSpy Vantage breaks URLs down into separate components and lets you analyze each part separately. Look at the Site Domains summary to remove sub-domains and see only flickr.com. Or perhaps you want to see the keywords a user entered into search engines like Google? Or perhaps the top pages accessed within a website? No problem. Just include the Site Keywords or Site Resource summaries in your Vantage reports.

6. No Grouping or Aliasing

There is no way to group users into departments or locations, or IP addresses into subnets, or extensions such as .html, .pdf or .exe into file types. The ability to group and represent raw log data in more meaningful ways, as offered by WebSpy Vantage, can increase the value of a report tremendously.

7. No Productivity Assessment

One of the major features introduced in TMG since ISA Server 2006 is the included URL categorization technology.

Although the TMG report gives you an overview of the categories that have been visited, the report does not use this information to display a productivity assessment for your users.

WebSpy Vantage not only provides this assessment, but also the ability to customize the categories that are deemed productive as this can vary wildly depending on the industry and organization.

8. Not browser independent

This is a minor limitation that can be a major annoyance. The report that TMG produces is a HTML report that only displays correctly in Internet Explorer. As Forefront TMG is a Microsoft product, this is not exactly surprising, but still very annoying if IE is not your default browser.

How to get awesome reports from Forefront TMG

If you have had personal experience with any of the above limitations, you’ve probably been hunting for an alternative solution. I strongly recommend checking out the WebSpy Vantage range of products, and if you would like secure report distribution via the ‘Web Module’, Vantage Ultimate is what you are after.

If you agree or disagree with anything in this article, I encourage you to leave your thoughts in the comments.

Cheers!

Scott

In the mean time, you can do it by editing an XML file manually.

If you go to the Web Module’s data folder you will find a file called “Vantage Web Module.Reports”.  If you open this in Notepad, you’ll notice chunks of xml for each report:

<WebReport>
<Guid>89208266-42e5-44bc-baa2-157c404c9688</Guid>
<Title>Business Unit Report</Title>
<Date>633945813293343143</Date>
<Type>Analysis</Type>
<Access>
<Everybody>False</Everybody>
<Attributed>True</Attributed>
<Specific>True</Specific>
<SpecificEntities>
<item>person:CN=Luke,OU=Users,OU=Australia,OU=Webspy,DC=wsy,DC=com</item>
</SpecificEntities>
<Managers>True</Managers>
<ManagerLevelRestriction>3</ManagerLevelRestriction>
</Access>
<Attribution>person:CN=Scott,OU=Users,OU=Australia,OU=Webspy,DC=wsy,DC=com
</Attribution>
</WebReport>


Depending on how you published your reports, the unique ID of the person that currently has access to the reports will be mentioned in either the ‘SpecificEntities’ or ‘Attribution’ section.

You just need to find/replace this with the unique ID of the person you would like to transfer these reports to. You can find the unique ID of a person on the Organization screen in Vantage.  It’s called ‘Distinguished Name’:

Finding a user's unique ID in Vantage

A few more things:

• Make a backup of your original “Vantage Web Module.Reports” file before making any change.
• As you can see above, people need to be entered into this XML file using the syntax
  “person:<uniqueID>“
• You will also need to stop IIS before making any change as the web module caches this data in its memory while running.

As mentioned, creating a user interface to do this is a planned feature so follow us on Twitter, or subscribe to our RSS feed for updates!

In the mean time, I thought I’d share a way to re-brand the main elements in the Web Module by editing a few files and replacing a few images.

The only issue with this technique is that any future auto-updates for the Web Module will overwrite your edited files, so you just need to keep a copy of your customized files, so that you can restore them again after the auto-update.

Before you begin

In order to edit anything, you first need to know where your Web Module is located on your web server’s hard drive. This can be found by opening IIS Manager (Start | Control Panel | Administrative Tools | Internet Information Services (IIS) Manager) expanding the left hand server/site tree to find your Web Module.

• In IIS7, select the Web Module and click Basic Settings… in the right hand ‘Actions’ panel. The location is specified in ‘Physical Path’.
  

  Finding the Web Module's physical path in IIS7

• In IIS6, right-click the Web Module and select Properties… then go to the Home Directory tab. The location is specified in ‘Local Path’.

Windows may also prevent you from editing these files directly due to permissions issues. I’ve found a good technique is to copy the files you want to edit to your desktop, edit them, and then copy them back into the Web Module’s physical path. Windows will then prompt you to elevate to administrator and the copy/replace will succeed.

Ready To Go…

There are a few places where the WebSpy logo and WebSpy Text is presented.

• The login page
• The header bar
• The welcome Page
• Report cover pages

The Login page

The logo displayed on the login page can be found at /images/logo.png. Replace this image with your own logo. Then open Default.aspx in the Web Module’s root folder in a text editor such as notepad, and replace the following line

<img runat=”server” alt=”WebSpy” src=”~/Images/Get.ashx?image=Logo” />

with

<img runat=”server” alt=”WebSpy” src=”~/Images/logo.png” />

Before

Web Module's Login Page Before logo.png Change

After

Web Module Login Page After logo.png change

The header bar

The header bar utilizes the image located a /Images/bauble.png. Replace this image with your own custom image.

Then open Navigation.Master in the Web Module’s root folder in a text editor such as notepad, and replace the following line

Also look for the text:

Before

Web Module's Header Bar Before Bauble.png and Text Changes

After

Web Module's Header Bar After Bauble.png and Text Change

The Welcome Page

When you first login to the Web Module, you are presented with a Welcome Page. The first line on this page reads “Welcome to the WebSpy Vantage Web Module. You can change this by editing the first line in the Welcome.aspx file located in the Web Module’s root folder. Edit the section in bold below:
<%@ Page Language=”C#” MasterPageFile=”~/Navigation.Master” AutoEventWireup=”true” CodeBehind=”Welcome.aspx.cs” Inherits=”WebSpy.Vantage.WebModule.Welcome” Title=”Insert Custom Text Here” %>

Before

Web Module's Welcome Page Before Text Change

After

Web Module's Welcome Page After Text Change

The Report Cover Pages

The Image used on the cover page of reports is much easier to change.

1. Login to the Web Module as Administrator
2. Go to the Options Tab
3. Click ‘Report Logo’ under Web Module Options
4. Click Choose File, and select the image or logo you would like displayed on your report cover page
5. Click Upload

Before

Web Module's Report Cover Page Before Report Logo Change

After

Web Module's Report Cover Page After Report Logo Change

Summary

The changes above cover a majority of the areas your users will come into contact with in the Web Module. There may be a few more instances of the word “WebSpy” but for the most part, it should just be a matter of opening the relevant .aspx file and editing the html.

As I mentioned, if you auto-update the Web Module (via the system tray icon on the Web Module server), your edited files will be overwritten. I recommend keeping a copy of your edited files in a safe place outside the Web Module’s physical folder, so that you can copy them back in after the update. If the only changes you make are the ones above, then you’ll need to keep a copy of:

• /Navigation.Master
• /Default.aspx
• /Welcome.aspx
• /Images/logo.png
• /Images/bauble.png

We will also be adding the functionality to make these changes ‘properly’ in a future build, so follow us on Twitter, or subscribe to our RSS feed for updates!

Cheers!

Most frequently suggested bypass methods includes the use of public proxies, circumventors and http tunneling. I don’t wish to go into details on any of these methods as their use is NOT recommended. However, it does prove a point: The main reasons organizations block certain websites is to prevent security risks and unproductive internet usage. Although, it is an indisputable fact that employees’ use of virus ridden public proxies, and other elaborate methods, to overcome blocking efforts can in fact increase security risks and unproductive behavior – making matters even worse.

Obviously all employees do not take these measures, but isn’t it enough that some do? Yes, the same high risk and time consuming bypassing “techniques” could be used when trying to stay anonymous from internet monitoring software. However, there are two main differences:

1. Using internet monitoring software reduces the need to block. Employees will be able to access the legitimate sites that often end up blocked thanks to a “block worthy” word in a corporate blog, or something of similar virtuousness. Not blocking means less time and effort spent trying to bypass blocking solution. After all, my mailbox is not full of alerts on how to bypass internet monitoring software.
2. Using internet monitoring software will allow employers to detect who is up to no good trying to bypass blocking rules or browse anonymously. For example, if an employee continuously use public proxies or tunneling, an internet monitoring solution (or at least a good internet monitoring solution) can assist the employer in tracking down the offender. (Please have a look at “How to Improve Public Proxy Management” blog for more info.)

This blog simply adds to the convincing case against organizations’ excessive use of blocking and filtering solutions. Porn sites, known malicious virus and phishing sites – by all means, block the living daylight out of them. But as for the rest, as for news site, online shopping sites, social networking and general interest sites – Don’t block, monitor.

I want to avoid repeating myself so please have a look at previous blog for the full story on “The Cost of Blocking Employee Internet Usage”

Unfortunately, the WebSpy Twitter account fell victim to a phishing scam, and as a result sent phishing spam to all our Twitter followers. We are embarrassed by the incident and we apologize to all of our followers, especially the ones that clicked the link in the DM and were caught by the phishing scam themselves.

Here’s a rundown of the event in the hope that it will help others know what to look out for.

What Happened?

The phishing scam works like this:

1. You receive a strange yet intriguing Direct Message from someone you follow and likely trust. This is the key element to the scams success.
2. The DM contains a link using a shortened URL such as dwarfurl.com/blah. In our case, most of them were using dwarfurl.com, wapurl.co.uk, and 3.ly
3. You click the link and get taken to what appears to be the Twitter login page. But if you look at the URL it is actually something like blogs.videos.dsfasdc.com or videos.twitter.dsfasdc.com. Checking the URL is the key to making sure the scam doesn’t get you too!
4. You enter your Twitter login details. Reports of what happens after this login page vary. You may see the Twitter fail whale, or a blank page, or a random blog.
5. Now that the phishing site has your login details, the same Direct Messages is sent to all your Twitter contacts.
6. You eventually discover what happened. You feel like a violated idiot and start scrambling to fix everything.

What to do if it happens to you

If the above sounds familiar, you need to login to Twitter right now and change your password to make sure the phishing site can no longer access your account. You also need to go to the Connections tab and disable any third party applications that look suspicious. You’ll then need to update the credentials in all the twitter clients, website/blog plug-ins, and anything else that may be using your old Twitter credentials.

Fortunately, we were still able to login to our Twitter account and change our password and disable third party connections. Thankfully there were not any new suspicious connections that we needed to worry about.

Lessons Learned

Now that we’ve fixed everything and regained control of our Twitter account, it’s good to sit back and reflect on what just happened and how to avoid it in the future.

You’ve probably heard all of this before. We had too. But it takes an incident like this to really think about and address any shortfalls in your own organization. Some of our followers were also caught out by the scam and these are people that are in the tech industry and generally know about these sorts of scams. We were definitely surprised that we fell for it! So take a moment of your time to imagine your own Twitter account was compromised in the same way, then imagine all the possible ways it could have happened. Now go and take every precaution to ensure it doesn’t happen.

Having now been through it, here are some tips to help you avoid the same fate in the future.

1. Just because a Direct Message comes from someone you trust, does not mean it is trustworthy. Always use caution!
2. Educate your employees – especially those that know your company’s Twitter credentials. The main goal you want to achieve here is getting your employees into the habit of glancing at the URL in the address bar of their browser before entering ANY login details. We used our own log analysis software (Vantage) to find out who ended up on the websites in question, and then spoke to them directly to ensure they understood what to look out for.
3. Use a Twitter application that can display the actual URL behind a shortened URL before clicking on the link. For TweetDeck users, go to Settings | General, and check ‘Show preview information for short URLs’. Please note, however that this function only works for a few specific URL shortening services.
4. If you’re using the Twitter web page directly, use a browser and plug-in that can expand shortened URLs such as Mozilla Firefox with Long URL Please.
5. Use a browser with integrated anti-phishing security (such as Firefox or Google Chrome) and keep it up to date, or ensure you have good third party anti-phishing / anti-malware software installed.
6. As always, keep your security software and OS up to date.

Our friends at Sophos also have some good information about the scam that you may like to read: http://www.sophos.com/blogs/sophoslabs/?p=7366

Sorry!

An event like this makes you realize how important Twitter is to the overall public perception of a company. Our followers trust us to deliver relevant and useful content about our key areas of expertise – log file analysis and reporting. We spend a large amount of effort researching and writing content to ensure our tweets provide our followers with a good source of information. Having a breach like this certainly degrades this public perception that we work so hard at trying to maintain.

I would therefore like to thank all our followers who have kept with us and not clicked the ‘Unfollow’ button. Now that everything is under control again we will continue to bring you the best content we can provide about the log analysis and surrounding industries.

Once again, many many apologies to all of our followers, especially those that were affected.

Vantage’s Storage Location

If you’re using Vantage Premium or Giga, there’s only one location you need to be aware of. That is where Vantage keeps its storages (Vantage’s custom database that log files are imported into). This setting is easily changed by going to Tools | Options | Paths and double clicking the Storages path. Easy.

Web Module Storage Locations

If you’re using Vantage Ultimate, you also need to be aware of the Storage location mentioned above, but you also may need to adjust where the Web Module stores its data.  There are two locations you need to be aware of here:

1. The Web Module Data Location
   This is where the Web Module permanently keeps it’s storages, reports and settings
2. The Windows temporary folder
   This is where Vantage keeps storages while they’re being processed before uploading them to the Web Module’s data location

The Web Module Data Location

The data location for the Web Module is specified during installation and defaults to C:\Vantage Web Data. If you have already installed the Web Module, you can change this location using the following steps:

1. Find the Web Module’s Web.Config file. The Web.Config file can be found in the Web Module’s physical folder. If you don’t know where the Web Module’s physical folder is:
   1. Open Microsoft IIS (Control Panel | Administrative Tools | Internet Information Services (IIS) Manager)
   2. Select the Web Module in the left hand side (e.g. Server-> Sites ->Default Web Site -> webmodule).
      • If you’re using IIS 6, Right-click the Web Module site and select Properties then go to the Home Directory tab to find the physical folder.
      • If you’re using II7, select your Web Module site and click Basic Settings…
2. Open the Web.Config file in Notepad.
3. Find the line that looks like this:

   <add key="SettingsPath" value="C:\Vantage Web Data"/>

4. Change c:\Vantage Web Data to the location you would like to use and save the file.
5. In Windows Explorer, copy all files and folders from C:\Vantage Web Data (or where ever your original location was) to the new location you specified in step 4
6. Restart IIS by going to Start | Run and type iisreset /restart

The Windows Temporary Folder

The Windows Temporary folder can also be modified, but please note this is a system wide change.

1. Right-click ‘My computer’ and select Properties.
2. Go to Advanced and Click the Environment Variables button
3. Change the location for the ‘TEMP’ and ‘TMP’ environment variables (do not use the same location specified in step 4 above)

Vantage and the Vantage Web Module (Ultimate only) will now use your new locations to temporarily and permanently keep your Storage files.

In an attempt to prevent the use of public proxies, it is common practice to subscribe to, or collect, regularly updated lists of public proxy sites. Many of these lists are freely available on the Internet (see examples at the bottom of the page).

We recommend the following procedures, using WebSpy’s solutions, to improve management and control of public proxies use

CREATE PUBLIC PROXIES PROFILE

• In your Summaries section, run an analysis, then go to Sites / Site Name
• Right click on any site known to be a public proxy site and chose `Include in profile’
• Either create a new profile called `Public Proxies’ or add to your existing `Public Proxies’ profile
• If you suspect a site to be a public proxy, simply right click and choose `Browse’ to investigate further

UPDATING PUBLIC PROXIES PROFILE

• Locate and copy a public proxy list published online
• In your Profile section, open your `Public Proxy’ profile and paste the list
• Even though a little bit time consuming, maintaining a list of the most common public proxies will increase your chances of easily locating novice public proxy culprits

INVESTIGATING INDIVIDUAL USERS

• In your Summaries section, run an analysis, then click on `Users’
• Right click on your selected user, then choose Drilldown | Sites / Site Name
• Many public proxies use IP addresses (as opposed to site names) to avoid easy detection, so a spike in IP address visits could be an indication that an employee or student may be using a public proxy
• Right click any IP address and choose `Browse’ to investigate further

(Please ensure that comprehensive Acceptable Usage Policies, prohibiting the use of public proxies, and breach consequences are explicitly communicated to employees)

There are numerous websites publishing updated public proxy lists online that can easily be located through search engines. Below are just a few examples:

http://www.publicproxyservers.com/

http://www.proxy4free.com/

http://bestproxy.info/

http://tools.rosinstrument.com/proxy/

http://www.fresh-proxy-list.net/

Don’t hesitate to contact us for further information.

The attack was quickly under control and IT updated firewall rules to prevent any employees from accessing the particular site again. However, instead of immediately blocking further emails from the malicious sender, the IT department saw this as an opportunity to educate the workforce about phishing attacks. They used WebSpy Vantage to report on their firewall and identified all employees who tried to access blocked phishing sites. As employees were identified, IT started organizing informal meetings to inform them about phishing, how to recognize attacks and the most common phishing techniques used.

Because IT was able to pinpoint exactly which employees needed more information about phishing attacks, these smaller meetings showed to be very effective. The employees attending the meetings naturally realized their phishing knowledge was insufficient and were eager to find out more as they didn’t want to make the same mistake again, at work or at home. The government department is still the target of phishing attacks but their employees are now educated enough to identify them before any harm is done.

If you want to share your monitoring and reporting experiences either comment below or email me directly at asa@webspy.com.

To use Windows Authentication, there are just a few things you need to do.

1. Set the Web Module’s Authentication type to IIS Integrated, and add your administrators in the form of domain\username
2. Enable Windows Authentication and disable Anonymous authentication in IIS.
3. Ensure all users in your Organization screen have a login name in the form of domain\username. Use the ‘Prefix’ option to prefix “domain\” (without the quotes) to your usernames names when importing your Organization from LDAP or LDIF.
4. Connect Vantage and the Web Module using the new authentication details and synchronize your Organization.

1. Set the Web Module’s Authentication type to IIS Integrated, and add your Administrators.

When you first install the Web Module, the first screen you see is the ‘Initial Configuration Wizard’ that guides you through the process of selecting your authentication type and specifying your administrator(s). If you have already been through this Wizard and are currently using Vantage In-Built or Client Certificate authentication, you can easily reset this initial configuration wizard. Simply login to the Web Module with your current administrator details and go to Options | Maintenance | Reset Initial Configuration Wizard.

Note: You can also change your authentication and administrator options individually using the Authentication and Administrator options on the Options tab of the Web Module.  However, for ease of demonstration, I’ll use the Initial Configuration Wizard method.

Now that you’re at the Initial Configuration Wizard, proceed through the wizard, selecting IIS Integrated authentication and entering your administrators in the form of domain\username (replace domain with your organization’s AD domain, and username with the sAMAccountName of your administrator.

Initial Configuration Wizard - Welcome Page

Initial Configuration Wizard - Authentication Page

Initial Configuration Wizard - Delegate Administrators Page

Initial Configuration Wizard - Summary Page

Click Finish, and if the authentication was successfully changed, you should get a message saying ‘The specified credentials were not accepted”.

The 'Specified credentials were not accepted' message.

Don’t panic at this point. This message is an indication that the authentication was successfully changed and that the Web Module is now listening for IIS to pass through Windows Usernames. The reason you’re getting this message is because IIS is not yet passing through Windows Usernames to the Web Module. This is configured in the next step.

2. Enable Windows Authentication and disable Anonymous authentication in IIS.

Now that the Web Module is expecting IIS to authenticate your users, you need to set up  IIS  to do this.

1. Open IIS by navigating to Start | Control Panel | Administrative Tools and double-clicking on Internet Information Services (IIS) Manager.
2. Navigate to the Web Module site or virtual directory in the left hand ‘Connections’ Panel. It will be located under <Server Name>\<Sites>. For example, MyServer->Sites->Default Web Site->webmodule.

• If you’re running IIS7 ( Windows Server 2008, Vista or Windows 7)
  1. Select the Web Module site and ensure the ‘Features’ tab is selected at the bottom of the middle pane.
  2. Double-click the ‘Authentication’ feature.
  3. Right-click ‘Anonymous Authentication’ and select Disable
  4. Right-click and ‘Windows Authentication’ and select Enable
  5. Restart IIS by selecting your server in the right hand connections pane, and clicking Restart in the ‘Actions’ pane on the right.

• If you’re running IIS6 or 5.1 (Windows Server 2003, Windows XP)
  1. Right-click Web Module site and select Properties.
  2. Go to the Directory Security tab
  3. Under ‘Authentication and access control’ click the Edit button.
  4. Uncheck ‘Enable anonymous access’ and check ‘Integrated Windows authentication’
  5. Restart IIS by right-clicking the local server, select All Tasks, and then click Restart IIS.

If you added your own Windows login name as an administrator in step 1, you can now test the authentication is working. Go back to the Web Module in your browser and click Refresh. You will be presented with an ‘Authentication Required’ dialog where you can enter your username and password.

Authentication Required Dialog

Again, ensure your username is in the form of domain\username. Click OK, and you should log straight into the Web Module using Windows Authentication.

3. Ensure all users in your Organization screen have a login name in the form of domain\username

Now your administrator account can log into the Web Module using Windows Authentication, but all other users will not be able to log in unless they have their login name specified in the form of domain\username. This is done in Vantage Ultimate on the Organization screen.

Organization Screen showing correct login name for Windows Authentication

If you’re importing your users from LDAP or LDIF, make sure you use the ‘Prefix’ option on the User Details page to prefix domain\ before your imported usernames. For example:

4. Connect Vantage and the Web Module using the new authentication details, and synchronize your Organization.

In order to publish information to the Web Module, you need to add a connection between Vantage and the Web Module. This is done on the Web Module screen in Vantage Ultimate.

1. Click Add Web Module (or if you already had a web module before changing the authentication details, select it and click Properties)
2. Enter the server & virtual directory of the Web module, and enter the correct credentials ensuring domain is specified.
3. Click OK to connect.

Connect to Web Module dialog

Once connected, synchronize Vantage with the Web Module by clicking the Synchronize link in the Web Module task pad. You may also want to provide permissions for your users in the Permissions section on the Web Module screen.

That’s it. You can now test that everything is working by getting one of your users to access the Web Module’s URL. They should sail straight in with no username/password prompt.

I hope this helps! Please let me know your feedback by emailing me, or leaving a comment below.

If you’re using WebSpy Vantage, you are probably interested in filtering your log file imports by date (only import files from the month of June for example). The obvious way to do this is to specify a date filter using the filters page in the Input Wizard. The problem is Vantage will still check every record in every log file being imported to see if it matches the date filter. If you have months or years worth of logs in the folder being imported, that’s a lot of data that Vantage has to pointlessly sift through.

The good news is, if your log files contain the date in their file name, then you can use file masks to instruct Vantage to never touch these unwanted files.

A bit about file masks…

You can specify file masks such as *, *.log, *.gzip, *WEB*.w3c, etc to import logs with specific file extensions, or with specific strings in the file name (such as WEB or FWS to import only Microsoft ISA Web Proxy or Firewall logs respectively).

But if your log file contains the date in the file name, you can also use date modifiers in the file mask to select logs from a particular month, date or year.

Say you have log files that look like this:

• 20090801.log

• 20090802.log

• 20090803.log

and so on..

You can create a simple file mask to only import log files from the month of August very easily using 200908*, or 200908*.log.

Using date modifiers in file masks

But if you’re using Tasks to automatically create a new storage each month, you don’t want to have to worry about manually changing the file mask to 200909*.log when the first day of the next month rolls around.

So intsead, you can use a date modifier in the file mask that will automatically select the logs for the current month, every time your task runs. For the above example, the file mask looks like this:

• %[yyyyMM]* (you can also use %[yyyyMM]*.log)

When the task runs, %[yyyyMM] will be replaced with actual values from the current date. So if the task runs on the 1st of August 2009, the file mask will become 200909* (or 200909*.log).

Dealing with different date formats

You can also use date modifiers for log files that look like this:

• 2009-Aug-01.log

• 2009-Aug-02.log

• 2009-Aug-03.log

In this case the file mask looks like:

• %[yyyy-MMM]* - notice the three MMM's as opposed to two MM's used previously.

Vantage uses the custom date and time format strings available in the .NET framework, so for more information on whether to use m or M or MMM, please refer to this article http://msdn.microsoft.com/en-us/library/8kb3ddd4.aspx

Importing logs from previous months

If you would like to import logs from a previous month, this can also be done by adding an additional element to the date modifier. For example, to import the previous months logs you can use:

• %[-1m,yyyyMM]*

Notice the -1m meaning ‘minus one month’. You can also use -1d (for minus one day), or -1y (for minus one year).

More examples

Here are some more examples to give you an idea of what is possible using date modifiers.  Assuming the date is 14th of August 2009:

• %[-1y,yyyyMM]*.log will create a file mask of 200808*.log

• %[yyyy-MM-dd]*.log will create a file mask of 2009-08-14*.log

• %[-4d,yyyyMMdd]*.log will create a file mask of 20090810*.log

• %[1-m,-4d,yyyyMMdd]*.log will create a file mask of 20090710*.log

• %[-1y,1-m,-4d,yyyyMMdd]*.log will create a file mask of 20080710*.log

• ISALOG_%[-1m,yyyyMM]*_WEB_*.w3c will create a file mask of  ISALOG_200907*_WEB_*.w3c

• *%[-1m,yyyyMM]* will create a file mask of  *200907*

Adding a file mask

File masks are configured on the Input Selection page of the Input Wizard, when you select Add | Folder.

Adding a File Mask

There is also an option to save the literal date into the file mask when the task is run.  For more information on this option, please see my previous blog about this feature.

Other uses for date modifiers

Date Modifiers are a great way to speed up log file imports, but you can also use them when specifying storage names as well as report names. For example, if you specify a storage name of %[yyyyMM]_storage, this will create storages with the names 200907_storage, 200908_storage and so on. When selecting the storages to report on, you can click the Add button on the storage selection toolbar in the Report Wizard, and specify storages such as %[-1m,yyyyMM]_storage, to report on the previous month’s storage.  For more information, please see Automatic Importing and Reporting using Tasks.

I hope this helps someone out there. Let me know how you go!

We’ve got a great deal at the moment where you get 20% off Live and Sentinel if you purchase them online with Analyzer Standard.

If you convert your logs to text using this script, they won’t import into WebSpy Vantage or Analyzer due to an extra line break in the header of the file (after #fields:).

We’ve therefore created a modified version of the script that creates compatible log files for WebSpy software.

Download the modified MSDEToText script:
MSDEToText.zip -26 KB

Also make sure the file names of your output log files contain the word WEB (for Web Proxy logs) or FWS (for Firewall Logs) as Analyzer and Vantage use these strings to automatically detect the type of ISA log file.

Happy converting!

Watching a single video on YouTube will probably generate a list of about three to five sites such as lax-v41.lax.youtube.com, www.youtube.com, img.youtube.com, and so on. Your list of top sites also probably contains hits to ad servers and tracking servers, such as doubleclick.net, google-analytics.com and imrworldwide.com. All this clutter gets in the way of determining what sites were ‘intentionally’ visited.

Fortunately there are a few simple steps you can take to exclude this information from your reports. Watching is much easier than reading, so I thought I’d create a video demo to walk you through the process.

By the way, this is the first video demo of what I hope will be many more to come. I created it using TechSmith’s Camtasia Studio which is by far the best screen recording software I’ve used. All the zooming you see throughout the demonstration is completely auto-magical! It’s a brilliant piece of software that has saved me hours of time. Props to the guys at TechSmith! The one pitfall of Camtasia is that it seems to make me sound like a geek with a raw Aussie accent… I hope they fix that in the next version.

Anyway, I hope you find this useful.

• Windows Vista & Server 2008:
  C:\Users\<user profile>\AppData\Roaming\WebSpy\Vantage <edition> <version>
• Windows XP & Server 2003:
  C:\Documents and Settings\<user profile>\Application Data\WebSpy\Vantage <edition> <version>

When Vantage opens, it loads the information contained in these files.

To move all your settings across, you can simply copy all the files in this location on your original machine, into the same folder on the new machine (make sure Vantage is closed when you do this).

When you open Vantage on the new machine and you’ll have all your settings moved across.

If you’re using scheduled tasks, one thing to note is that copying these files will not recreate the Windows scheduled task jobs. To get your tasks functioning on the new machine:

• Go to the Tasks tab and double-click each scheduled task.
• Proceed through the wizard and make sure all the settings are correct.
• Enter your authentication details on the last page and click OK.

This will create a new Windows scheduled task job which should run as it did on the old machine.

Please note that this process only moves ’settings’ across to the new machine. If Storages and Reports were being written to a location on the old machine, then you will need to move these across to the Storages and Reports locations on the new machine (check Tools | Options | Paths for these locations).

Also take access privileges into account. If you’re old installation was set to create storages in \\server\mystorages, make sure the new machine also has write privileges to this location.

That’s pretty much all there is too it. Happy migrating!

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
	<appSettings>
		<add key="synopsisCallStack" value="true" />
		<add key="traceLog" value="true" />
	</appSettings>
	<system.diagnostics>
		<switches>
			<add name="LogOutput" value="4" />
		</switches>
	</system.diagnostics>
</configuration>

After creating the file, restart Vantage. A trace log will be created in your local application data folder;

XP; C:\Documents and Settings\%username%\Local Settings\Application Data\WebSpy\Vantage (edition)\(build)\trace.log

Vista; C:\Users\%username%\AppData\Local\WebSpy\Vantage (edition)\(build)\trace.log

The trace log shows the list of steps performed, and the details of any errors encountered. This information is useful to WebSpy when diagnosing errors reported by users.

I went hunting through the options in the Virtual PC UI but didn’t find anything related to disabling this option. A bit of googling later and I’ve got it disabled by adding a few lines of XML to the .vmc file.

First, make sure your VM is shut down and Virtual PC is closed.

Then find your .vmc file and open it in a text editor such as Notepad.  By default, Virtual PC creates .vmc files in My Documents\My Virtual Machines.

Find the </microsoft> tag and insert the following lines directly above it:

<components>
<host_time_sync>
<enabled type="boolean">false</enabled>
</host_time_sync>
</components>

For example:

Disabling the host time synchoption option in a .vmc file

Then Open Virtual PC, start your VM and you’re all done!

This was tested on Windows Vista 6.0.6001, SP1 using Microsoft Virtual PC 2007 (6.0.156.0)

Unfortunately, Exchange 2007 tracking logs are not used to simple questions, and are likely to return a complicated and / or misleading answer.

But the confusion it seems, all comes down to definitions. Once you understand these definintions, things start to make a bit more sense.

What is an Email?

If you send an email to one person, you’ve sent one email. But if you’ve sent that same email to 500 people, have you sent one email, or 500? I will take a guess, and say that a large majority of you will want to see 500 in your reports.

Microsoft Exchange 2007 tracking logs contain an excellent field called ‘Message ID’. If you send an email to someone, that message is uniquely identified by a Message ID that persists though Exchange’s various functions for the lifetime of the message.

At first glance, it seems that counting Message IDs will give us what we want. But if you send the same email to 500 recipients, all those emails get the same unique message ID. So counting message IDs will show us that only one email has been sent. No good.

Then next obvious step is to count the number of recipients that received the email.

What is a Recipient?

The definition of recipient can also get clouded when you start talking about distribution lists. If you send an email to one real person, then that is one recipient. If you send the same email to five real people then that is five recipients. If you send an email to an internal distribution list, the number of recipients is the number of people that are members of that distribution list.

If you send an email to an external distribution list (such as SalesDL@othercompany.com) this will only be recorded as only one recipient, as your Exchange box has no way of knowing how many real people are members of that DL at the other company.

How do I count Recipients?

Again, Exchange Tracking logs contain another excellent field called ‘Recipient Count’. But don’t get carried away as this too can be misleading.

Without going into specifics, Exchange has a bunch of internal functions to deal with an entire message transmission. The tracking logs files contain another excellent field called Internal Message ID that identifies each of these processes per-message.

Unfortunately, each Internal Message ID contains its own value for ‘Recipient Count’. So when you sum the Recipient Count field for a single message, the final result may be much larger than the actual number of real recipients.

To illustrate, WebSpy Vantage imports Recipient Count into a Summary of the same name. Here is a screenshot of the Recipient Count Summary for one message

The Recipient Count Summary for a Single Message

As you can see, there are multiple rows of individual Recipient Counts. The first row, is actually correct. This email was actually sent to 961 people. But there are additional entries where Exchange performed an internal operation with a subset of those messages. Therefore, summing the Recipient Count field for a message is also no good.

Counting recipients “properly”

The best way to count recipients is to use WebSpy Vantage to import your logs, then drilldown into a message to the Recipients summary and look at the total number of recipients at the bottom. Alternatively, add a Count Distinct aggregate for the Recipients summary to any report template.

Here’s a screenshot of the Recipients summary:

The Recipients Summary showing Total 'Real' Recipients

And here’s a screenshot showing how to add the aggregate to a report template:

Adding the Number of Recipients to a report template

Counting Total Number of Emails

The above screenshot will give you a count of all the recipients you have ever sent email to. However, what you really want is a count of recipients per message. You can do this by concatenating the Recipient with the Message ID, and counting the total number of rows. To do this, edit the Number of recipients aggregate column above and enter [Recipient] + [MessageID] in the ‘Custom’ edit box.

Customizing an aggregate column to concatenate Recipient and MessageID

Exchange 2007 Report Templates

You can download a WebSpy Vantage Templates file here that includes three reports (Email Overview, User Email Activity, and Email Trends) that uses columns such as Number of Emails, Number of Unique Messages and Number of Recipients.

Tip: You can convert any email template that has the schema ‘All Mail Schemas’ into an Exchange 2007 template in order to report and filter using all the fields available in Exchange 2007.

To do this:

1. Right click an ‘All Mail Schema’ email template and select Duplicate.
2. Select Microsoft Exchange 2007 from the schema drop down and click OK.
3. When you edit the nodes in your new template, you will have access to all the fields that Exchange 2007 records.

Cheers!

So here it is (you might want to grab a coffee first!).

Unless you have a sophisticated end point security or file auditing solution in place, you’re pretty much limited to the quality of data found in your Windows Security Event log. By default, accesses to your confidential files are not going to trigger any entries to be written to the Event log. You first need to setup file or folder auditing.

WebSpy have written a nice article to help you out with this: Managing Event Logs

Personally, I’m running Windows Vista SP1.  So I first turned on Object Access auditing by going to Control Panel | Administrative Tools | Local Security Policy | Local Policy | Audit Policy and set Audit Object Access for Success and Failure.

Windows Vista Local Security Policy

In Windows Explorer, navigate to the folder or files to audit, then Right-click | Properties | Security | Advanced | Auditing and click Continue when Vista’s User Access Control gets in the way.  Here you get the option to add Users or Groups to the audit policy. So if you only want to know when Joe Bloggs access the file/folder, then only add Joe Bloggs. If you want to know when anyone accesses the file/folder then add your entire company.

Scroll….

Click OK and apply the changes. If applying this to a folder, take note of the setting to ‘apply the auditing entries to containers within this container’ at the bottom and use as required.

Congratulations. That’s the auditing setup. Once people start accessing these files(s), the auditing information will get recorded to the Security Event Log on the machine that hosts the file(s) in question.

The next step is to import the Windows Security log into your flavour of WebSpy Vantage. I’m using Vantage Ultimate, but the steps are the same for Premium and Giga.

1. Run Vantage (as Administrator if on Vista)
2. Go to the Storages tab and click Import Logs
3. Run through the Import Wizard with these settings:

• Storage: New storage
  

  Input Dialog: Storages Page

• Input Type: Windows Event Log
  

  Input Dialog: Input Type Page

• Loader Selection: Microsoft


Input Dialog: Loader Selection

• Input Selection: Add
  Select either local computer, or multiple computers, enter authentication details and Click ‘Filter Event Logs’. Check the ‘Security’ Log and click OK.
  

  Input Dialog: Input Selection Page - Adding Event Logs

• Click OK to start the import.

If there are any issues with the import process, consult these three WebSpy Knowledgebase articles to do with issues importing event logs:

• Event Log Troubleshooting (Known Issues and Fixes)
• Importing Event Logs from machines on a different domain
• Required Services for Event Log Importing

The first article came in handy for me as I’m running on Vista and in order to import from the Local Security log, you need to run Vantage as Administrator. To do this, go to C:\Program Files\WebSpy\Vantage Ultimate 2.1\ right-click the WebSpy.Vantage.exe and select ‘Run as Administrator’.

Once data has been imported into your storage, check it out on the Summaries screen.

To to the Summaries Tab, Run an Analysis on your new storage (ad-hoc analysis will do) , and go to the Category Summary. There should be some ‘File System’ items there assuming the file has been accessed since setting up file auditing. You can then drilldown to Event Type to see ‘Audit Success’ or ‘Audit Failure’. To see who has Successfully accessed a certain file, drilldown into the ‘Audit Success’ item.

Unfortunately the good stuff is buried in the ‘Message’ field, which you can only access in the Individual Records view. This is because the Message field in Event logs is free form and could vary wildly resulting in millions of unique items. A Message Summary has therefore been excluded from a default ad-hoc analysis for very good performance reasons.

Event logs can also be quite verbose, and if you drilldown to Individual Records at this stage, you’ll see lots of messages like ‘A handle to an object was requested’ which probably isn’t of any great value from a reporting perspective. One way to filter out this noise is by Event ID.

I’ve discovered that the events that correspond to ‘An attempt was made to access an object’ have the ID 4663. (One day I’ll create an alias to map Event IDs to their meaningful description. If you come across a good  resource I can use for this, let me know!).  So go to the Event ID summary and drilldown into 4463 to the Individual Records view.

Once you’re at Individual Records, you can hover over the message field to get details. You can also use the find edit box to search for a particular user or file:

Drilldown into Successful File System Accesses (Event ID 4663)

You can export this view To Word Document, HTML, Text or CSV by right-clicking the Individual Records summary and clicking Export.

You can also create a report template to access this same information, but as there is no ‘Message’ summary to choose from, you need to use the Custom expression options, both when adding a column to a node in a Template, and when specifying your filter.

To add a column to a report that displays an Event Message:

1. Go to the Reports Tab and click New Template
2. Create an Analysis template based on the ‘All Windows Event Schemas’ schema
3. Click New Node and click the Advanced button to launch the Advanced editor.
4. On the General page, delete any existing Key columns and select Add | Key. In the Custom Expression section enter [Message] (include the square brackets) and click OK.

To filter the report:

1. Go to the Filters page of the New Node dialog (alternatively you can specify this filter in for all nodes using the Template Properties dialog)
2. Click Add | Field Value Filter. Select Category from the Summary drop down, and click Add. Enter ‘File System’ (without the quotes) and click OK. Click OK to add the filter.
3. Click Add | Field Value Filter. Select Event ID from the Summary drop down and click Add. Enter ‘4463′ (without the quotes) and click OK.
4. To filter on the Message field, Select Add | Manual Filter Expression.
5. Enter the expression:
6. [Message] LIKE “text to filter for”
   Change ‘text to filter for’ to the user or file that you want to search for. If you want to search for multiple strings, repeat the above expression separated by an AND or an OR, and place brackets wherever it makes sense. For example:
   • [Message] LIKE “scottg” AND [Message] LIKE “.avi”
     Will filter for all .avi files that scottg has accessed.
   • [Message] LIKE “scottg” OR [Message] LIKE “.avi”
     Will filter for any file that scottg has accessed and any avi that anyone has accessed.
   • ([Message] LIKE “scottg” AND [Message] LIKE “.avi”) OR [Message] LIKE “andrew”
     Will filter for any all avi files that scottg has accessed and any file that Andrew has accessed.
7. You can add the individual filters using Add | Manual Filter Expression multiple times, and then using the Manual Filter Expression editor at the bottom to change ANDs to Ors and place brackets appropriately, like so:
   

   Filtering for File Access Events by particular users

8. Right-click the Manual Filter Expression edit box and select Validate to make sure everything is good with the expression.
9. Modify chart settings, sorting, etc as appropriate.

Here’s the resulting report template for you, but please note that it includes the filter above (events for the user’s  ‘Asa’ and ‘Scottw’), so you will need to modify the filter and enter the users or files you want to filter on. Just use the user’s windows login name, and/or the name of the file.  Alternatively, remove the filter altogether if you want to see all File Audit events.

That’s it! Now run your report, automate it using the Tasks screen, and your set!