The next morning I woke to find several more messages from my VPS host, each with a higher and more significant excess transfer than the last. At this point it occurred to me that it was unusual for my VPS to reach its quota, let alone exceed it. The excess transfer was now enough that it was going to incur significant cost, so I set about investigating the cause.

I downloaded some firewall logs for the previous few days from the server and imported them into Vantage. The first place I looked was in an analysis at the “Source Address” summary, to see where the activity was coming from. What I found was a single host with a disproportionately larger amount of transferred data than the other addresses listed, so I drilled down to the “Destination Port” summary for this source address to see what services it was accessing. I found that all the traffic was going to port 53 – my DNS. More accurately, the large amount of data was going from my DNS to the source address. Drilling down to the “Individual records” view then showed that my server was providing a large response to a small DNS request from that source address – about 20 times per second.

Curious about why this single machine somewhere on the Internet was bombarding my server with small DNS requests at such a high rate, I set my server’s firewall to deny packets from that address and began searching around online for any information.

I quickly found out that I hadn’t configured my DNS properly, and it was set to allow recursive requests, meaning that if a request came in for a domain my server wasn’t authoritative for, it would then forward the request to another DNS that could answer, or given a blank request it would respond with the full list of root servers. Running tcpdump on the VPS revealed that every request coming in was blank, and my server was responding with the full list of root servers for each request.

It still seemed odd that a server would be constantly sending small requests to my server and receiving large responses. Then it dawned on me; I was looking at a Distributed Reflected Denial of Service (DRDoS) attack. The source address in all the requests I had looked at was forged by the attackers, so that my server – and many other servers out there also receiving the requests – would send their responses to the forged source address in an attempt to flood its connection. The source address in my firewall logs was the target of the attack. I found more information about this specific type of attack here.

Having disabled recursion on my DNS, my server’s contribution to the attack was significantly reduced. However, my server was now responding with a much smaller “request denied” packet for each incoming request. I wanted some way of preventing my DNS from responding at all, so again I headed out to the Internet to see what I could find.

I discovered a package called “fail2ban”, which dynamically updates your firewall rules to block addresses that are abusing your server’s services. I installed it using this guide, and immediately my bandwidth usage dropped off as it blocked further DNS requests. Even now the requests are still flooding in, but now my VPS contributes only a handful of packets towards the attack instead of the previous millions per day.

As the VM starts, it will present a boot prompt. At this prompt, type

boot> boot -c

This will boot you into  “User Kernel Config”. Once in there, type the following commands;

UKC> disable uhci
UKC> quit

This will exit User Kernel Config, and continue a normal boot. The “nvram: invalid checksum” error still appears, but the OpenBSD VM should now boot correctly. However, it will fail again on next reboot of the VM, so to make the change permanent, log in to the VM as root and enter the following commands;

# config -e -f /bsd
ukc> disable uhci
ukc> quit

Now the change is permanent, and the OpenBSD VM will boot correctly in future.

I posted this because I was unable to find any English information on how to fix this problem. I got the solution from an article written in Japanese.

An alternative, if not more sensible, representation of subnets is the slash notation, or “/n” where ‘n’ is the ‘mask length’ – the number of bits that must match between two IP addresses to make them part of the same subnet.

Consider the following;

     IP 192.168.0.1
          (binary: 11000000 10101000 00000000 00000001)

     Subnet 255.255.255.0
          (binary: 11111111 11111111 11111111 00000000)

This would be the same as 192.168.0.1/24, as 255.255.255.0 has the first 24 bits turned on, meaning anything that starts with 192.168.0.x will match it.

A /16 is equivalent to 255.255.0.0, as only the first 16 bits are on. 192.168.0.1/16 would match anything between 192.168.0.0 and 192.168.255.255.

It gets a bit messy with mask lengths that aren’t multiples of 8, for example 10.0.0.1/12 (/12 is equivalent to 255.240.0.0) would match anything between 10.0.0.0 and 10.15.255.255, and 10.16.0.1/12 would match anything between 10.16.0.0 and 10.31.255.255. This is because the addresses must match all 8 bits of the first segment (10), and only the most significant 4 bits of the second segment (16), making for a total match of 12 bits, or a /12 subnet. For example;

     IP 10.0.0.1
          (00001010 00000000 00000000 00000001)

     IP 10.16.0.1
          (00001010 00010000 00000000 00000001)

     Subnet 255.240.0.0 (/12)
          (11111111 11110000 00000000 00000000)

In effect, the shown bits must be the same for the /12 subnet to match an IP in that subnet range;

     IP/Subnet 10.16.0.0/12
          (00001010 0001xxxx xxxxxxxx xxxxxxxx)

     IP 10.16.1.5 matches 10.16.0.0/12
          (00001010 00010000 00000001 00000101)

     IP 10.19.3.1 matches 10.16.0.0/12
          (00001010 00010011 00000011 00000001)

     IP 10.31.1.1 matches 10.16.0.0/12
          (00001010 00011111 00000001 00000001)

     IP 10.32.1.1 does not match 10.16.0.0/12
          (00001010 00100000 00000001 00000001)

     IP 10.0.1.1 does not match 10.16.0.0/12
          (00001010 00000000 00000001 00000001)

WebSpy Vantage uses the slash notation to represent subnet values in Subnet Aliases because it is easier to understand and represent. The slash notation is also used to represent IPv6 subnets, where there is no suitable analogue to the #.#.#.# representation of IPv4 subnets.

I found a subnet calculator that converts between the slash notation and the #.#.#.# representation at http://www.nettoolsonline.com/slash_conversion.php, which comes in handy if you don’t want to have to do the maths yourself.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
	<appSettings>
		<add key="synopsisCallStack" value="true" />
		<add key="traceLog" value="true" />
	</appSettings>
	<system.diagnostics>
		<switches>
			<add name="LogOutput" value="4" />
		</switches>
	</system.diagnostics>
</configuration>

After creating the file, restart Vantage. A trace log will be created in your local application data folder;

XP; C:\Documents and Settings\%username%\Local Settings\Application Data\WebSpy\Vantage (edition)\(build)\trace.log

Vista; C:\Users\%username%\AppData\Local\WebSpy\Vantage (edition)\(build)\trace.log

The trace log shows the list of steps performed, and the details of any errors encountered. This information is useful to WebSpy when diagnosing errors reported by users.