I updated the firmware on the WRT54G router today to the latest release of Sveasoft’s code. Dyndns updates are working again, after being broken in Alchemy 5.2.

I modified the firewall to block the XBox’s access to XBox Live (ports 88 and 3074), since I don’t want my son connecting to it by accident. I had blocked all access to the Internet from the XBox, but that also broke the weather lookup in XBMC, which is becoming handy as we move into fall.

Part of the process for filtering XBox Live involved defining new services through the web interface. I kept getting errors on the page when I tried to apply the updates. After a fair amount of searching, mainly because I didn’t make a good choice of keywords, I found the problem is a bug, or limitation, of IE. Apparently it can’t handle Java variables of 2.3K in size very well. Repeating the process in Firefox worked perfectly.

Despite my triumph earlier in the day in discovering a duplex mismatch that was overlooked, I also spent far to much time thinking about another problem in which I forgot my basic IP handshaking and routing fundamentals.

We’re migrating to a new ISP, so in a flash of brilliance I figured I’d setup another interface on the PIX to test the new connection. The connection appeared to be up, since I could see the flood of Windows zombies attempting to connect already. But I was unable to actually pass any data over the link.

After poking around the logs I determined that the firewall was not blocking anything, nor was iptables on the web server. I plugged in a hub between the web server any the firewall, and ran ethereal on my notebook to see what was happening. What I saw was the handshaking failing on the link. SYN packets would come in from my external host, and the server would send the SYN-ACK back. The external host would then reset the link with an RST.

I thought about it for a bit before realizing my downfall: the web server is NAT’d behind the firewall. The external host was sending to one address, but the firewall was sending the reply on a different address. The external host thinks “spoof” and resets the connection.

We have been receiving complaints of slow printing services on and off for the last couple of months at the centre, but had been unable to nail down any problems. Endless combing through documentation and reviews of the settings turned up nothing. Today I happened to recognized the problem while doing some other network configuration work.

It turns out the server that handles the print queues had its interface locked at 100Mbs full-duplex. The switch was set to auto-negotiate, resulting in it setting the connection at half-duplex. This is a recipe for exactly the kinds of problems we were experiencing - poor performance under load. Once the server was configured to auto-negotiate the link (since it will be moving to a new switch soon), printing performance drastically improved. Printing a PDF with graphics that took several minutes at times now prints in seconds.

This quote excerpt from Critical Networks Advanced Ethernet pages explains netter than I could how the duplex mismatch occurs:

So how does a duplex mis-match occur? The process below describes just one common scenario where one device is auto-negotiating but the other device is hard coded for full duplex:

1. The hard coded device is configured for 100baseTX full duplex (100FD) operation. It does not listen for FLPs sent by the other device, it simply waits for the correct idle signal before making the link active
2. The auto-negotiating device will notice the absence of FLPs and will see the 100baseTX idle signal using parallel detection. It will make the link active as 100baseTX half duplex (100HD).
3. Now link is active on both devices and data will be transmitted, but there will be performance problems
4. The hard coded 100FD device can send data at any time regardless of whether the 100HD device is transmitting. This causes the 100HD to detect both excess and late collisions and to abort the current frame in mid-transmission. This in turn causes the 100FD device to see many runts, alignment, CRC and FCS errors.

We forklift upgraded the two main wiring closets in the hospital on the weekend. As expected, there were some problems with workstations not connecting properly or having their connection speed throttled. At the end of the day yesterday a couple of techs finally resolved a problem with a station failing to connect to the network.

A few minutes later the network became completely unusable. Of course we were foolish and didn’t immediately go to the last change made, and spent the next three hours going through our normal diagnosis. Similar problems have occured in the past when worm-infected laptops were connected to the LAN. In the end, we went back to the prior changes, and disconnected the now-working node. Lo and behold, the problems went away immediately.

The experience was good for us. I recall one of Pournelle’s Laws is something to the effect of: always start with the cables, because you’ll end up there. But the time spent hunting wasn’t wasted because we discovered some other traffic to be inspected while running Ethereal.