It’s six o’clock on a Saturday, the regular crowd shuffles in, my phone starts ringing next to me, telling me the network is down. Or something like that.

Last night I received a call from our helpdesk that one of our departments was complaining about a server not being reachable after a set of power outages at the facility. Figuring that the server had not been powered back up, I called and talked with the department “tech”. I was wrong, it was powered up. My gut told me that there must be a small unmanaged switch in play somewhere between this server and the infrastructure switch. I could see one single port on the appropriate VLAN in a down state. With the onsite tech serving as my hands and eyes we traced the network cable in the back of the server to a wall plate and he told me that the NIC had link and activity lights.

Not wanting to make an hour trip to the facility and back, I called another engineer that was on call and lived near the facility. He agreed to drive in. Once he was onsite he called me and again the tracing began. This time I was told that the server had four NICs and that only three of them had cables attached. All three of them were going to separate unmanaged switches! My fellow engineer pointed out too that one of them was not showing a link or activity light. Figuring that this was our problem child, I had him follow it to the switch and asked him to power cycle the switch.

This is where it got comical, or sad, depending on your point of view. He found that the switch had no power. Tracing the power cable he found it was a USB cable and that it was plugged into a nearby desktop PC. Unfortunately, the PC had never been turned back on since the power went out. Once my colleague powered up the PC, everything was restored and we both got to resume our Saturday night.

So the moral of the story is to always respect layer one. No power, no blinking lights! Seriously though, whomever originally set this up should be forced to work a support line for new gadgets the week after Christmas.