This article was originally posted on July 8th, 2013 to my old blog. I thought that it was still interesting and relevant enough to be recycled.

The Symptoms

So last week was a short one for me with the combination of the 4th of July holiday and some vacation time. Unfortunately, the last working day ended up being a rather long and frustrating day.

Around 9 am a ticket came in that one of the managers’ phones was dead. Keeping in mind that my phone system is an ancient Rolm/Siemens 9751 system, I started with the assumption of hardware failure. My first step was to try a known good phone on the line. Nope, still dead as a doornail. I also tried changing the line to a different port on the system fearing that the line card could be the problem, but that also failed to change anything.

By this time I was figuring on a cabling issue. Over the next 4 hours or so I toned and re-terminated the cabling from the phone to the PBX. Normally this wouldn’t take this long, but I kept getting the tone to the phone, but the phone still refused to work so I kept repeating and segmenting the process. Finally, I believed that I had isolated the problem as being between the last IDF and the phone jack itself.

Experience to the Rescue

At this point in the day, the manager had already left for the day so I had to get the plant engineering director to unlock the office for me. My plan was to re-terminate the phone jack hoping that there was a short in the wall jack. The plant engineering director was a bit intrigued by my day of troubleshooting so he stuck around to help me move the desk out of the way. As we were doing this, he started to trace where the line left the room and noticed that it when through a box on the wall with a switch. I flipped the switch and bam, the phone worked again. He explained that he recognized the switch as an old line switch (to switch a single line phone from one line to another) because his father used to work for the phone company.

So now, that others may learn from my day of troubleshooting, here is what the switch looked like. After I added some labels for the future. My lesson, expect the unexpected.

Phone Switch

Looking Back

Looking back at this incident almost six years later, it reminds me that sometimes previous knowledge can be the most important part of troubleshooting. How do you try to pass on these lessons so that your newer engineers don’t have to learn the lesson in the trench? Is there a moment like this in your troubleshooting past that you would like to share? Please let me know in the comments section.