I'm enjoying the Cisco Internetworking & Troubleshooting book (the middle one in the sidebar). And I can already see the effects of my studies through my work.
No, I didn't do anything grand or spectacular. It was just a greater feeling of confidence in my abilities. And the actual technical stuff wasn't directly related to anything I had read. Somehow my studies had unconsciously affected a greater area than the subject matter. It was almost like I had somehow given myself permission to think.
I was on one of the easiest jobs - a 3rd party deal where I basically act as a 2,000 mile screwdriver. I'm the 'on-site' technician and the company providing the service (our customer) is servicing one of their contracts (their customer). The engineer providing service (our customer's engineer) is the one responsible for resolving the issue. I just do what they tell me to do. Easy, right? And the proposed work seemed easy enough too. There was a problem with a content caching engine. Another one had been pre-configured and shipped by the engineers of the main site. All I really needed to do was swap devices. In addition, another device (not configured) had been shipped - just in case. Often I claim that my job is 90% non-technical - it's true. I found the Data Center. Then I found the Shipping Department and found the device. I brought the device to the Data Center and installed it into the rack. I plugged in all the cables. Voila! Uh...nope. [Here comes that other 10%...] Their customer (who is also remote) still can't connect to it. Since the work was supposed to be plug-and-pray, our customer let me handle it on my own. So, here I am on the phone with their client and he's saying that it's still not working. Rut-roh.
I start off by doing whatever their customer wants me to do to troubleshoot. Power it off. Power it on. Connect the modem. Yadda, yadda. Still...nothing was working. I could hear the frustration in his voice and had heard that this was a long-standing problem that they were trying to fix. Suddenly I heard the voice of Jean Luc Picard saying, "Number One - Engage".... and my brain went into warp 7.
I automatically discounted the modem setup. There were too many things that could possibly be wrong to sit there and try to troubleshoot that rat's nest. Instead, I focused on the Ethernet connection. It's connected to the internal network. They *should* be able to ping that. [They can't.] The patch cord from the Ethernet interface went to a patch panel. From there, it went into the floor of the Data Center - so there's no telling where it finally ends up. But, I should be able to test that length of cable. I have a handy, dandy Ethernet interface on my laptop... So I connect the cable to my laptop and give the NIC the same IP address as the device. Can you ping me now? Good. [Well, that tested everything up to the device.] The other suggestion that I made was that perhaps since the device connected to a switch, maybe we should try a crossover cable...? Since I had never seen one before, I had no idea if the cache device was seen as a type of switch or type of server, and it was an easy test to try. The customer agreed that it was an idea worth trying, but wanted to try a few other things first. As he tried over and over to dial into the device using the modem and console cable, I could see that he wasn't getting anywhere. I then suggested that I should disconnect the console cable from the modem and plug it into my laptop, thereby eliminating some of the complexity. It would at least tell me if the device has booted up or perhaps is in ROMMON mode.
Maybe ROMMON was the magic word. I guess it struck him that if it was in ROMMON mode, there would be no getting to it from the ethernet interface (or dial-in) no matter what we did. I connected the console cable and opened Hyperterminal. OMG. It was still booting! I watched the image finish decompressing. I sat and watched the rest of the initialization process and reported the progress. Once it completely finished initializing, I had him try to ping the Ethernet interface. Ta-da! It works. Well, yeah, of course it works once you let it finish booting. For whatever reason, that device takes an extremely long time to initialize. Go figure.
Then I assisted with troubleshooting the out-of-band access. I reconnected the modem and console cable and had him try to dial into the device. Still a no-go. I told him which dip switches were down. We adjusted the dip switches and he tried again. Bingo! He was into the device. He logged in. Everything was fine. Then for security purposes, I carefully disconnected the phone line to the modem and placed the entire setup in the server rack. It would be there for any emergency that might arise and has been checked for proper operation. Another happy customer. Yeah.
Then I had to find my way back to the Shipping Department and send back the part that I replaced. And I found the other device (that we didn't need to use) and shipped that back too. Then another 15 minutes of calling our customer and reporting completion and the shipment tracking numbers, calling my office and reporting completion and the shipment tracking numbers, and yes, finally I was done.
It was only during the long ride home that I realized - hey, I actually handled that one on my own. It wasn't any spectacular feat of troubleshooting, but I really did do that myself. Then I started seeing small green symbols running down my dashboard. No, wait... that was my ride-home Matrix fantasy. Oh well, another day in the trenches. I made a customer happy. I gave myself a boost of confidence. And I learned something. Yes, the content caching engine is seen as a server. Making mental note in case I ever run across one again.
Posted by BlueWolf on January 9, 2006 11:14 PM