BlueWolf's Howl

« Strange Problem | Bluewolf's Howl | Word Shortage »

March 02, 2008

Upgrades

Well, the worst of the upgrade is finally over. I was tasked to move servers off of the old Nortel switches and move them to Cisco switches. Sounds simple, eh? Yeah, until you involve the business aspect of it.

There's some oddness about legacy systems. They've been there for a long time. That means they probably grew up in a patchwork manner without a clear focus toward any particular design. Of course, since it just mutated from some little thing to the monster it has become... nobody really knows what's on the system. The switch people (well, me) don't know what servers are on the switch. The cable labels can't be trusted since some cables were 'reused', you can't tell which connection is an idle backup connection and which is not in use, and there are cables everywhere running like dreadlocks throughout the switch interface. Once you fight through all those obstacles -- you let the server people know which servers will be affected. Great - however, the server people don't really know what applications are on each server. Some of the apps are authorized and documented, but many are not. Especially the really old ones that were placed before a documentation system was put in place. You know - the apps that everyone uses *all the time* without thinking (or knowing where it came from).

So you prep and prep from your 'network' point of view. New switch is ready. Cables are in place. All you have to do is switch out the cables at the patch panel and the servers are moved. Well, that's only part of it. The actual swapping of the cables and moving the gateway for that VLAN will take only a short time. VERIFICATION of the servers and apps will take up the majority of your maintenance window.

The first 'lift and shift' move took over 4 hours for the server team to verify their servers. I had planned and put in for a 4 hour window. [The actual move of the cables took less than 20 minutes.] I can't prove it, but I think it took so long because they were using the server list that I provided to them to verify connectivity. Two people were manually pinging each server (and probably not starting at opposite ends). I got a bit more slick for the next move and was able to cut the verification time down considerably. Although I provided the server teams with server names and IP addresses, I let them know that I would personally ping all those servers. I wrote a small batch file which did a ping on each address and returned the results to a text file. It was not the fastest, best or most efficient script, but it got the job done. I ran the script before starting and then after the cables were moved. Once I resolved all those, I checked the speed and duplex on the switch connections (a simple sh int status) against what I knew them to be in the old switch. At that point, I knew I had done all that I could do on my own. Sure, it really was the server team's responsibility to ping the servers. Sure, they should have been able to come up with such a simple batch file. But they didn't - and wouldn't if just given a list of servers and IP addresses. Once I let them know that connectivity will already be tested and that they are responsible for 'everything else' - they magically found other ways to show they were doing something. Suddenly SMS reports sprouted up and services that were historically problematic were checked. I guess if you take away the simplest and easiest thing to do, they'll look for something to show they checked the servers...

So now I've moved 4 subnets and decommissioned 3 switches. There are two more to do with some kind of HA or load balancing on them. They might be tricky, but there are only a handful of connections. Then there are two other switches that are all 'monitoring' connections. Yeah, I just found out about those. I still don't have a switch to move those connections to yet, so I guess that's off in the distant future. [Read: ignored by management until someone decides it's an immediate emergency.]

Wonder what happens after you complete all the 12 tasks of Hercules that they put upon you when you first arrive? Can I outpace their buying so that I can get some easy days at work? I guess I'm 4 switches away from finding out.

Posted by BlueWolf on March 2, 2008 12:13 PM