I just finished the first DVD. It took hours, but the material was good and was presented in an interesting manner. We all have to admit - this is dry stuff. You're taking captures of network traffic and then looking at them under a microscope to figure out which one of the million possible things could be pissing on your connectivity parade. Okay, so there's really no microscope. That's an analogy. But when you look that closely, it can be tedious and the material can get very detailed. These videos help to keep you interested and capture your attention. And it's pretty much like actually being in a classroom course.

The first video basically focuses on how to use Wireshark. I had a little head start in that area, having already read the Ethereal book. And the Wireshark interface/GUI is somewhat intuitive. Basically, your average network admin can stumble their way through it and find the obvious problems. But, naturally, that's not good enough for me. I want to be able to go a step beyond that. These videos definitely help in that area.

Just within the first DVD, I've already learned how to do some customizations. These are the little things that will make an average person look more polished. And it will definitely help the captures and analysis go much faster. On top of that, Laura Chappell also lends us some of her tricks and tips. She shows us not only how to use this, but how *she* uses it. The section on graphing the data and displaying it professionally will definitely 'up' your game. You will not only know what you're doing, but you'll also *look like* you know what you're doing. Professionalism usually lends points to credibility.

I'm up to the second chapter of the second DVD (went through that tonight). The second DVD is exploring the protocols in depth and relating it to what you see in a capture. Yeah, you probably already know how DNS works. But don't skip that part. You've probably learned enough to fill a book about DNS. This focuses that knowledge so that you look at the key areas and it relates specific parts of the packets to the tools that will help you look at the protocol from a trace analysis viewpoint (versus your install, configure, maintain viewpoint). It's a little less overwhelming that way. You don't have to look at every line of every packet in a 546,000 packet capture. She shows you how to build filters to make the capture 'speak' to you and tell you its story.

This is some good stuff. And it comes from a name that should already be familiar. If you've read the ICRC or ACRC or CIT books, you've seen the name before. She knows her stuff - and knows how to teach it too. That's something that isn't found very often in this field. The 'experts' put you in a coma with their stuff. The good teachers often don't give you enough 'meat' in their classes. This set of videos has both. Get them if you can.

One of the server guys wanted me to change the port settings from 'hard-coded' at 100 Mbps / Full Duplex to Auto Detect (both speed and duplex). I asked him for the MAC or IP addresses of the connections so that I could find the ports to change. Since the device he was working with was a little different from your average server (they are on the SAN management connections), he was having quite the time coming up with that info. Finally, he came by my cubie. He told me that all my worries were over and that he knew for sure what the ports were. He was positive of it because he had asked ____ (insert smart guy's name here). Then he confidently told me that it was Port 80 and Port 443.

Yeeeeeeaaaaaaaah. I'm sure this stuff does run over ports 80 and 443. So we took a walk down the hall and I explained the difference between a logical port and a physical port. I explained to him what actually comes 'over the wire' to the switch... what I can see and not see. Yes, I can always see the MAC address. That's why we always ask you for it - that's the one way we can be sure that we're working on the same exact connection. If we only go by server name (and whatever resolves in DNS), we might be tweaking the wrong connection - since most of the servers have several connections to different networks. We're not being lazy and not looking it up for you. We're just trying to make sure that we can compare our information to yours and verify it.

So he asks me "Can't you just do a trace?" Well...that's another word that had to be reused. Did he mean a 'trace' as in pulling out WireShark and capturing several hundred MB of data to see something? Or did he mean the results of the 'tracert' or 'traceroute' command? He meant 'tracert' -- since he didn't even know about the other kind of trace. Again, I had to point out that a 'trace' will only show the path (of routers/layer 3) to the device ... as resolved by DNS. This will not show you the connections - just ones that resolve to that device name. In this case, it's not giving you any more information than a ping.

And this got me thinking of all the reused words in our field. When the Sun admins ask for a 'trunk' ... they're not passing several VLANs over it. They just want an EtherChannel bundle (using LACP). So why are both things called a trunk? Didn't they know this would be confusing? Why are both things called ports? Can't you call one of them something else? Did they isolate the creators of TCP/IP on a Survivor island? Did the Sun company not know about 802.1q when they created the ability to bundle their connections?

Of course they knew about the other definition of their word. But they used it anyway. So there you have it - proof that they didn't have enough words. And they used the verbal equivalent of PAT. One word - multiple translated meanings. Maybe someone will come up with Nomenclature V6 or something. That way we can have globally unique names for these technologies. Nah, on second thought... let's keep it confusing. I got a pretty good giggle out of the port thing today.

There's some oddness about legacy systems. They've been there for a long time. That means they probably grew up in a patchwork manner without a clear focus toward any particular design. Of course, since it just mutated from some little thing to the monster it has become... nobody really knows what's on the system. The switch people (well, me) don't know what servers are on the switch. The cable labels can't be trusted since some cables were 'reused', you can't tell which connection is an idle backup connection and which is not in use, and there are cables everywhere running like dreadlocks throughout the switch interface. Once you fight through all those obstacles -- you let the server people know which servers will be affected. Great - however, the server people don't really know what applications are on each server. Some of the apps are authorized and documented, but many are not. Especially the really old ones that were placed before a documentation system was put in place. You know - the apps that everyone uses *all the time* without thinking (or knowing where it came from).

So you prep and prep from your 'network' point of view. New switch is ready. Cables are in place. All you have to do is switch out the cables at the patch panel and the servers are moved. Well, that's only part of it. The actual swapping of the cables and moving the gateway for that VLAN will take only a short time. VERIFICATION of the servers and apps will take up the majority of your maintenance window.

The first 'lift and shift' move took over 4 hours for the server team to verify their servers. I had planned and put in for a 4 hour window. [The actual move of the cables took less than 20 minutes.] I can't prove it, but I think it took so long because they were using the server list that I provided to them to verify connectivity. Two people were manually pinging each server (and probably not starting at opposite ends). I got a bit more slick for the next move and was able to cut the verification time down considerably. Although I provided the server teams with server names and IP addresses, I let them know that I would personally ping all those servers. I wrote a small batch file which did a ping on each address and returned the results to a text file. It was not the fastest, best or most efficient script, but it got the job done. I ran the script before starting and then after the cables were moved. Once I resolved all those, I checked the speed and duplex on the switch connections (a simple sh int status) against what I knew them to be in the old switch. At that point, I knew I had done all that I could do on my own. Sure, it really was the server team's responsibility to ping the servers. Sure, they should have been able to come up with such a simple batch file. But they didn't - and wouldn't if just given a list of servers and IP addresses. Once I let them know that connectivity will already be tested and that they are responsible for 'everything else' - they magically found other ways to show they were doing something. Suddenly SMS reports sprouted up and services that were historically problematic were checked. I guess if you take away the simplest and easiest thing to do, they'll look for something to show they checked the servers...

So now I've moved 4 subnets and decommissioned 3 switches. There are two more to do with some kind of HA or load balancing on them. They might be tricky, but there are only a handful of connections. Then there are two other switches that are all 'monitoring' connections. Yeah, I just found out about those. I still don't have a switch to move those connections to yet, so I guess that's off in the distant future. [Read: ignored by management until someone decides it's an immediate emergency.]

Wonder what happens after you complete all the 12 tasks of Hercules that they put upon you when you first arrive? Can I outpace their buying so that I can get some easy days at work? I guess I'm 4 switches away from finding out.

Since I use my computer quite a bit, I really didn't have the time to properly troubleshoot the issue. I worked around it and figured I would get to it when I got a chance. So one night last week, I took the time and thoroughly updated SpyBot and ran a check on my system. It turned up nothing. I took another night and thoroughly updated Symantec and ran a scan. It took hours - but turned up nothing. It was quite frustrating.

This weekend I got a bigger block of time to work on it. I had resigned myself to having to format the entire hard drive and reinstall everything. Luckily I developed the habit of keeping the OS on one drive and the data on another. It would be long and tedious, but I didn't have to worry about losing any data. Friday night I started to make my list. I was checking everything that was on the computer so that I could duplicate it and not miss any apps. One thing I find annoying is having to do a task and needing to re-install the app first. As I was gathering the information I ended up eventually on the screen saver settings. == I had selected the CCIE study screen saver == How nice, but I never remember seeing any test questions when the screen saver comes on. Oh yeah, I haven't seen the screen saver come on - ever. I changed it and no more screen freezes or app errors. Lesson learned.

The time that I saved helped to move a few more things on my 'to do' list along rather nicely. I finally yanked the SPARC10 out of the closet and tried to get that working. That's going to take a bit of troubleshooting, but having it out is a step in the right direction. I know it works - and it has a fresh install of Solaris (can't remember what version) that I put on it last time I played with it. I'll get that rolling eventually. So then I turned my attention to the Linux box. Since now I have a router connecting to the DSL, I don't have to fiddle with getting PPoE running on it. I had previously installed RH7 and was going to update it. Yeah...that's an old version. Either the pages that *used to* work moved or were removed. Then the lightbulb went on: I have a better computer - I can install Fedora. Usually I put Linux on my 'spare' computer. Usually it's a computer that has very little cpu or memory. Quite often I've seen some of the older comps choke when trying to load Fedora, so I use an older version. Then I run out of time and never get to update. This time I got lucky. Everyone is happily computing on the best and fastest systems possible. And that leaves me with a PIII with 256 RAM. Wow. That's the fastest system I've ever had as a 'spare.' And instead of trying to shove a system on a 2 gig hard drive, right now I have two 8 gig hard drives. I went for the whole enchillada - complete install with all packages. And it's on the Internet. And it has two nic cards in it. I can finally use that computer to *do* something... [I have a 'project' that I've been wanting to work on for a long time now. Perhaps I can finally do it.]

Then I updated/finished the Sniffer laptop for work. It has:
Basic, but updated install of Win2K Pro (unneeded services set to manual)
Word and Excel (basic install - most options installed on first use or not used)
Wireshark
PuTTy
Telnet (the old one)
A small text comparison proggie that I picked up in my travels
Edit Pad

Quite the productive weekend. I'm really enjoying the Fedora computer. I'll have to put more about that one soon.

The latest realization of a wish granted came in the form of an email. It was a topic reply notification. I had always wanted to have a message board with a lot of users and a lot of chatter. I set one up about 3 to 4 years ago. It really didn't take off at all. So I forgot about it. Then I started another board. That one has a moderate amount of success and a smattering of chatter now and then. A handful of people visit regularly and play the games. Nice.

So this topic reply notification... Yeah, it was from the *old* board. The one I forgot about. WTF? So I went there today and found out my wish had been granted. Yes, it's a LIVELY board. Apparently a large number of people found it and saw it was abandoned. They gave themselves squatter's rights. The board had thousands and thousands of posts. There were tons of visitors. And it looked like some alt.* group from Usenet! Needless to say - I was mortified.

After I regained my senses and figured out the password to log in... I deleted the database for the board and removed the entire folder for the site. It's gone now. Wonder how many people were regulars and will be disappointed that it's gone? And I had no idea that this little smutty society sprung up all on its own - and thrived! Quite a lesson to be learned about monitoring the sites you run/own.

The other wish to come true came in a much more pleasant form. [They're not *all* tough lessons.] I had always wanted a laptop that was purposed for sniffing and other network troubleshooting. Luckily my supervisor gave me an old laptop to work with for just that purpose. Yipeeee! No more having to sacrifice my laptop while doing a trace. I can actually get some work done while the trace is running. I can actually pull all the 'other' stuff off and focus its every bit of resources towards the task at hand. Granted, it's an old stinkpad with only 512MB of RAM. But it will be streamlined and focused. That will make the difference - or so I hope.

I'm formatting the hard drive now. Once I get it set up, I'll probably post the specs here. Although the laptop came with XP Pro, I'm putting 2K Pro on it. I've always found it to be a tighter and cleaner OS than XP Pro. I'm going to strip it down and customize it. The challenge will be to only keep that which is necessary without throwing the baby out with the bathwater. Of course, I'll be putting Wireshark on it. I debated Word and Excel and figured I might really want those on it too. There might be a need for a csv file or two at some point and screen captures just don't work in Notepad. The OLD telnet is a must along with some type of term program - but I'm still debating which one (or two). TFTP is another 'must-have' along with WS_FTP or some ftp program. Perhaps I'll splurge and stick an IP subnet calculator on there. The more fluff I yank out, the more justified I'll feel in putting a small proggie of convenience on it.

I have to go that route because I'm going to have to share the laptop with others at work. But once that challenge is over, I would like to repeat the same thing on one of my old laptops so that I can keep it for myself. The greater challenge is going to be repeating it using Linux. I could probably eeek out a bit more power by using the command line interface and skipping X Windows. Now that would be a challenge. We shall see.

I'm still getting that wish. Now I wish it would stop for a while and give me a breather.

It started when the client asked for a network trace to figure out some issue with a file server. It was real obvious that the problem was connectivity between our site and the remote site. But they wanted to make sure and get a trace at the server. The 'network sniffer' that *used to* be there was there no longer. I got the cavalier 'put Wireshark on a laptop' answer when asked for tools. So I did. But I read Ethereal - this is a bit more advanced. So I whinned until they bought the Wireshark University CDs. I made it through part of the first CD.

No sooner was it in my hands then someone *else* needed to do a trace. The CDs were handed over (reluctantly). Then they had to be handed back. I'd really like to get through those -- however I'm so busy doing the dang traces and analyzing them, that I haven't had time to finish watching them. [They're really good, btw.]

So here I am, muddling through - learning by the seat of my pants. Again. I'm fighting my way through capturing a good trace. Apparently Wireshark gets lonely when you leave it running and spits out an error and stops. However, if you're sitting there watching it, more often than not, it won't error. Yeah, I've seen a lot of "Out Of Memory" error screens.

To save you some time - there's really no rhyme nor reason for the errors. Well, none that you can anticipate. I had someone (the guy I had to surrender the CDs to grrrr) tell me that I was taking too large of a capture file. The file for this particular trace was much smaller than traces I took previously. Sometimes (not all the time, but sometimes) Wireshark doesn't like to roll over a log. We set the file size to 50MB and it was choking - having to truncate and start new files waaaaay too often. So I set the file size to 500MB and got the capture. Navigating around in the resultant file wasn't anything resembling quick, though. Sometimes I can capture and watch it as it's going -- as long as I turn off the name resolution. Sometimes I can't and just have to wait for the capture to finish. Mostly I find that there's so much hand-holding necessary while the capture is running (for the requestor of the capture), that I really don't have time to sit and watch it. At the same time, they want some kind of generalized 'eyeball' of the results as the capture is still running. Yeah, not immediately after it's done -- AS IT'S RUNNING.... So it's helpful if you can view the capture in real time.

File size and the ability to open the file have nothing in common that I can tell. I've seen huge files open without a problem (over 700MB) and small files (about 50MB) choke on opening. [From what I saw in my travels - it may have more to do with the amount and type of metadata collected rather than total file size.]

Here is where I met my good friend Editcap. It's a command-line utility that will chop up your trace file into smaller files that are easier to manage (or open). I figured out how to use it by necessity. I was getting the Out Of Memory error. I doubled the RAM in the laptop. I increased the page file size. I still had problems. So I forced myself to figure out editcap and chopped up my captures.

When you chop up your captures, sometimes you can end up with a TON of files. The six large files (for about an hour's worth of traffic) ended up as 109 little files. Yes, and each one opened easily. However, now I had to open each one! A lot of packets is a lot of packets no matter how you slice them. [I put them in 50,000 packet files.]

Now my next task is to figure out how to merge the files and pull out some information. I know there's a command line utility to do this. I know there's a way to perform a network baseline (from a large amount of data). I saw it in passing while I was learning how to chop up my file. When I get a chance (read: when it becomes necessary due to a request), I will probably have to learn that on the fly too. Good thing I got those training CDs! Now if I can get a chance to view them -before- I have to accomplish those tasks, it would be a really nice thing.

Oh...and if you're using Secure Remote on the computer used to capture the file -- don't. You have to not only turn off Secure Remote, but also uncheck the box in the protocol properties. Otherwise, you'll end up with only one side of every conversation. Oh, you'll get traffic in both directions, but only one side of each conversation. Yeah, it kind of *looks* like you got stuff going in both directions, but when you try to follow the TCP stream... oooops. Yeah. Appearances can be deceiving.

Other helpful tidbits:
When you put a connection on a switch to mirror the port, make sure the speed and duplex match. Don't cause more errors from tapping the connection. You'll surely find plenty of retransmissions to blame your problem on, however none of them will be the original cause of the issue.

Don't let the large amounts of any bunch of packets fool you. Sure you can sort for broadcasts and multicasts and see a bunch of them. But how many are there in relation to the total number of packets in the capture? Perspective is everything.

Look at the traffic in several different ways. If you're doing a trace, this is probably a tricky problem that doesn't have an obvious solution. Network traffic is bursty by nature. Your average over a long period of time may hide a five minute burst that pegs out the bandwidth. [This will be most noticable if you're watching the capture as it happens.]

Coordinate with the users experiencing the problem. You don't want to waste your time capturing data when the trouble traffic is not on the wire. As you capture, have them do whatever it is that they do when they see the symptoms. Watch in real-time as they attempt to re-create the problem. You may notice something as it happens that would be harder to detect after the fact.

Learn all you can about the protocol that you're trying to analyze. It will be hard to know what's amiss if you don't know what you should expect to happen. You may have to do some research (maybe even fast and dirty on the fly in many cases), but try to find out as much as you can beforehand. I was surprised to find out that SQL traffic isn't going to be listed as SQL. It's known as TDS (Tabular Data Stream) traffic. This was very helpful to know and I was lucky to have stumbled across it during my research pre-capture.

Yeah - sometimes I just get lucky like that. I spent the first 6 to 9 months here without any requests for network traces. Then I got 7 in a two month time period. And it seems like that's their new solution to everything - do a trace. Not a one of them has any clue as to how much information that gathers or how much time it takes to sift through all this info. And I don't have a clue as to how much I'm learning from all this. Perhaps eventually I'll get that wish.

And so it is with the newest debacle with which I've gotten entwined. In the early morning hours, for whatever reason, 10 servers all lost connectivity with the network. There are 150 to 200 servers on the same VLAN. All are connected to the same switch. The only servers that lost connectivity were these 10 Sun servers. Other Sun servers - in the same VLAN, on the same subnet, on the same switch - did not show any interruption. Not one of the many Windows Servers - in the same VLAN, on the same subnet, on the same switch - did not show any interruption. There were no entries in the switch logs regarding any interruption or issues. Yet, this is still a "network" issue. [Read: I have to figure out what the damn problem is in order to get out of the hot seat.]

Here is why they don't believe it's not a network issue:
1) They are on a switch that has been historically unreliable.
2) More than one server was affected.
3) They imagine that perhaps there was a 'network' issue, but only these 'special' servers are sensitive enough to detect the problem.
4) A high-visibility app was affected. [This is the *real* reason they need a neck to choke.]

The commonality between the affected servers -- they were all running IPMP and the non-affected servers were not.

Now, often people think I'm odd because I have a thing for Operating Systems. I can configure, install, and maintain any Win-based server -- after years of experience and lots of study. See the Cert Battle Stats. But, I'm not satisfied there. I also have been playing/dabbling with Linux. And I have an old Sun Sparc that's running Solaris. Yeah, I could take another job as a Windows Admin. No, I'm not proficient enough to take a job as a Unix Admin. But, I really like to know these systems and how they work. As you can see - even though I work with routers and switches, it's still something I need to know for my job. You can't rely on the 'other guy' being as sharp (or as motivated) as yourself. And if they're blaming the network, why should the server guy waste cycles on investigating any further?

So I found a white paper on the Sun site which fully explains IPMP. Apparently, the way the cables know they're still good is that they send a multicast packet (224.0.0.1) and the first few devices to respond end up being the 'target host' for probe-based failover. Okay...so say host 10.4.3.15 responds quickest because it's on a powerful server? And what if all the IPMP connections are pinging that server to prove to themselves that they're alive? And what happens if that server just happens to go down? Well...if the target host fails, all the connections using that target host are going to falsely believe that they're down. Boom. It makes sense.

So I run this by the Sun guys. Apparently, they're under the understanding that the connections use 'the router address' for their probe-based failover. However, what they believe to be a 'router' address - it's actually the IP address attached to the VLAN. Yes, it's like a loopback address - only exists in SOFTWARE. So if (and nobody's checked it yet) they were using that address, there is no way it could not reach it. Perhaps the router was overwhelmed. Ah...no. It's not hardware - it can't be overwhelmed. Either they're using some other host address or the servers themselves were too busy. If the whole VLAN was inaccessible, other devices would have noticed. More servers would have lost connectivity and there would be an error message in the log.

The Sun guy is going to check it out. At least one person realizes that there's a possibility (no matter how slim a chance he thinks it may be) that it might actually be a server problem. One down. Oh so very many to go...

Ever have some 3750s that you want to stack? Great. There are quite a few pages online (and at Cisco) that you can read to know all about stacking your switches.

But what if you have some switches that are already stacked and you want to _un-stack_ them? Yeah. Can't find anything can you? I found a PDF on the Cisco site for Managing Stack Switches, but the only pertinent point in the chapter was that when you un-stack live switches, they both end up with the same configuration and could cause a duplicate IP address conflict. Other than that, I couldn't find anything.

I had a supervisor that once told me "if you can't find anything on it, then it must be so simple that there doesn't *need* to be anything on it." Although I've mostly found that to be true, it was no comfort when staring down a maintenance window for a task I had never performed before.

Luckily, the switches that I had to un-stack were relatively new and not really "in production"... There were only 5 connections on the 4 switches (two stacks to un-stack). Here's how I went about it:

1. Back up the configs (for any work you do, this is a good idea).

2. Power down the switch with the least amount of connections. This allowed me to work on one stack at a time and focus my efforts. Remove the stack cables in the back.

3. See what you have in the live stack. Make sure it knows it's the stack master. Make sure that it's at the default priority (1). Make sure that it is switch number 1. [You may have to change the switch number. The above link *will* tell you how to do that.] Save the config (wr mem). Reload the switch.

Note: These switches take a very long time to come back up. The switch comes up - and tests the hardware of the ports. Then it has to wait while the stack (now a stack of one) elects the stack master. This is in addition to the image decompressing (which always takes a while anyway). As they say in Hitchhiker's Guide: Don't Panic. This will undoubtedly take longer than you want it to - but it's working as designed.

4. Check the config after the switch comes up. Make sure that the changes you think you made actually occurred. (Again, a good idea regardless of task). Once that is done, you have one good switch. But you still have to remove the missing member. To do this, you need to use the "no" form of the provision command (for a provisioned switch). [Yeah, I had to figure that one out on the fly...]

5. Remove the connections from the switch that is currently powered down. Yes, you will have to use the console, but removing all the connections will put the second switch in a "lab" environment. Someone has to be there to physically remove the stack cables, so there has to be someone who can console into it.

6. Once you *know* there is no way that switch can talk to anything (with its duplicate IP address), power on the switch. Again, this will take quite some time to boot up. Make the IP changes that you need to make. Make the stack number and stack master changes. Remove the other switch. Save the config and reboot.

7. Check the second switch. Check it thoroughly to make sure your changes were made. Once you are confident in your new config, connect the cables.

8. Check everything again.

9. Pat yourself on the back (or wipe the sweat from your forehead).

For planning purposes, you'll want to allow for at least one hour per stack. Once you do this a few times (but, who gets to do this more than once or twice?), you may be able to shorten the time. But, from what I've seen so far - this is not something that you'll do on a regular basis. YMMV.

Then they went to hokey exams that you could only register on one computer. Inconvenient, but still pretty easy to install and use.

Now they're through Boson. Yeah. So now, you buy a book and you *have to* create an account at Boson (who also tries to sell you -more- practice exams) and register and then download.

They obviously spent a LOT of time and effort just to insure that you can't sell the book afterwards or give it to someone and let them use the practice tests. Perhaps the next user could register the book by 'de-registering' your computer and then registering, but there's already a lot of hoops to jump through - why should there be more? And I haven't seen anything as far as # of questions or difficulty of questions that would make me think these tests are any *better* than the ones Cisco Press used to put out.

And after all the effort they put into protecting it -- GAWD help me if I find a bunch of typos or grammatical errors. I will let them know that perhaps they should redirect their efforts towards the quality realm if that should happen. One thing I can't stand about many of these computer-based tests is that some of the questions really don't make sense in English. And my grade is depending on these questions being comprehensible! Granted, you expect an abortion of a study guide if you download Joe Zuba's FREE Network Exam Guide. But when you're paying (especially when it's over $50) for an exam guide, practice test or the like, you *expect* to be able to read it. And at $125 to $300 for a computer-based exam (on which your certification or re-cert depends), you should be able to read and understand the questions. Even if you don't know the answer. You see instances here and there where you *know* someone copied and pasted something and just didn't copy the whole thing. A sentence ends at the end of a line without a period or completing a thought. Exhibits are referenced that don't appear. Something contradicts itself or isn't clear about what they're asking. These things happen in the exams. They shouldn't - but they do. And you have to score high enough to score around that. So to be able, ready and practiced enough to do so -- you need GOOD practice exams.

I'll let you know if it was worth the download and other rigamarole once I go through the whole exam a time or two. I'm hoping I will be pleasantly surprised. Boson used to have a really good reputation for running you through the paces on their Sim software to ready you for the CCNA exam. Pricey - but worth it in the long run.

I know I'm going to use every available Sim, Rack, Lab and device that I can get my hands on once I start the Lab study. But the written test -- how many is too many? [Before you start using them to procrastinate actually *taking* the exam...]

Off the blog and back onto the book!

It eliminated a lot of confusion very quickly. Server people think in server terms. However, no matter where the IP address is or isn't located on the virtual server chassis -- I can tell for sure which port is seeing that IP address. If I have 12 addresses and I'm seeing those all on one port, yet I'm seeing traffic on 6 ports - something's wrong. Yeah, those are the ONLY IP addresses configured for that network. Uh huh. So what magic traffic is travelling on those ports that do not have an IP address? Even if the chassis is load balancing on its own, if only ILO traffic is going to the ILO ports, then what traffic (and IP) is going to the other 4 ports? And it's going in both directions...

Oh...yeah.... I forgot. Yeah. Just as magically as the traffic flowed, someone magically remembered that about a month ago (when the problem started), he configured two servers to back up on that network. Oh yeah - here's two IP addresses on that subnet that I forgot about. Uh huh.

It makes quite a few lightbulbs go off in people's heads when they're looking at a diagram and seeing connections that they don't have an answer for...

"Well, nothing's supposed to be on that network." Fine. However, there IS traffic on that network. In both directions. How does that happen when you supposedly haven't configured anything for it? Suddenly there's a nagging suspicion that perhaps there's something missing ~

A picture is worth a thousand words.

They call that incidental learning. I call it learning whatever you can whenever you can at any given moment. It all comes to bear in some way - learning is never wasted. With that in mind, I don't go so hard on myself when my studies take a detour onto a momentarily interesting path. I've always found that I need to know whatever I'm reading for some reason that I can't even see yet. It always ends up being useful.

What I'm rationalizing here is my detour into reading the Microsoft Office Visio 2003 Inside Out book.

No, it's not directly related to the CCIE exam. Yes, I know I'm behind schedule to begin with... But for some reason I feel drawn to go through this book right now. As any Network Engineer knows, you have to be able to manipulate Visio diagrams. Most engineers I've met use someone else's old diagram as a starting point. Others know how to alter a diagram, but couldn't draw one if their paycheck depended on it. I want to not only know how to create my own diagrams, but also to do it well. No...better than "well"... I want to really have a solid grasp of the program and know how to customize it and do a trick or two that the "average Joe" doesn't know.

There's going to be a number of CCNPs (and CCIEs) out there. What I think will set one apart from the pack are the things that add polish to the job. The professional-looking diagrams, the expertise in packet sniffing, these are the things that will separate you from the guy (or gal) that just does their job.

The other side of this coin is that once you learn something thoroughly, you can manipulate it. You can make it work for you instead of IT being the work. A well-honed stencil is like a fine tool. You can snap off a snazzy diagram in minutes - with the right template. Compare that to someone slopping together something that 'works' but is just 'good enough.' You can't take that and use it in a briefing. Someone is going to have to take time to clean it up for you. If you can create a professional document from the start (and do it quickly), it saves a lot of time.

I want my documentation to have polish. And I don't want it to take all day to get it that way. I've spent (wasted) time in the past, fishing for the right shape to put in a diagram. Sometimes I knew the shape I wanted to use, but couldn't find it and had to use a shape I wasn't satisfied with instead. It got the job done, but wasn't something that I wanted to use in a portfolio. It wasn't something that I would consider a deliverable to be proud of... That's where I was. Now I need to find where I will be.

An anycast address represents a service rather than a device, and the same address can reside on one or more devices providing the same service. In Figure 2-3, some service is offered by three servers, all advertising the service at the IPv6 address 3ffe:205:1100::15. The router, receiving advertisements for the address, does not know that it is being advertised by three different devices; instead, the router assumes that it has three routes to the same destination and chooses the lowest-cost route. The advantage of anycast addresses is that a router always routes to the "closest" or "lowest-cost" server.

Isn't that special? Yeah... and dangerous. Think of how that can break -- and what would happen. A DHCP request hits the router and it gets sent to the rogue DHCP server that Mr. Wannabe set up for testing. No, take that one step further - that Dr. Evil set up to hack your network... Overwhelm the *real* server with unicast traffic and let *your* server take over dispensing that service to the network. Bwahahahaha...

I'm sure there must be more to it and there are safeguards in place. I hope. I have an entire book about IPv6 on my list to read, so perhaps that book will cover it more fully. But it seems kind of odd that the protocol would be created with such a trusting heart. DNS was built on the assumption that everyone would play nice - as was most of the early technologies. When the whole Internet was a bunch of college geeks making their computers talk to each other, all was fine. Everyone was trying to make it work. Now we have to make it work *despite* the people who want to break it. On purpose.

You may also want to note that the IPv6 addressing space originally included a Site-Local Unicast address which has now become deprecated. The IETF IPv6 Working Group determined that they introduced a number of difficulties and deprecated site-local addresses in RFC 3879. They were originally somewhat like the RFC 1918 addresses - and were squashed because of the difficulties already seen in that realm. The problems include private addresses being 'leaked' outside of their intended site and the increased complexity for applications and routers that would need to recognize and cope with site-local addresses.

So as the idea evolves, perhaps they may reconsider the anycast address the way they reconsidered the site-local unicast addresses? Since it will be some time before widespread implementation, it's anybody's guess. Or maybe there's more to it that I haven't read yet?

On that note - it's back to the books!

In the interim, I've changed jobs. In some ways, I'm a bit self-conscious about blogging while working in a corporate environment. I guess the more geeks you work with, the more likely your coworkers are to stumble across your blog and bust your nuggets about it.

Sooooo... I finally finished the next book in the 2.5 foot of reference materials for the CCIE written exam. Not so sure I'll break the 3K page barrier this year, but that's okay because it'll just boost next year's stats. Although it looks like I've done less reading this year, that's not true. What I've done less of is -completing- the books I start. It's more a lack of focus than a lack of reading. I'm mid-way through 4 books and doubt I'll finish them before year end. But it all builds on itself and eventually reaches critical mass and the stats jump.

I've been reading a LOT of switching books. Probably because my current job focuses mainly on Layer 2 rather than Layer 3 activities. And although I'm not yet a switching 'expert,' I have deepened my switching knowledge considerably. Unfortunately, it has come at the expense of my routing skills. So to stave off total extinction of routing abilites, my next set of books to read will be the Routing TCP/IP books [Vol 1 and 2 of the CCIE Professional Development Series].

I'd really like to be able to get into some of the network design books I have, but I do need to realize that any deviation from the recommended book list just pushes out the written exam that much further. I'm planning on taking it sometime in March. I wanted to take it by Christmas, but with the holidays just around the corner - nah, not happening.

So if I don't post again for a while... Happy Holidays! Keep your fingers crossed that Murphy is out of town for the season and nothing crashes. Maybe this year my New Year resolution will be to blog more. Maybe.

The article mentions two instances in which people were arrested for using their position to defraud Cisco Systems of more than $10 million.

Basically, what they were doing was abusing the RMA system. When a part "goes south" and you need to replace it, Cisco will send you a part and you're supposed to return the bad part to them. I've been through this process many times with many companies. You're -supposed to- send the defective part back rather quickly. If you don't do it immediately, you can easily lose track of it in a large company.

I've heard of smaller levels of such abuse, but these guys had it down to a science. "In one case, the complaint says, Cisco sent Kyereme a one-port optical card worth $260,000, and Kyereme sent back an eight-port adapter worth about $2,000."

I've heard a few people claim that they built labs off of the system. They RMA good parts and use the replacement in production and the removed part in the lab. It lasts for a while and then Cisco screams for the part back and they finally send it in and then request another. I guess if you're desperate for lab equipment, it might be worth the hassle...? But that just hurts all of us.

I'd hate to end up at a company where such abuse has been going on for a while. Perhaps Cisco might not send a needed part right away due to prior abuse. That would make a Network Engineer's job sooooooo much harder. And if you're doing that to provide a lab for yourself, you'll NEVER convince anyone in the company to ever put ANY money towards lab equipment, no matter how necessary it is (even if they REQUIRE it). Once you set the precedent at that company, any time you ask for money for lab equipment, they will always point towards the precedent and state that the previous occupant of that job didn't need any, so why should they spend -unneccessary- money? Can't you just do whatever magical-geeky thing that your predecessor did? And you sound pretty snippy when you explain that unlike your predecessor, you are not open to defrauding Cisco into providing a lab for you to test your configurations prior to deployment in a production environment. You never know how well-liked your predecessor was - and it would not bode well to speak ill of someone who 'made things happen' without asking for more money.

Not to mention the PERSONAL RISK that you are taking by performing such miracles of geeky magic. What judge is going to listen to you when you explain that you -had to- have this equipment for testing prior to deployment? It's going to be your own hide that hangs out to dry when they catch up to you. And management is going to quickly and easily deny any knowledge of the double-bind they put you in. It is all so very not worth it.

One thing that I've heard about (but not personally used) is that Cisco has a program for lab equipment. You can purchase equipment specifically for a lab at a greatly reduced price. The stipulation is that the equipment must not be used in a production environment - ever. [Yeah, like that's going to work well.] The first time something goes down and you need a part -immediately- the lab equipment is going to be eyeballed and you'll be on the hot seat for not using it. After all, it's there, it works, and we need to get back up and running NOW. Is the convenience and confidence that you can get from testing in a lab worth the pressure later?

A better option may be to convince them to rent rack space slots. You can get just about any configuration you need and buy slots of time. You can do this on a project-by-project basis and don't have to worry about pressure to use the equipment in an emergency. It may require a lot of research to find the right vendor and configurre the equipment to simulate your environment, but it would be well worth the time and effort. It would reduce YOUR risk.

One thing that I noticed is that the book seemed to be more oriented towards recapping the highlights of things that should already be known. There's no way that this book (or any other) will cover the information necessary for the exam. It seemed to be more of a 'review' type of material rather than providing the meat of the subject matter.

I haven't finished my self-generated notes for the General Networking Theory section of the exam blueprint so I'm going to re-read the Sybex CCIE Study Guide next. I read the book quite some time ago, but I think it would be helpful to re-read it and refresh my memory. I will be adding those numbers to my page counts and listing it again in the Books Read - since I AM reading it again. By that time, I should be done with the General Section in my notes and can move on to the LAN Switching portion.

It's a bit overwhelming to look at, but I have six books for that section. In the pile, I put the old CCNP Switching exam guide. I remember it had quite a bit of good information in it, so it might be worth another look-see. I think it spooked me to have only one LAN Switching book from the recommended reading list. So I added some more books from my personal library. I have a book from Prentice Hall on Gigabit Ethernet and another from Wiley on Fast Ethernet. I know I've had those books for a while, but they still might be useful. I also have two McGraw-Hill books - one on VLANs and another on Catalyst LAN Switching. Granted, these books are old, but Fast Ethernet is still Fast Ethernet. I'm sure there are a few white papers on the Cisco site that will caulk any gaps between what I already have and what will be presented on the exam.