BlueWolf's Howl

« Snow Day | Bluewolf's Howl | Another Shot »

February 21, 2006

Spurious Memory Access

I'm about halfway through my assignment at the NOC. I can't say I'll miss the ride... but I might miss being around people on a daily basis. [Ooops. Did I say that out loud?] And I am learning a lot of things that I probably wouldn't run into without this experience.

One of the cases I encountered was a spurious memory access problem.
Feb 12 20:43:09 EST: %ALIGN-3-SPURIOUS: Spurious memory access made at 0x619719E4 reading 0xABC
Feb 12 20:43:09 EST: %ALIGN-3-TRACE: -Traceback= 0x619719E4 0x61927618 0x61922B78 0x619230CC 0x6192B9AC 0x6196E5D8 0x619720D4 0x61973E90

For the more technical reader:
Troubleshooting Spurious Accesses, Alignment Errors, and Spurious Interrupts

The cool part of this is that it follows along with what I was reading in Inside Cisco IOS Software Architecture. This is one of the books in the 2 1/2 ft pile of CCIE books, so I started reading it. I thought it might give me some insight into the philosophy behind the syntax used in the IOS or cover the command structure. Instead, it deals with memory. IN DEPTH. I had no idea that you could learn about this level of detail regarding memory use.

When the router boots up, it takes the memory and carves it into areas. Each area is used for a specific purpose. And the book details how different routers use memory in ever-increasing ways. You learn about memory regions, heaps and buffers. It also covers the exact process involved in receiving a packet on an interface, looking up where it needs to go (from the routing table or cache) and moving it to the outgoing interface and sending it off. It details the packet switching process of the routers so finely, that you can clearly see why process-switched packets process much more slowly than cef, fast-switched, or optimum-switched packets. [With process-switching, once the packet is copied into the router, it has to be moved to a special area for processing (using a CPU interrupt) and then moved to the area to be sent out. The other switching methods can hold it in the incoming area, look up the cached switching information, and then move it to the outgoing area without bothering the CPU.]

Something else that I found noteworthy... When the router boots up, it takes all the memory available and carves it into sections. It creates buffers and memory pools. Some of these pools are created to accomodate different MTU sizes. And interface buffers are created for all the interfaces installed in the router. If all of the interfaces that you're using all use a 1500 MTU size, then only one pool is created to accomodate the 1500 MTU (and shared). If any of the interfaces (including the ones that are not used and/or administratively shut) have a different MTU configured, a memory pool is created for it -- and wasted. Likewise, an unused interface causes an interface buffer to be wasted just in case it might be enabled at some point.

I can see this playing out in a future troubleshooting scenario. You look at the router statistics and see a lot of packets dropped because the buffer is full. You look at the memory and it doesn't look full - or you see areas of the memory that are unused - so it must not be *really* full...right? There just must be some *other* problem... Yeah...a *memory* problem. Hopefully I remember this and don't have a spurious memory access problem of my own.

Posted by BlueWolf on February 21, 2006 08:34 AM