I ran into a problem recently while trying to monitor several Cisco ASA 5505 firewalls. The main problem was that no matter the IOS version, I could only discover a single ASA 5505 within SCOM. I ran into this while importing many customer ASA’s that we manage. It seems that the ASA 5510 and up are fine. The only problem was the ASA 5505. So I dug a little deeper.
Here is the problem. There were absolutely no monitoring problems with SCOM 2007 R2 as it pertains to the Cisco ASA 5505. When we tried to discover multiple Cisco ASA 5505 firewalls, it would always overwrite a single discovery and usually the last one to be discovered was the one that you were allowed to monitor. The problem lies in how SCOM 2012 and the Smarts engine discover and store the device objects in the database.
With SCOM 2007R2, network devices used the IP address of the device as the key. This made sense, since every device would at least have an IP address. Well, of course we know, SCOM 2012 uses the MAC address as the key when it stores the object. This is a fundamental change (and one I do not like very much). I understand why this change was made, but I don’t like it.
You would think that every single device in the world has a unique MAC address and you are correct. That is the idea, anyway. If you look closer at the ASA 5505, you will see and internal interface that CIsco has added to the firewall. You cannot change the MAC address on it and it is not an interface that you can manipulate. It does, however, show up when you start to monitor it. That alone does not cause a problem.
I spent hours tracking this down. I used several ASA 5510’s and several ASA 5505’s. I tried older IOS versions and the most current IOS version and many in between. It was finally a relief when I could replicate the issue. This gave me common ground to compare. As I said before, this only shows up on the ASA 5505 and not any other hardware model, but across IOS versions. This told me that is was hardware related.
When digging deeper into the ASA 5505, I noticed that each device had the same internal interface…AND THE SAME MAC ADDRESS!!! Cisco assigned the same MAC address to the internal interface on all of their ASA 5505 models. This also is not normally a problem, but it is in this situation. The reason for the issue is that the MAC address is something like 00-00-00-30-00-02. OK so far, except this happens to be the lowest value MAC address on the device. Apparently SCOM 2012 collects all of the MAC addresses on the device, and then locates the lowest MAC address, and then assigns this as the DeviceKey to the device. I have seen technologies such as Spanning Tree and BGP select IDs based on highest or lowest, so this kind of thing sort of makes sense to me.
My next step; call Microsoft and report this to see if there is a fix. After at least a month and some escalation to the product team, it looks like there is an end in sight. It looks like the fix will be in UR4 which apparently comes out sometime in the next month or so. We will see if this fixes the issue. Right now, we are not able to fully monitor all of the ASA 5505’s that we manage. We do have a workaround, but it is short term.
In the end, I think this is Microsoft’s issue to resolve, even though it is Cisco’s fault this is an issue. The software should be smart enough to handle circumstances like this. What I can’t believe is that someone hasn’t reported this sooner.
We will see.