An updated version for the ReSearch This management pack has been created for the R2 version of the management pack and has been updated to work for MOM 2005 as well. There had been issues with this management pack and how it integrated with the new systemcentercentral.com website which should be addressed with the updated version. This management pack provides an easy way to investigate alerts in Operations Manager (and System Center Essentials, and MOM) and search for resolutions which the community has found when the alert occurs. The updated version of this management pack is available at: http://www.systemcentercentral.com/PackCatalog/PackCatalogDetails/tabid/145/IndexID/21716/Default.aspx.

I am working on a version which will work on OpsMgr 2007 RTM – hopefully later in the week.

This also applies when focusing on an area of specialization. I am a Microsoft Operations Manager geek. I am at the point in my life where from a technical perspective I am living and breathing this particular technology on a daily basis. However, I am well aware that even within a specific technology there are people who truly alpha-geeks in an area within that technology. Within Operations Manager there are guys who know a lot more about management pack development, targeting, scripting in OpsMgr, ACS and other topics.

So what the heck does that have to do with community written books?

Over the last year a group of us who had written on System Center Operations Manager 2007 Unleashed (www.tinyurl.com/27mqnm) came to the conclusion that there was enough new stuff within Operations Manager 2007 R2 to justify writing a book on what is new in the R2 release. We originally were going to take our four core authors, map out what was new in R2 and divide and hopefully conquer. We didn’t think it made sense to revise the current book to cover the content because the original book was comprehensive for the original product so we decided to write a supplement to the original book which would cover R2.

So what the heck does that have to do with community written books?

I’m getting there I promise. As a group we decided to try something different on this book. Instead of just writing on what was new in R2 we decided to also delve into where the community had evolved the product since the original book had been written. To make this happen, we went into the community and sought out the best of the best in their specific areas of subject matter expertise. We brought in SME’s in targeting, management pack authoring, ACS and other areas. In total this book was directly written by almost a dozen different authors or contributors who physically spanned the globe. We also received technical input and reference technical blogs which also span the globe. What I eventually realized is what should be readily apparent by now – this book was written by the community for the community.

Technology adapts quickly, and people who focus on a specific aspect of technology can quickly become the go-to-SME in that technology. With blogging people are sharing new developments more quickly and others can build on those advancements. A community written book (shepherded by a lead author for coordination) may make a lot of sense as a model for the technical book industry.

Microsoft System Center Operations Manager 2007 R2 Unleashed is currently (and unofficially) estimated to be available in mid-March so about 40 days from now we’ll see what the feedback is on this concept. Regardless of how it goes, it’s been an incredible experience being part of this technical community and to have the opportunity to work with such incredibly talented people who are part of it.

Issue: The rule performs a DNS query of type “NS” (as provided in the Query Type parameter), which means the query is search for the name servers of the domain provided in the Host parameter. The problem here is that the domain name provided is “www.microsoft.com”. Since this is a host name rather than a domain, the query returns a referral rather than a list of DNS servers. This results in the error message referenced above. This was occurring in MP version number 6.0.6480.0.

Resolution: You can fix the error in one of two ways (pick one, not both):

• Set the Host parameter to “microsoft.com” (without the quotes). Then the query returns a list of DNS servers for the microsoft.com domain OR
• Set the Query Type parameter to “A”. Then the query returns the IP address(es) for www.microsoft.com

Issue: Agents were reporting error number 7002 (The Health Service could not log on the RunAs account for management group because it has not been granted the ‘Log On Locally’ right.

Agents were grey. There were errors reporting on the Health Service Unloading System Rule(s). The agents were in the root domain, the gateway action account was in the root domain.

Resolution: Added the GWAA account to the domain administrators in the root domain. We also ended up needing to uninstall and re-install the agents reporting the error after making this change.

1) Logged into the server via RDP.

2) To determine the settings for the agent we opened control panel, Add/Remove programs and started a change (not uninstall) of the agent. This provides a quick way to see what management group was specified during the installation.

3) When we found these we incorrect, we did the change in Add/Remove programs again and added the second management group name with the correct name.

4) Once that was done we re-opened Add/Remove programs, and removed the incorrect management group name from the agent.

5) To complete the process we restarted the System Center Management service on the agent.

However, this debug took a new direction when I looked in add/remove programs and saw that this system had both the Operations Manager RMS installed, and had the Operations Manager R2 Agent installed on it. We figured out that when the private bytes counter went too high, it restarted the "client" which is ALSO the Health service for the OpsMgr RMS – Ouch. I ended doing a workaround for this by removing one of the management groups which was assigned to this server (the RMS was actually reporting to three different management groups, but only one of them was really functional).

After I tracked this down and was preparing to write this up, I ran across Walter Eikenboom’s article which addresses this specific issue in his  “How to monitor another management groups OpsMgr infrastructure” article. Walter summarizes the issue I ran well as follows:

“When the monitoring opsmgr looks at the health service of the RMS or MS from another management group it thinks its just an agent and is not aware that’s a health service of another opsmgr infrastructure.”

So when a health service goes higher than the threshold for an agent, it automatically restarts the agent which in this case is the RMS’s Health Service. Which is what we were seeing here.

So, if you have both Forefront an an Operations Manager environment which is monitoring the Forefront RMS make sure that you check out Walter’s blog available at:
 

http://weblogwally.spaces.live.com/blog/cns!A913F865098E0556!1574.entry

Additional solid reading material related to how the health service restarts is available at:

http://blogs.technet.com/kevinholman/archive/2009/12/21/the-new-and-improved-guide-on-healthservice-restarts-aka-agents-bouncing-their-own-healthservice.aspx 

Can you just installed the OpsMgr R2 agent from the RMS if you don’t the server to be monitored by OpsMgr?

I also curious what would happen if you have an environment like this (RMS with agent installed) and you decided to uninstall the Operations Manager R2 agent… My uncertainty came from the fact that both the RMS server and the Agent both run the same service name (System Center Management in R2, OpsMgr Health Service in RTM/SP1). I was wondering if the uninstallation of the agent could result in the destruction of the RMS as an unfortunate side-effect. 

To test this configuration I took an RMS which I have in a lab environment and I installed the OpsMgr R2 agent on it. Next I uninstalled the OpsMgr R2 agent from the RMS, and it uninstalled without an issue and didn’t cause a problem for my RMS. [This was just my lab environment, don’t try this in production without a solid backup methodology in place just in the off-chance my lab didn’t reflect real-world production environments!]

• On decommissioned servers these alerts are not required and can be disabled as discussed here: http://www.systemcentercentral.com/BlogDetails/tabid/143/IndexID/25009/Default.aspx 
• On some servers (such as test servers) these alerts are not required and can be disabled as discussed here: http://www.systemcentercentral.com/BlogDetails/tabid/143/IndexID/25006/Default.aspx
• On production servers these alerts are not necessarily critical (see http://www.systemcentercentral.com/BlogDetails/tabid/143/IndexID/26595/Default.aspx for reference on defining what’s critical in an environment). For these I recommend the creation of overrides to set these to informational as discussed here: http://www.systemcentercentral.com/BlogDetails/tabid/143/IndexID/25007/Default.aspx 

To accomplish the requirement we plan to use a subscription to provide an email notification when the specific job fails to complete successfully – by limiting the subscription to this alert, and by limiting to only those which match a portion of the description field. As an example, we create a subscription for “A SQL job failed to complete successfully” where description contains “abcjob”. This approach allows us to categorize the alerts as informational (keeping the environment green and reducing email alert volume), but to still receive notifications for specific long running jobs which are identified as critical to the business.

1) Registry: Regedit to key HKLM\Software\Microsoft\Microsoft Operations Manager\3.0\Setup in the DatabaseServerName string. (Chapter 12, page 566 in System Center Operations Manager 2007 Unleashed book) on a management server.

2) Command prompt: From a command prompt type netstat and look for “Ms-sql-s”.

3) OpsMgr UI: Open the Monitoring node, Operations Manager folder, Management Group Diagram. Open the SystemCenterOperationsManager icon, highlight the RMS server underneath it, and at the bottom of the screen it shows the Operational Database Server Name and the Operational Database Name as well.

Thanks to Steve Ross, Kerrie Meyler, and Adam Rafels for their help gathering these up and hopefully I’ll be able to remember this next time I run into it as well!

Issue: We ran into an issue recently where we saw issues with most of our distributed applications changing to a grey state seemingly for no reason. After a little digging we determined that the RMS server itself was actually in a grey state. We tried things that have worked in the past like reviewing the Operations Manager logs on the RMS, restarting services on the RMS, rebooting the RMS and even went so far as to reboot pretty much all OpsMgr related servers just to be thorough. Still grey. We verified that database connectivity was functional, there was plenty of free space in the Operations Manager database and we found no indications of the root cause. Just grey, grey and more grey.

Resolution: I doubt if this is supported, but at least it allowed us to bring back up our Operations Manager environment. We stopped each of the three services on the RMS, renamed the %programfiles%\system center operations manager 2007\health service state\health service store\ to another file name, and then restarted the services. This re-created the health service store folder, and rebuilt the files within it. This is the second time I have seen this in one environment in less than a year.

Resolution: Check the free space on the OperationsManager database. There are alerts to provide warning if there is less than 40% available free disk space however if an OpsMgr environment is completely out of disk space all management packs will fail to import until there is sufficient free disk space. Best practice wuld be to increase it at least another 50% if possible since the database was  already out of space and OpsMgr wants 40% for indexing.

P.S. Thanks to Blake and Anders who originally tracked this down/I just re-posted this as I hadn’t seen it happen recently and didn’t remember the cause until I found their discussions on the topic.

P.P.S. Marnix already blogged on this a while ago as well! http://thoughtsonopsmgr.blogspot.com/2009/09/mps-wont-import-any-more-is-opsmgr.html

Select distinct

v_R_System_Valid.Netbios_Name0 AS "Netbios Name",

v_GS_INSTALLED_EXECUTABLE.ExecutableName0 AS "File Name",

v_GS_INSTALLED_EXECUTABLE.Product0 AS "File Description",

"File Version" = CASE when (v_GS_INSTALLED_EXECUTABLE.Product0 is NULL or v_GS_INSTALLED_EXECUTABLE.Product0  = '-1') then 'Unknown'

Else v_GS_INSTALLED_EXECUTABLE.Product0 End,

"File Size" = CASE when (v_GS_INSTALLED_EXECUTABLE.FileSize0 IS NULL or v_GS_INSTALLED_EXECUTABLE.FileSize0 = '-1') then 'Unknown'

Else v_GS_INSTALLED_EXECUTABLE.FileSize0 End,

"Modified Date" = CASE when (v_GS_INSTALLED_EXECUTABLE.TimeStamp is NULL ) then 'Unknown'

Else CAST(v_GS_INSTALLED_EXECUTABLE.TimeStamp as varchar)End,

"File Path" = CASE when (v_GS_INSTALLED_EXECUTABLE.InstalledFilePath0 IS NULL or v_GS_INSTALLED_EXECUTABLE. InstalledFilePath0= '-1') then 'Unknown'

Else v_GS_INSTALLED_EXECUTABLE. InstalledFilePath0 End

FROM v_GS_INSTALLED_EXECUTABLE

INNER JOIN v_R_System_Valid

on v_R_System_Valid.ResourceID = v_GS_INSTALLED_EXECUTABLE.ResourceID

INNER JOIN v_FullCollectionMembership

on v_R_System_Valid.ResourceID= v_FullCollectionMembership.ResourceID

Where Product0 like 'Adobe%'

and v_FullCollectionMembership.CollectionID  = @Collection

order by v_GS_INSTALLED_EXECUTABLE.Product0, 'File Version'

Big thanks to Weston and Julian for working with me on this!

Question: How do I determine what the processor thresholds are for a specific server in my OpsMgr environment?

Answer:

In the Operations Console under Monitoring / Microsoft Windows Server / Windows Server State / find the server(s) you are looking for.

Open health explorer / Entity Health / Performance / Operating System Performance Rollup / Performance / Total CPU Utilization Percentage (shown below) and right-click on the monitor and go to properties.

Some of the default thresholds are shown on the thresholds screen and are summarized below:

Alert on CPU utilization percentage of 95% over 3 samples.

And alert when we see the CPU Queue Threshold which is a value of 15 over two samples.

We combine this with how often the monitor is scheduled to run (every 2 minutes shown below).

So, if we put that all together what is our default configuration?

OpsMgr Processor Utilization Threshold default:

Alert when CPU utilization is > 95% for 6 minutes (3 samples at a 2 minute schedule) AND when CPU Queue is > 15% for 4 minutes (2 samples at a 2 minute schedule)

2nd Level Question: How do I set this to override to create a critical high priority alert when a server is using more than 80% cpu over a ten minute period?

Answer:

1. In the Operations Console under Monitoring / Microsoft Windows Server / Windows Server State / find the server(s) you are looking for.
2. Open health explorer / Entity Health / Performance / Operating System Performance Rollup / Performance / Total CPU Utilization Percentage (shown below) and right-click on the monitor and go to properties.
3. Open the Overrides tab and create an override for the object (in this case the Total CPU Utilization Percentage for the specific server I want to override).
4. For the override define Alert on State as Critical, Alert Priority as High, CPU Utilization Percentage Threshold to 80, CPU Queue Length Threshold to 1, and Number of Samples to 5.

SELECT SourceMonitoringObjectDisplayName AS 'Group'

FROM RelationshipGenericView

WHERE TargetMonitoringObjectDisplayName like ('%computername%')

AND (SourceMonitoringObjectDisplayName IN

(SELECT ManagedEntityGenericView.DisplayName

FROM ManagedEntityGenericView INNER JOIN

(SELECT     BaseManagedEntityId

FROM          BaseManagedEntity WITH (NOLOCK)

WHERE      (BaseManagedEntityId = TopLevelHostEntityId) AND (BaseManagedEntityId NOT IN

(SELECT     R.TargetEntityId

FROM          Relationship AS R WITH (NOLOCK) INNER JOIN

dbo.fn_ContainmentRelationshipTypes() AS CRT ON R.RelationshipTypeId = CRT.RelationshipTypeId

WHERE      (R.IsDeleted = 0)))) AS GetTopLevelEntities ON

GetTopLevelEntities.BaseManagedEntityId = ManagedEntityGenericView.Id INNER JOIN

(SELECT DISTINCT BaseManagedEntityId

FROM          TypedManagedEntity WITH (NOLOCK)

WHERE      (ManagedTypeId IN

(SELECT     DerivedManagedTypeId

FROM dbo.fn_DerivedManagedTypes(dbo.fn_ManagedTypeId_Group()) AS fn_DerivedManagedTypes_1))) AS GetOnlyGroups ON

GetOnlyGroups.BaseManagedEntityId = ManagedEntityGenericView.Id))

ORDER BY 'Group'

Thanks to Hamid Yusuff for find this and to the original author who’s article is available at: http://www.virmansec.com/forums/p/97/247.aspx

Issue: In an environment with two DHCP servers with the scope split we had one of the DHCP scopes that were completely empty while the other scope had only three addresses in use. We determined that the IP helper information was incorrect for the second DHCP server.

Resolution: In the network environment, re-configured the iphelper configuration to point to both DHCP servers with the correct IP addresses.

Situation: I was not seeing any of my clustered Exchange databases in the Exchange management pack. I initially thought that this was an issue within the Exchange 2007 management pack.

Debug STEP 1: I started with the assumption that I had not not enabled the appropriate discovery rules on the clustered Exchange pieces, but after checking on those I found that I had already enabled them.

Debug STEP 2: Next I checked the clustering management pack and saw that only one of my six clusters in the environment were appearing in the clustering management pack/so I figured it was cluster Management Pack related.

Debug STEP 3: To debug the clustering MP I opened the Operations Manager logs on the agent and found the following:

Client event: 1102 Source HealthService

Rule/Monitor "System.Mom.BackwardCompatibility.Computer.IsHostingMSCS.Discovery" running for instance "SERVER" with id:"{ED873637-669D-57E6-6645-1872603D143E}" cannot be initialized and will not be loaded. Management group "MG1"

Debug STEP 4: I dug into the RMS and ran across these events which would recur every time I restarted the System Center Management service on the cluster nodes.

RMS event: 33333 DataAccess Layer

Data Access Layer rejected retry on SqlError:

Request: p_RelationshipDiscovered -- (RelationshipId=24d9c744-f0f0-f598-16a5-c2f224db276d), (SourceEntityId=8cee6f8d-a6dc-3000-9879-af0b6fbf427b), (TargetEntityId=f6623c80-23a7-77e7-21ac-894396297850), (RelationshipTypeId=6284518a-90cc-fadd-e56b-26669e2a0a8f), (DiscoverySourceId=08e62d16-8a8d-c913-e23d-767a046cb17d), (HealthServiceEntityId=4c0bc849-29e2-f792-ba3d-36c840629c50), (PerformHealthServiceCheck=True), (TimeGenerated=11/12/2009 8:50:48 PM), (RETURN_VALUE=1)

Class: 16

Number: 777980002

Message: The specified relationship doesn't have a valid source.

RMS event: 10801 Health Service Modules

Discovery data couldn't be inserted to the database. This could have happened because of one of the following reasons:

- Discovery data is stale. The discovery data is generated by an MP recently deleted.

- Database connectivity problems or database running out of space.

- Discovery data received is not valid.

The following details should help to further diagnose:

DiscoveryId: 0f1fba2c-d205-edb2-2259-1430d6206c10

HealthServiceId: 4c0bc849-29e2-f792-ba3d-36c840629c50

Invalid relationship source specified in the discovery data item.

RelationshipSourceBaseManagedEntityId: 8cee6f8d-a6dc-3000-9879-af0b6fbf427b

RuleId: 0f1fba2c-d205-edb2-2259-1430d6206c10

Instance:

5c324096-d928-76db-e9e7-e629dcc261b1HPSMXCLUSTER01.hps.com5c324096-d928-76db-e9e7-e629dcc261b1HPSMXCLUSTER01.hps.coma27b57c5-6479-9ce8-8682-b83228266cebCCR Cluster.

Debug STEP5: After reviewing these events I needed to translate some of the GUID type numbers into something that I could figure out what it was.

Using this query I determined what systems were reporting errors: (where the actual GUID value is specified in the alert)

Select * from dbo.BaseManagedEntity where BaseManagedEntityID = 'GUID'

Using this query I determined what discovery was failing: (where the actual GUID value is specified in the alert)

select MP.MPName, D.DiscoveryName, MT.TypeName from Discovery D with (nolock)

join ManagementPack MP with(nolock) on MP.ManagementPackId = D.ManagementPackId

join ManagedType MT with(nolock) on MT.ManagedTypeId = D.DiscoveryTarget

where D.DiscoveryId = 'GUID'

Using this query I determined what the relationshiptypeid was: (where the actual GUID value is specified in the alert)

select * from relationshiptype where RelationshipTypeId='GUID'

Debug STEP 6: From web research based upon what I received in the event logs and the systems reporting the error it seemed to be pointing out that I had not enabled proxy on these servers  (http://systemcenterinfo.blogspot.com/2008/06/scom-alert-agent-proxying-needs-to-be.html). So I verified that I had enabled proxy on the systems. I tried enabling proxy both from the UI, and even from ProxyCfg (http://wchomak.spaces.live.com/blog/cns!F56EFE25599555EC!738.entry). But any way I investigated this it was enabled.

Debug STEP 7: Figuring that this might be a bad agent configuration, I tried to uninstall the OpsMgr Agents and delete them from the console. Then I re-deployed the agents, re-configured proxy (which was still enabled) and re-tested. The same events were occurring on the RMS and client systems.

Side-Thoughts: At about this point in time my mind started rambling to the title of this blog article because it seemed like anything I tried wouldn’t get the darn cluster identified: No Cluster for You!

(for those who haven’t seen this or it’s before your time see below)