Critical alerts when failing over Operations Manager database in Always On configuration

We ran into an interesting issue when we attempted to fail over our OperationsManager database from one node to the other node in an Always On SQL 2012 configuration. Operations Manager was functioning correctly when the database was on the primary node, but when it was moved to the other node we started to receive a variety of critical alerts. These included:

  • Performance data collection process unable to write to the Data Warehouse
  • Object Health State data collection process unable to write to the Data Warehouse
  • Event data collection process unable to write to the Data Warehouse
  • Event data collection process unable to write to the Data Warehouse in a timely manner
  • Data Warehouse relationship synchronization process failed to read state
  • Data Warehouse object health state data dedicated maintenance process failed to perform
  • Data Warehouse monitor initial state data synchronization process failed to read state
  • Data Warehouse managed object type synchronization process failed to read state
  • Data Warehouse managed object synchronization process failed to read state
  • Data Warehouse maintenance mode synchronization process failed to read state
  • Data Warehouse health service availability data synchronization failed to read state
  • Data Warehouse failed to request a list of management packs that have all of their dependencies
  • Data Warehouse failed to request a list of management packs installed for their management
  • Data Warehouse failed to enumerate database components to be deployed
  • Data Warehouse failed to deploy database component
  • Data Warehouse data set enumeration process failed to obtain list of data sets
  • Data Warehouse configuration synchronization process failed to read state
  • Data Warehouse alert data dedicated maintenance process failed to perform maintenance

Needless to say, it was pretty obvious that SOMETHING was wrong. When reverted the database back the original node, the alerts closed so this was a repeatable problem.

Resolution: When configuring SQL always on, it is important that the logins are configured consistently on each node of the cluster. Otherwise the above alerts may occur if the appropriate account cannot log into the database. In our case, the primary node had the logins defined correctly but the secondary node was missing one of the required accounts. Once we fixed this the alerts auto-closed and the issue has not recurred.

 

4 thoughts on “Critical alerts when failing over Operations Manager database in Always On configuration

  1. Lee

    Had this with our environment, I believe we were 1 of the first companies to go for always on and a full 2012 env.

     

    This can also happen with orchestrator I believe (i’m pretty sure it happened on there for us too)

     

    Good write up Cameron 🙂

  2. Cameron Fuller Post author

    Thanks Lee! Always On brings interesting functionality but there isn’t much documentation on how to debug when weird stuff occurs like this. Seeing it in action however, it’s a very cool approach to providing highly available SQL for System Center!

  3. Pingback: SCOM link - IT Consult

  4. Pingback: link scom 2012 troubleshooting, how-to, management pack, patch, cmdlet, planning, deployment, installation | IT Consult

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.