SCOM 2012 RC – Side-by-Side Migration Phase 1 – Preparations

Important Note! 2012 RC is not supported in a production environment unless you are in the Microsoft TAP program for Operations Manager.
This blog post is to be considered an experience report, and not an official guideline.

 

Table of contents
– Rant
– Challenges
– Brainstorming
– Phase 1 – Removing 2 of the Management Servers from the current environment
– Closing Words

 

Rant
There’s always a hand full of servers that just aren’t compatible with a new technology, relatively the operators of these servers aren’t… always.

The main issue I am facing is that some servers just won’t receive Service Pack 2 for Windows Server 2003/2008, which is a prerequisite for the Operations Manager 2012 Agent. They will be decommissioned soon™ anyway, but until then they still need to be monitored. What to do in this case? Basically I have these options on my table:
– #1 Setup a new 2007 R2 environment, move the incompatible servers to that Management Group. Upgrade original one to 2012
– #2 Setup a new 2012 environment, move compatible servers to that Management Group. Leave 2007 R2 environment “as it is”.
– #3 Just migrate the Management Group and be like “well if you want your servers monitored, patch them”

Of course, the third option is not going to happen, so it’s either 1 or 2. Option 1 is maybe the most efficient, while Option 2 is the most secure – and more importantly, the more fun approach.

Now let me tell you something about our environment. It went from a 2005 Database, to a 2008 Database, from 2007 to 2007 R2, and from a 2008 Database to a 2008 R2 Database. And I have to move the Data Warehouse from that Environment to a new Database, which is a monster btw. – Our current Operational Database is quite a beast as well but the new machine to which we are supposed to move the Data Warehouse, is just slightly better, especially the Disks. So, I was playing with the idea to move the Operational DB to that machine anyway.

Taking all this into account, the most logical approach to me is to spare me the whole database juggling, the downtime during the upgrade, the dangers of it failing, etc. and go with option 2 – side-by-side migration and have a clean new installation.
Aside from that, it should be fun moving all those Management Packs, Groups and Overrides to the new environment.

This part of the series will be a bit chaotic since there’s been a lot of work during the past few months already to get all the Management Packs and SDK code to be 2012 compatible. I won’t cover that here, simply because it’s very individual.
Management Packs were absolutely no problem btw., but I wanted to verify that they work – and they do. Migrating the SDK code was a bit more tricky.

 

Challenges
Consoles: The consoles can only connect to their respective version, meaning a 2007 console can only connect to 2007, a 2012 only to 2012. On top of that, one can’t have both consoles installed at the same time. Gladly, we have more than one Terminal Server.
Moving Groups and Overrides: Dynamic Groups aren’t a big issue, groups with explicit members are though. And then of course, the Overrides. Some SDK Wizardry will come in handy here Smile
Distribution of the Agents: We are using Active Directory Integration right now, which we will still use in the 2012 environment. The tricky part here will be to get the Agents to not communicate with the 2007 R2 environment anymore.

 

Brainstorming
The first thing I need is new Management Servers, a new Operational Database, and a new Data Warehouse Database. And that’s what I’m going to cover here.

#1 Hardware
Since we have quite a few Management Servers, I’m going to take 2 away from the 2007 R2 environment.
The Operational Database will be moved to the new Database Server, the Data Warehouse Database will be put on the “old” Operational Database Server.

And then of course, I need Gateway Servers in the 2012 environment. I will replace the current ones by virtual machines and use them for the 2012 setup. But that’ll be done way later on.

#2 Agents
2012 Agents are compatible with 2007 Management/Gateway Servers. Therefore they can report to both Management Groups.
Most of our servers already have received the upgrade for the agent which has been done through SCCM. There really isn’t much to say about this.
The command line we used: msiexec /i MOMAgent.msi /qn /l*v C:\windows\logs\AgentUpgrade.log

The Management Group configuration is distributed through Active Directory Integration. Now what I need to do is reconfiguring the AD Integration rules to totally not use the 2 Management Servers I selected for being decommissioned from the 2007 R2 environment, whether as primary nor as secondary.

#3 Gateways
The Gateways will be setup later on once the new environment is up and running and the Agents of the main domain were all moved.

In the current phase I need to make sure though that the Gateway Servers don’t try to communicate with the 2 Management Servers I selected for being decommissioned from the 2007 R2 environment.
For this, I simply adjust my rules in my Gateway Server Failover Management Pack which you can find here btw. – http://www.systemcentercentral.com/BlogDetails/tabid/143/IndexID/88019/Default.aspx

 

Phase 1 – Removing 2 of the Management Servers from the current environment
Agent Distribution:
I had to make sure that the Agents don’t try to communicate with these servers anymore, for this I adjusted the Active Directory Integration rules to whether primarily communicate with the 2 servers I selected, nor as failover. This lead to some big trouble again. I can’t tell whether it was my fault, if replication was too slow or if it is a bug with the AD Integration rules in general.
Basically what happened was that I added a new rule, verified that it was in AD, then I waited some time and deleted an old one leading to the Agents not getting any configuration from AD anymore. That alone wouldn’t hurt, what really hurt is that they just didn’t get the new configuration.  And the best part, the Agents stopped.

Now, when you have to start a service again on some 400 machines – use SCOM…. hahaha that doesn’t work in this case. What saved the day was Remote PowerShell.
So what I did was, I queried for all Heartbeat Failure Alerts and started the HealthService again. The script looked something like this:
$HeartbeatFailures = Get-Alert | where {$_.Name.StartsWith(“Health Service Heartbeat”)}

foreach ($HeartbeatFailure in $HeartbeatFailures)
{   
    trap { continue }
    $ServerName = $HeartbeatFailure.MonitoringObjectDisplayName
    Write-Host “Starting HealthService on Server $ServerName”
    $Service = Get-Service -Name “HealthService” -ComputerName $HeartbeatFailure $ServerName
    if ($Service.Status -eq “Stopped”) { $Service.Start() }
    if ($Error.Count -gt 0) { Write-Host “Error starting Service – $Error” }
}

Domain Controllers can’t receive their configuration from AD which is… well I guess Microsoft had their reasons.

Once all is set, I checked under Administration\Agent Managed that the Management Servers in question don’t have any Agents assigned to them anymore.
image

 

Gateway Server Communication:
As pointed out already, I have a Management Pack for that which you can find here – http://www.systemcentercentral.com/BlogDetails/tabid/143/IndexID/88019/Default.aspx
I simply changed the Primary/Failover Server setting so the gateways don’t use these 2 Management Servers anymore.

I verified that the config is up to date by checking the registry keys on the servers (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Server Management Groups\BWIN\Parent Health Services)
And of course, in the Operations Manager Shell
foreach ($MS in (Get-ManagementServer))
{
$Name = $MS.Name
$Prim = $MS | Get-PrimaryManagementServer
$Fail = $MS | Get-FailoverManagementServer
write-host “MS:”$Name “Primary:”$Prim.NAme “Failover:”$Fail.Name
}

Removing the Management Servers:
Before I remove them, I set them into Maintenance mode for some time to see if there isn’t some leftover Agent that then fails to heartbeat. Or if there’s even a bulk of Agents that fail to hearbeat because I have overlooked something.

Nothing like that happened, time to uninstall the Management Server components.
image

Once that is done, I need to remove them from the Management Group.
image

 

Cleaning up the Management Servers:
The uninstall of SCOM 2007 isn’t really clean, so I have to manually remove a few things:
Delete folder %programfiles%\System Center Opaerations Manager 2007
Delete Registry Key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager

Closing words
Now I have everything I need to setup the 2012 environment.
2 Servers that will be my 2012 Management Servers
1 Database for Operational Data
1 Database for DWH Data

Look forward to the next blog post when I show you all about clicking next-next in the SCOM 2012 RC setup Winking smile

Leave a Reply