Virtual Operations Manager: Part 1 - Should I Virtualize?

This is the first installment of our Virtual Operations Manager series published originally on SystemCenterForum.org. Just updating for R2 and moving to our new home here at System Center Central…

Virtualization offers a number of potential benefits, such as reduced power consumption, hardware expense and licensing expense through server consolidation. Snapshot technologies make rolling back changes easier than on a physical server. Clustering options offer flexible options for high availability and disaster recovery scenarios as well. Virtualization is everywhere, and it's only natural that Operations Manager 2007 architects and administrators would be curious know about the possibilities and best practices for virtualizing OpsMgr 2007 server roles in their production environments.

In our first installment, we're going to examine what is supported versus what is recommended. In short we'll answer the questions "Which OpsMgr roles can I virtualize?" versus "Which OpsMgr roles should I virtualize?" I'll be sure to cite official Microsoft sources whenever possible so you have materials to make your case to management when arguing in favor of virtualization for your OpsMgr environment. Fundamentally, this information should be equally valid for Operations Manager 2007 SP1 and R2 releases.

NOTE: In this series, we will limit our discussions to Hyper-V and VMware Virtual Infrastructure 3.x. There will be a later architecture installment in which we cover platform-specific pros and cons of Hyper-v and VMware platforms.

Supported for Virtualization Versus Recommended for Virtualization

All Operations Manager roles are supported on Microsoft enterprise virtualization platforms (Virtual Server 2005 R2 and Hyper-V officially - more on VMware in a moment), but this does not mean that all should be. Let's have a look at each of the OpsMgr roles and examine which are good candidates for virtualization.

IMPORTANT: See the OpsMgr Performance and Scalability Whitepaper and for detailed hardware sizing info, and provide CPU, memory and disk in equivalent quantities to virtual machines hosting these roles.

Role	Supported	Recommendation
Root Management Server	Yes	Not Recommended. This role uses large amounts of memory and CPU. Should only be virtualized in smaller environments (<500 servers). I've seen the RMS virtualized in smaller environments, but memory and processor demands (which can reach multi-proc servers with 12GB RAM in large deployments) make virtualization a challenge in high scale scenarios.
Management Server	Yes	Recommended. Since resource requirements are light (4GB RAM max), this role can be easily virtualized. Most important resources for this role are memory and CPU.
Gateway Server	Yes	Recommended. Since resource requirements are light (about 2GB RAM in most cases), this role can be easily virtualized. Most important resources for this role are memory and CPU.
Operational DB	Yes	Not Recommended. This role uses large amounts of CPU, Memory and Disk I/O. Should only be virtualized in smaller environments (<500 servers).
Data Warehouse	Yes	Not Recommended. This role uses large amounts of CPU, Memory and Disk I/O. Should only be virtualized in smaller environments (<500 servers).
Web Console*	Yes	Conditionally Recommended (unofficial) There are no official sizing guidelines for a dedicated server hosting the Web Console today, but we've virtualized this role successfully in several environments. This leaves more resources on the RMS for cocnsole connections and other core RMS functions. CPU and memory are important, but in large and busy environments, the IIS log writes will generate some disk I/O, so make sure you keep an eye on the disk subsystem.Start with 2GB RAM and equivalent of dual core CPU allocated to the VM. Scale up as usage levels demand.
Reporting Role*	Yes	Conditionally Recommended (unofficial) Moving SQL Reporting Services off of the RMS can eliminate competition for resources and provide SRS with dedicated resources, resulting in a better reporting experience. HOWEVER, the Operations Console communicates directly with this role, so failure to provide adequate resources can affect the console experience.Start with 2GB RAM and equivalent of dual core CPU allocated to the VM. Scale up as usage levels demand.
Management Server hosting ACS Collector role	Yes	Usually Not Recommended. The primary scenarios where this role is successfully virtualized are for either the DR/Failover Collector (a feature added in OpsMgr SP1), or in scenarios where high scalability is not necessary. Such scenarios would inlude satellite offices or perimeter networks with a small number of hosts.Let performance dictate when a move from virtual to physical is required.

Table 1 - Virtualization Support and Recommendations for OpsMgr Server Roles

*I see and hear of these two roles most often being separated (and sometimes virtualized) in very large environnments. I do not suspect most environments will see the need to do this. Because these roles can affect the user experience, it's important to get it right if you do choose to virtualize.

Resource Requirements and the Virtualization Decision

To expand a bit on the recommendations above, whether it is practical to virtualize comes down to resource requirements, performance and scalability. Gateways and secondary management servers (SMS) can be easily virtualized because their resource requirements are generally more modest when compared to the RMS. Gateways and Secondary Management Servers are also easier to virtualize because these roles do not scale up, they scale out. I'll clarify what I mean here with an example.

Example:

In an OpsMgr implementation for an environment with 1,000 - 3,000 servers, consider the following sizing requirements as described in the OpsMgr Performance and Scalability Whitepaper.

Secondary management servers (secondary meaning "not the RMS") will have a max of about 4GB RAM and a 2-way processor configuration. When a maximum of 2,000 agents are reporting to a single management server, you must add additional management servers to share the additional load.
A Gateway in comparison will have 2GB RAM and a 2-way processor configuration. When a maximum of 800 agents are reporting to a single Gateway, you must add additional management servers to share the additional load.

By comparison, in a 1000-3000 agent environment, the RMS sizing recommendations are 12GB RAM 64-bit quad processor and 4-drive RAID 10! This is because the RMS handles several important roles not shared by other management servers. For example, the RMS is responsible for Operations Console connections, state calculation, and notification.

In the case of the RMS we cannot scale out (add additional servers), and the RMS is going to aggressively make use of these resources (especially RAM) - they will not sit idle. With the resource requirements, the virtualization host would have to be quite large to provide this level of dedicated resources.

The bottom line is this: If you have to scale up and high performance is a high priority, you should probably go physical.

Common Mistakes / Considerations

Overcommitting Memory Resources on the Host- Not committing enough CPU and memory to a server role is a common source of poor performance. You need simply follow the resource requirements listed in the Performance and Scalability Whitepaper regardless of whether the server is virtual or physical. On the VMware platform, this means being careful to not overcommit memory. This is really a more of an issue on VMware, which offers advanced memory management (transparent memory page sharing and memory ballooning) allowing you to achieve greater VM density (higher VM counts) on a single host, but at a potential cost of performance. This type of memory management is not possible in Hyper-V today. I don't want this to sound like I'm knocking VMware, because this is a great feature when used appropriately. The moral of the story is, be very sure to dedicate the required CPU and memory resources to OpsMgr server roles, especially when virtualizing the RMS role, as it needs both in substantial quantities. If you place the RMS (or any MS, for that matter) on an overcommitted host, you are asking for trouble. Failing to plan for availability - If you put multiple management and / or gateway servers on the same host, when the host goes down, you lose all those server roles. If you virtualize multiple server roles, spread them across multiple VM hosts and host clusters when possible.

Inadequate Storage Infrastructure - VMs generally reside on shared storage, whether local direct-attached (DAS), SAN or iSCSI. When placed on the same shared array, VMs compete for disk I/O. Be sure to plan for adequate disk I/O, just as you would with a physical server.

Virtualizing SQL Server (any version)

While there are potential hardware, power and SQL licensing savings in virtualizing SQL Server, SQL is not a good candidate for virtualization in scenarios that require high throughput. In addition to the CPU and memory intensive nature of SQL Server, the most significant hurdle is the disk subsystem. You will most likely use the native virtual storage technique that comes with the virtual server software. This means that the hard drive that the virtual machine sees is actually a single file stored on the host machine's hard drive. You could opt for SAN or iSCSI storage to provide greater (and easily expandable) disk I/O, but even this fails to address performance in one respect. In a virtual environment, your I/O is going to pass through an additional virtual layer. In Hyper-V this is a virtual device driver in the root partition, or in VMware, a device driver in the hypervisor itself. This results in lower throughput to the disk subsystem. For the official Microsoft party line on virtualizing SQL Server, see Using SQL Server in a Virtual Environment.

Platform Support

While Hyper-V is Micrsoft's preferred platform for virtualizing Windows, VMware has gained a degree of official support through the Microsoft Server Virtualization Validation Program (SVVP). For more information, visit the SVVP website. I would estimate that more than 95% of the OpsMgr environments I see virtualized today reside in a VMware environment, so it definitely works.

Making the Case for Virtualization

There are several good resources out there you can use to make your case for virtualizing OpsMgr

The OpsMgr 2007 Infrastructure Planning and Design Solution Accelerator includes recommendations around virtualization. This is officially supported guidance.
Satya Vel (aka DeploymentMan) has a post on the Operations Manager Team Blog where he discusses support for the RMS, MS, Gateway and SQL in particular and his personal recommendations.
As mentioned previously, always consult the OpsMgr Performance and Scalability Whiteaper for hardware sizing requirements. VMs should be allocated equivalent resources required by their physical counterparts!

Conclusion

Hopefully this provides some food for thought and the basis for useful community discussion. Next in our series we will tackle an interesting topic.....storage options and considerations for your virtual OpsMgr infrastructure.

Please leave your feedback as a comment on this post.