Automating Scale-out of an Application Tier with System Center and the Veeam v7 MP

This article, which details how to scale out a VMM service instance with Orchestrator to automate the provisioning of additional capacity, is a supplemental post to the Veeam whitepaper “Automate Management of your Hyper-V Environment: Next Level Integration of System Center and Veeam Management Pack v7”, available at http://veeam.com/.

Assumptions

Before we get started, there are a couple of assumptions I am making:

  • You have read the Veeam whitepaper
  • You are familiar with Orchestrator and Orchestrator integration with other System Center 2012 components, most importantly SCOM and Virtual Machine Manager (VMM)
  • You have administrator access to a System Center environment, including:
    • SCOM (with the Veeam v7 MP for Hyper-V deployed
    • Orchestrator (with the SCOM and SCSM Integration Packs deployed)
    • SCSM

If you are all good on the points above, read on!

A Note on Scale Out of a Service in VMM

In case you are not familiar with machine tier scale out, it is the process of adding an additional VM to a specific tier of a running service in VMM. After you have deployed a service in VMM and you need to deploy additional virtual machines to a tier of the service, you can use the scale out functionality of VMM. For example, during the holiday shopping season, your website may need additional web servers deployed to handle the increase in traffic to your site.When you create a tier in a service template, you can specify whether that tier can be scaled out and the minimum and maximum number of virtual machines that you can deploy in the tier.

If you try to scale out a tier beyond its maximum tier size, you will receive a warning, but VMM will not prevent you from scaling out the tier. However, after the virtual machine is deployed, the tier and the service will show a status of Needs Attention in the VMs and Services workspace.

NOTE: In writing this chapter, we assumed not everyone is a VMM expert and so provided quite a bit of background info in chapter 16 to give you the basics of services and service templates in VMM 2012 SP1.

In This Post

In this post, we will look at the details of a simple runbook to scale out a VMM machine tier, as well as a before and after snapshot of the running service instance in System Center 2012 Virtual Machine Manager SP1

  • The Scale Out ServiceRunbook
    • High Level View
    • Runbook Configuration
  • Service Instance Before Scale Out
  • Service Instance After Scale Out
  • Fully Automated Scale Out with the Veeam 7 MP

I’ve added a few tips throughout to help you get it right in the real world.

High Level View of the Scale Out Service Runbook

Here is the high-level view of the runbook, which can easily be called from SC 2012 Service Manager or externally using C# or PowerShell.

image

This runbook performs the following actions steps:

  • Accepts user input required to ensure the scale out operation is performed on correct running service instance and machine tier
  • Retrieves the running service instance provided in user input
  • Retrieves the  machine tier targeted for scale out (as provided in user input)
  • Initiates the scale out operation

TIP: You will notice this runbook has no error checking, it’s just a simple version to show how to carry out the process (what I call a drag-drop-done version). Use the recommendations we provide in the book for adding error handling to this version before you put it into production use.

The Scale Out Service Runbook

The runbook requires two pieces of information from the user, collected by the Initialize Data activity, as shown in the image below:

  • Service Name – The display name of the running service instance in VMM
  • Tier Name – The display name of the service template

image

The Get Service activity (shown below), which retrieves the details of the running service instance, requires one parameter, the Service Name, provided in the user input and retrieved using the subscribe to published data feature in Orchestrator.

Service Name Equals {Service Name from “Initialize Data”}

image

The Get Tier activity (shown below), which retrieves the details of the machine tier targeted for scale out within running service instance , requires two parameter, the Tier Name and the Service Name, provided in the user input and retrieved using the subscribe to published data feature in Orchestrator.

Tier Name Equals {Tier Name from “Initialize Data”}

Service Name Equals {Service Name from “Get Service”}

TIP: I recommend to customers to establish consistent names for machine tiers within their services (like ‘Web Tier’, ‘App Tier’, ‘Data Tier’, etc.). If you follow this strategy, you will definitely need to provide the service instance by name or ID to ensure the runbook only retrieves the machine tier instance within the target service. Otherwise, multiple machine tiers (in multiple services) may be retrieved by this activity. If you use service name, you will also need to be sure you use unique names when creating services in VMM. More on this in an article on service deployment in VMM with Orchestrator.

image

The Scale Tier Out activity (shown below), which initiates the scale out operation, requires four parameters, the Service Name,  Tier Name, StartAction and Stopaction, configured as described below

Service  Name Equals {Service  Name from “Initialize Data”}+

Tier Name Equals {Tier Name from “Initialize Data”}+

StartAction to set VM startup behavior, set to ‘Never automatically turn on…’ in this example

StopAction to set VM stop behavior, set to ‘Save state’ in this example

Use the picker in StartAction and StopAction to choose preferred behavior from the list provided.

TIP: You could also retrieve the Service Name and Tier Name from the Get Service and Get Tier activities, though in this case presumably those will have failed if the values provided to Initialize Data were not correct. You always want to configure the most reliable options in your runbooks.

image

In Optional Properties, you will find you can add properties such as Computer Name, Virtual Machine Name and Description if you wish to specify the VM name in advance or to add a description.

Service Instance Before Scale Out

Service instance before scale out

image

Service Instance After Scale Out

Service instance after scale out

Immediately after the runbook executes, you will see the scale out operation results in an additional VM being added to the tier.

image

Please leave questions and feedback in the comments section below this post.

Fully Automated Scale Out with the Veeam 7 MP

What is great about automating the scale out is that you can use your existing scale-out runbook if you already have one. The primary challenge with automating scale-out is finding an accurate performmance metric to trigger the scale-out process.  This is due to the fact that you cannot accurately measure the resource utilization of a virtual machine through performance metrics gathered in the virtual machine. Fortunately, the Veeam 7 MP includes multiple unit monitors that enable administrators to potentially automate the scale out process.

Veeam HyperV: Memory Pressure Analysis – This monitor tracks threshold breaches for the following metric: ‘Hyper-V Dynamic Memory VM \ % Average Pressure,’ which measures the average memory pressure in the VM; 100 or less is normal, more than 100 represents a shortage.

Veeam HyperV: VM CPU Usage – This monitor tracks threshold breaches for the ‘Hyper-V Virtual Processor \ % Total Run Time’ counter, which measures the amount of CPU usage for the guest OS.

The VM performance counters collected by the Veeam v7 MP for Hyper-V is that the counters are collected from a Hyper-V host perspective, ensuring accurate detection of performance spikes.

The next challenge in the process is providing input to the scale-out runbook, specifically the service name and machine tier within the service where the virtual machine resides. The issue here is that the virtual machine class in SCOM does not include properties detailing the service name and machine tier to which the virtual machine belongs. We have to perform a lookup of this value. You have a couple of options for this step:

  • You can store these two properties for each VM in a reference database and perform the lookup , or
  • configure a runbook to lookup these properties directly from VMM at runtime.

I prefer the second option, as it reduces complexity of the solution.

High-level Process and Runbook Design

Scaling out a machine tier within a service is often a fully automated activity for presentation tier components behind NLB or hardware load balancers.The high level process is shown in the figure below.

supp_post2_process_diag2

With the high level process and runbook design details in hand, lets shift focus to the details of runbook configuration.

Runbook Configuration

The following outlines the configuration of the activities within the runbook.

Monitor Alert

As you would expect, this is the Monitor Alert activity from the SCOM Integration Pack. The fields you should configure in this activity are:

  • Name: Alert name, should be configured to match on “Alert Name” contains “Veeam”
  • IsMonitorAlert: should be set to true.
  • NetBiosComputerName: Should be configured to match the naming convention of your application or database tier servers…those tiers that cannot scale out, and so must be scaled up.

Find Veeam Alerts

This renamed Map Published Data activity is meant to filter out the Veeam alerts for:

  • for CPU and memory utilization
  • for only the VMs in the application or data tier (depending on your scenario)

The objective of this activity is to filter down to only the alerts we want to use as scale-out triggers. In the properties of the activity, you click the Add button to open the Add Mapping dialogue, where you can insert

  • Source data: Alert name from Monitor Alert
  • Pattern: What alert name we are looking for
  • Map To: Type the new text that replaces the text of those items that match Pattern. This is the data that will be inserted into the incident and / or change request.

TIP: You can add entries to map as many different alerts from the Veeam MP as you like, along with properties of the various fields in the SCOM alert, enabling automatic creation of very descriptive incidents and changes, providing detailed information to assist subject-matter experts in making good decisions when adding compute resources to affected systems.

Lookup Service Tier

This renamed Run .Net Script activity will run a PowerShell script to connect to your VMM instance and loop through the active service instances to find the service and machine tier containing the VM that raised the alert. Lots of options for how to do this, but this is a fairly short script in VMM PowerShell-speak.

Trigger Scale-out Runbook

This renamed Invoke Runbook activity triggers the scale-out runbook detailed earlier in this article. The required inputs, if you look above, are Service Name, Machine Tier, VM StartAction and VM StopAction.

Conclusion

I hope you have found both the whitepaper and this supplemental post informative. If you have not already, I encourage you to download “Automate Management of your Hyper-V Environment: Next Level Integration of System Center and Veeam Management Pack v7”, available at http://veeam.com/.

Leave a Reply