Orchestrator Runbook Automation JumpStart: Part 2 – Advanced Features and Functionality

If you read part 1 and completed the hands-on labs, you now have a foundation in Orchestrator runbook functionality and authoring. In part 2, we are going to build on this functionality. This installment will discuss advanced runbook concepts that will allow you to incorporate advanced decision logic into modular runbooks that will support reuse.

HAND ON LABS: As mentioned in part 1, to reinforce the concepts covered in this installment, you will find a pointer to accompanying hands-on lab exercises (on Microsoft’s TechNet Virtual Labs) that reinforce the concepts discussed here. These labs are largely Opalis 6.3 labs, but the authoring experience (interface, runbook concepts, etc.) in Orchestrator is basically unchanged from that of Opalis (although some terminology has been updated). In fact, the runbooks used in these Opalis labs all work in Orchestrator 2012.

In the Series

The series will consist of the following articles. Since the entire series is ready, you should see a new article every 3 days or so.

In This Installment

  • Advanced Runbook Features and Functionality
  • Testing and Troubleshooting Your Runbooks
  • Taking Automation to the next level with Orchestrator and System Center
  • Hands on Labs

New to Orchestrator? Go back and read part 1 in the series before continuing.

Advanced Runbook Features and Functionality

With the basics under your belt, it’s time to explore some of the more advanced features of Orchestrator. This section will touch a number of advanced runbook elements, including:

  • Runbook Nesting
  • Looping
  • Junctions
  • Data Manipulation Functions
  • Variables
  • Complex Link Logic (advanced branching) [coming in part 3]

Runbook Nesting

You can initiate one Orchestrator runbook from within another runbook using an Invoke Runbook activity. The runbook being called is referred to as the child runbook, and the runbook that is making the call using the Invoke Runbook activity is known as the parent runbook. It is possible for data to be passed to the child runbook from the parent runbook, allowing data from the child runbook to be published for use in the parent runbook.

As discussed in part 1, data publishing behavior at the activity level within a single runbook is automatic. This is not the case when working with multiple runbooks and attempting to publish data between them. By default, no published data from within the child runbook is visible to the parent; you must perform some additional configuration steps to make the output of the child runbook available to the parent. For a look at how to configure data publishing between runbooks, check out this blog post “Opalis: working around limitations with workflow activities and link operators” from Opalis Integration Server 6.3 Unleashed contributing author Marcus Oh.

You will want to get familiar with Invoke Runbook and publishing data between runbooks, because this plays an important role in Orchestrator runbook authoring best practices. This will be discussed more in part 3 of this series.

Looping

Looping allows you to build automatic retries, monitoring, and validation into a runbook, and at multiple points within the runbook if needed. Orchestrator enables you to configure conditions to exit the loop. For example, if you wish to exit after a minimum number of executions or after a certain amount of time, you can select the associated “Loop” Common Published data on the Exit tab, then select ‘Number of attempts’, as displayed in Figure 1.

image

 Figure 1 – Exit Tab Configuration in runbook Properties

You can define looping behaviors on the Exit and Do Not Exit tabs in Looping properties. Note that the rules on the Do Not Exit tab always take precedence over the rules on the Exit tab.

  • The Exit tab shown in Figure 1 specifies the conditions that will determine if the loop will exit.
  • The Do Not Exit tab specifies the conditions that will cause the loop to continue.

Looping can be configured only available at the activity level, not at the runbook level. However, you can work around this limitation with the Invoke Runbook activity, as explained by Ryan Andorfer in his post “Looping a Policy.”

Junctions

The Junction activity enables you to configure a runbook to allow multiple branches in the runbook to complete before it continues past the Junction activity. In addition, the Junction activity can republish data from any one branch so that activities downstream of the Junction activity can consume the data. Data from branches other than the one you select will not be available. You can also choose “None” to stop propagation of published data from any of the branches upstream of the Junction activity if the data is not needed downstream in the runbook.

For example, the Operations Manager maintenance mode runbook in Figure 2 uses a Junction to ensure the Start Maintenance Mode activities for the Windows Computer, Operations Manager Health Service, and Health Service Watcher (HSW) have completed before attempting to restart the computer. A copy of this runbook is included in the code download included with this article.

image

Figure 2 – Operations Manager 2007 R2 Computer Maintenance Mode runbook*

 

*What maintenance mode looks like in a runbook for OpsMgr 2012 will be a bit different than what you see in figure 2.  More on this in a later post.

Data Manipulation Functions

The data manipulation functions in Orchestrator enable you to manipulate string, numeric, and date/time data from published data items or other sources, and convert this into a usable form. You can also perform arithmetic operations on numeric data, including addition, subtraction, multiplication, and division. For example, you can extract data from a text file using a text file management activity, trim leading and trailing spaces from the text, and then retrieve specific parts of the text that you can pass to other activities as Published Data. Microsoft’s article at http://technet.microsoft.com/en-us/library/gg440683.aspx provides a detailed description of all data manipulation functions.

Variables

As you are building runbooks, you may find there are values that are the same across different activities in multiple runbooks. When these values need to be updated, it becomes inconvenient to individually change each activity. Variables act as a placeholder in your runbooks, enabling you to specify a value in one location and then use that value globally in any activity. At run time, the workflow engine translates variables configured in an activity during design time.

The funny thing about a variable in Orchestrator is that it rarely varies at all! A variable is almost always set to a static value, meaning a variable in Orchestrator is much like a constant in common programming languages. However, there are two exceptions to the rule:

  • The NOW() function, which returns info on the current date/time
  • System environment variables (such as %WINDIR%, shown in Figure 3)

image

Figure 3 – An environment variable used as the variable value

You can read more about these special Orchestrator variables at http://technet.microsoft.com/en-us/library/gg440631.aspx.

Complex Link Logic (advanced branching)

Links connect activities in a runbook and direct the flow of activity and data within a runbook. Links provide precedence between two activities. The default logic on a link between two activities is “Success”. This means that if the activity runs (regardless of result), the downstream activity to which it is linked will execute. However, link conditions can provide sophisticated functionality for implementing complex decision flows involving text, numeric or time-related data.

Part 3 of this series discusses some of the possibilities of links and branching in depth.

Testing and Troubleshooting Your Runbooks

When it is running, every runbook generates a log which you can view in the Runbook Designer. When you are viewing a runbook, the Log and Log History windows will display the real-time and historic logs for the selected runbook. The Orchestrator Web Console allows you to view runbook execution in real time:

  • The Log window displays the logging of the current run of the runbook. The log displays when the runbook started, and the activity currently running in the runbook will have the running label beside it. The log enables you to determine if there are any problems with specific activities in your runbook.
  • The Log History window, shown in Figure 4, displays all previous executions of the runbook you selected. The time that the runbook started and ended is displayed at the top of each log entry, and the result of each activity execution is displayed for each entry.

image

Figure 4 – Log and Log History tabs in the Runbook Designer

Double-clicking an activity log entry in the log lets you view the result of the execution of that activity. The Details dialog displays the Name, Type, Status, Start Time, and End Time of the activity. If enabled, the Details dialog also enables you to browse the Published Data of the activity when it was executed. Use this information when troubleshooting your runbook.

During the development process, you can click the Test button in the toolbar of the Runbook Designer to launch the Runbook Testing Console to test runbook execution. You can step through the runbook activity-by-activity in the runbook Testing Console, manually advancing through the runbook activities and viewing detailed activity output. Keep in mind that the runbook Testing Console actually executes the runbook activities; this is not a theoretical execution. For example, if you test a runbook that deletes records from a database, those records are actually deleted!

Notes from the Field: A common issue encountered by Orchestrator users is a runbook that generates a different result in the Runbook Testing Console than returned when running the runbook from the Runbook Designer. This is usually explained by user security. Runbooks you run in the Runbook Designer run in the context of the Runbook Server service account. On the other hand, runbooks you run in the runbook Testing Console execute in the context of your current user credentials. If your user account does not have the same permissions as the Runbook Server service account, some runbooks that successfully run in the Runbook Designer may fail when run from the Runbook Testing Console.

Taking Automation to the next level with Orchestrator and System Center

The System Center IPs for Orchestrator deliver integration between System Center products for a variety of use case scenarios, including:

  • Incident Remediation (via Operations Manager 2007 R2)
  • Virtual Machine Provisioning (via Virtual Machine Manager 2008 R2)
  • Change Management and CMDB automation (via Service Manager 2010)
  • Backup and Recovery (via Data Protection Manager 2010)
  • Software and Update Distribution (via Configuration Manager 2007)

Note: The System Center 2012 IPs are currently in beta and available for download at http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=28725

For example, the runbook shown in Figure 5 interacts with Operations Manager 2007 R2 to automate incident remediation.

image

Figure 5 – Sample Incident Remediation runbook from Orchestrator 2012

Here is a high-level description of the steps performed in this runbook:

  • The Monitor Alert activity monitors for an alert titled ‘DHCP Service Stopped’.
  • The Get Service Status activity checks to verify the DHCP Client service is actually stopped.
  • If the service is running, a custom field on the Operations Manager alert is updated with a note indicating a false alarm.
  • If the service is stopped, the Start DHCP Service activity restarts the service.
  • Once the service is started, the alert is updated with a note that the service has been restarted by Orchestrator.

The activities from these IPs also can be used together to create advanced automation scenarios incorporating multiple System Center products in a single automation sequence. A copy of the sample runbook shown in Figure 5 is included in the code download included with this article.

Hands-On Lab

UPDATE MARCH 12, 2013: These labs are currently unavailable. I’ll update this post when new versions / replacements are online and available.

Now, you’re ready to apply what you’ve learned in the creation of more advanced Orchestrator runbooks. As with the hands-on labs in part 1, the runbooks in these labs will run equally well in Opalis and Orchestrator.

REMEMBER:

  • Authoring experience and interface in Opalis and Orchestrator is basically the same.
  • Policies in Opalis are what we now call Runbooks in Orchestrator….same concept, new terminology.
  • Foundation Objects are now called Standard Activities….same concept, new terminology.

Next Installment

That’s it for this installment. The third installment continues this discussion of Orchestrator fundamentals, including:

  • Complex Link Logic (Advanced Branching)
  • Bridging the Gaps with PowerShell
  • Extending Your Capabilities with Community-Developed IPs
  • Runbook Authoring Best Practices

Get Orchestrator Unleashed

Pete Zerger is an author for the upcoming System Center 2012 Orchestrator Unleashed (http://www.amazon.com/System-Center-2012-Orchestrator-Unleashed/dp/0672336103). This book is authored by an expert team of current and former Microsoft MVPs who also wrote the industry’s leading guide to Opalis 6.3, Orchestrator 2012’s predecessor product.

9 thoughts on “Orchestrator Runbook Automation JumpStart: Part 2 – Advanced Features and Functionality

  1. Scott

    I have a third party vendor that delivers a file to me at usually at 10PM.  I need to monitor that file and send messages to people if the file is late.  So for this example, I want to send an email to people at 10:30 that the file is not here yet and keep sending them emails say evert 1/2 hr till the file shows up.  Say the file shows up at 11:15 (or 2:00 AM).  I would send an email at 10:30 then at 11:00 and then start may processing at 11:15.  I was going try looping but there is does not appear to be a way to this.   Am I missing something here ?

  2. Scott

    Yes that is what I have done, but I would like to send emails during the looping process and prior to the activity completing.  I was wondering if there was a way to send an email after say every 4th  loop iteration.   Something like a do not exit and send email?

  3. Pete Zerger Post author

    To do that every fourth loop, you’d need to use a Counter (and the Update Counter activity) to increment the counter with each loop, e-mail when you hit 4 and then Update Counter to zero. Make sense?

  4. Scott

    What I ended up wanting to do create a ‘cycle’ error.  I had to have a parent runbook initiate a child runbook and than add the counter comment you made.

    One other unrelated question.  I have a folder structure for my runbooks/policies.  Getting the runbook name is a common variable.  How to I get the full folder path for the runbook so I can send it in an email on errors.

  5. Stuart Douglas

    I know of no way to include the folder information for where a runbook is located as subscribed data, perhaps Pete or one of the experts does. I would say that if you’re not already following an organizational hierarchy for your runbooks along the lines described in this blog post, I would encourage you to do so…it makes finding runbooks simplel the name tells you exactly where it is.

    http://blogs.technet.com/b/privatecloud/archive/2013/03/08/automation-system-center-2012-orchestrator-best-practice-series-naming-conventions.aspx

     

  6. Pete Zerger Post author

    You’d need to use a SQL query to get the path, but if you use best practices in naming and organizing your runbooks, I think doing what you are suggesting would be unnecessary. If you’re open to adopting a different folder / runbook naming convention and organization structure, I’d happily write a blog post to give you some pointers.

    Also suggest you see our coverage of this issue in our MMS 2013 session – http://channel9.msdn.com/Events/MMS/2013/SD-B317

  7. Scott

    We have been using Opalis for a very long time, we started about 5 years before MS purchased it, and have about 90 or so active policies running in about 30 policy sub levels i.e. Policies – top folder | 30 sub folders with some folders below the sub.  For example a structure is

    Policy
          LOB application
                  Daily
                          policies/runbooks here
                  Monthly

    Our usage is more line of business application oriented than monitoring system availability.  We use it to push files and reports around.

    We are moving to Orchestartor from Opalis 6.2.2 and would like to follow a series of best practices.  I liked the named heirachy described in the referenced blogs.  I want not wild about duplicating the folder structure as the prefix to the runbook name.  It seemed quite redundant and since the names are so long you see very few runbook tabs.  I understand you can click the tab arrow right and left but I’d rather see more and know exactly where to go vs. having to click and iterate through tabs.

  8. Pete Zerger Post author

    I use a combination of names and numbers in my naming conventions. For example:

    1. Monitoring Runbooks <– folder
    1.1. Service Manager Runbooks <- sub-folder
    1.1.1 Critical Incident Monitoring <- sub-sub folder
    1.2 Operations Manager Runbooks
    1.2.1 SCOM Maintenance Mode

    2. Ad-hoc Runbooks
    2.1 Service Manager Runbooks
    2.1.1. Self-Service Group Membership

    Why I do this is to ensure that I always know where runbooks belong even if a folder accidentally gets dragged and dropped somewhere else. I’ve had this happen a few times and because I had a numeric sequence incorporated into my naming conventions I always knew where it came from and where belong.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.