Alert: Power Shell script was dropped
Management Pack Name: System Center Core Monitoring
Management Pack Version: 7.0.9538.0
Rule or Monitor: Rule
Rule or Monitor Name: Alert on Dropped Power Shell Scripts
Rule or Monitor Notes: It’s a generic rule that alerts on a generic powershell workflows issues
Issue: The PowerShell script will be dropped because the it has been waiting in the queue for more than 10 minutes.
Script Name: PowerShellScriptName
One or more workflows were affected by this.
Workflow name: MyWorkflow.Name
Resolution: A bit of theory:
The Runspace Manager
In order to allow multiple Windows PowerShell scripts to run simultaneously, a mechanism known as a Runspace Manager is used. The Runspace Manager provides a runspace in an isolated application domain in which to run scripts. It implements a First In, First Out (FIFO) queue in which scripts are placed for execution, and handles limiting the number of scripts that are running at the same time.
By default, the maximum number of scripts that can be running in at a time is 20, and scripts in the runspace manager’s queue will expire and be removed after 10 minutes.
As you can see from the theory this alert could mean two things:
1. Too many powershell scripts are executed on agent simultaneously
2. There are scripts that takes much time to complete
First of all, you need to evaluate a script-based module (workflow). You need to find answers to following questions:
– What is the target for this script-based workflow?
– How many instances of this target class are exist on the affected computer?
– Does this script leveraging a cookdown concept?
– How long it take for script to complete?
The real world example:
Recently I’ve seen the ‘Power Shell script was dropped’ alert from a Hyper-V host. After a short investigation, I’ve found a script-based workflow that queries a Virtual Machine’s property. Answers:
– The target is the ‘Virtual Machine’ class
– More than 40 VMs are hosted by this Hyper-V server
– No, it does not because the workflow passes a VM name to the script.
– 20-40 seconds
So, we have found answers. What’s next?
First, try to fix the workflow. Yes, I know it could be a problem because you’ll need an authoring skills. However, that’s the most effective way. You can redesign the workflow (if it is in an unsealed management pack) or create your own workflow and disable the original one.
Some tips for a MP authors:
– Use a cookdown as much as possible
– Cookdown is not possible in some cases because of Health Service limits: maximum 128 dataitems per a workflow, maximum 5120 dataitems simultaneously in a HealthService queue (waiting to be processed by HealthService) and 4MBytes max for a single dataitem size.
– If you can do the same task in VBScript instead of PowerShell – do it, powershell modules in OpsMgr are resource-intensive and… a bit slow…
– IF you are a programmer it’s worth to check this: http://opsmgr2012sdk.codeplex.com/
Here is some tips for those who cannot develop their own workflows:
– If this workflow doesn’t use a cookdown – try to override an interval for a half of targets. Let’s say we have a workflow that executes a script for every of 40 VMs every 5 minutes. Override this interval to 6 or 7 minutes for 20 VMs. I know it is a bit of manual work that nobody likes, but this could be done by every OpsMgr admin.
– If this workflow doesn’t use a cookdown AND has the SyncTime parameter- try to override the SyncTime. It will keep the default interval (sometimes it’s important) and will spread the scripts over the time.
– The last method is the Runspace Manager tuning:
Here is the two keys you can use:
Key Description Default Value
ScriptLimit Controls how many hosted Windows PowerShell scripts are allowed to run globally. 0x00000014
QueueMinutes Defines how many minutes before a script expires from the queue. 0x0000000a
Be EXTREMELY careful with those keys and NEVER change it in a production environment without a good testing.
A few helpful links:
(Most of those links for OpsMgr 2007. OpsMgr 2012 uses the same concepts so almost all of those are applicable to 2012\2012 SP1)
PowerShell modules in OpsMgr: http://msdn.microsoft.com/en-us/library/ee809360.aspx