OnDemand Detection and Cook Down
| |
OnDemand Detection and Cook Down
Posted: Wed, Jul 22, 2009 6:53 PM :: Rank: 1 |
Author
|
|
|
Points: 1002
Level: System Center Specialist |
Thank you for your rating!
|
I am seeing a huge impact on HealthService performance when I add OnDemandDetection to an MP that discovers many instances of a given class. Any time the HealthService is reloaded, every instance spawns an independent ProbeAction (a script in my case which returns a multi instance property bag). That isn't a problem when only 2 or 3 instances exist but with 20 to 200 that stalls the agent. If the agent survives that initial load however, OnDemandDetection works as expected.
Is there any workaround for this or do we currently have to live with the fact that using OnDemandDetection is only good if very few instances are discovered? I am looking for a way to force all of these initial OnDemandDetections into a single call of the workflow. Just as I can achieve it with a SyncTime in the scheduler of the ConditionDetections' datasource.
Any thoughts?
Raphael
|
|
Reply
Report Abuse
| Your Reports Help Protect the Community |
|
The community depends on each member to help keep Answers a safe and positive place. Do your part by using the form below to report Q&A that violates the Community Guidelines.
|
Additional Detail(optional)
|
|
Report AbuseCancel
|
|
|
|
|
RE: OnDemand Detection and Cook Down
Posted: Thu, Jul 23, 2009 11:17 AM :: Rank: 0 |
Author
|
|
|
Points: 545
Level: System Center Hero |
Thank you for your rating!
|
First, there is a bug in SP1 where on demand triggering data source is not cooking down properly. This means you will have an on demand workflow started per state the monitor can change to so it is adding additional pressure to the runtime. Not sure if we are releasing fix, will look ...
Second, all releases up to date have this behavior, where all workflows related to monitor are started when monitor is registered for the first time. Regular detection workflows may cook-down if they use same data source module (meaning same configuration and such cook-down will possibly fork at some condition that decides what is monitor state). In R2, on demand cooks down data source correctly.
when you are using script as your probe in on demand detection, it gives you additional problem (even if proper cookdown is in place, at least two instances of script will run per instance and race to try to be the first to make a state change). This is actually a good test to see if your cookdown is working, have script raise an event for single instance case and count how many events you have.
but overall, you are correct, on demand gives you some burden, not sure if we are changing much in this space but I can surely try to file some work items for tracking. I guess we did not have much negative feedback which could be caused by the fact that even MS management packs are not using on demand detection a lot ...
|
|
Reply
Report Abuse
| Your Reports Help Protect the Community |
|
The community depends on each member to help keep Answers a safe and positive place. Do your part by using the form below to report Q&A that violates the Community Guidelines.
|
Additional Detail(optional)
|
|
Report AbuseCancel
|
|
|
|
|
RE: OnDemand Detection and Cook Down
Posted: Thu, Jul 23, 2009 1:02 PM :: Rank: 0 |
Author
|
|
|
Points: 1002
Level: System Center Specialist |
Thank you for your rating!
|
Marius
Many thanks for the detailed answer. I should have mentioned that I am using R2 and that I was aware of the race condition problem which exists in SP1.
Just a question though: You mention two scripts being fired simultaneously when the monitor is registered. You mean one for the ConditionDetection and the other one for OnDemandDetection?
My theory is that when using several instances, cook down for OnDemand is not going to work since the monitors are not registered at the exact same time for all the instances. The OnDemandModule is run immediately - and a ConditionDetection in parallel - and indepemdently for each instance. Right?
The ConditionDetection for the regular monitor states I configured with a schedule that has a SyncTime. Such they cook down perfectly as I can easily verify by an event written by the script. Three monitors and up to several hundred instances always just spawn a single script. If I omit the SyncTime however, then the same race condition happens. I believe the schedule's time counter for each instance of an object starts counting when the instance is loaded - and that doesn't seem to be on the same timetick for all of them. Am I more or less on the right track?
|
|
Reply
Report Abuse
| Your Reports Help Protect the Community |
|
The community depends on each member to help keep Answers a safe and positive place. Do your part by using the form below to report Q&A that violates the Community Guidelines.
|
Additional Detail(optional)
|
|
Report AbuseCancel
|
|
|
|
|
RE: OnDemand Detection and Cook Down
Posted: Thu, Jul 23, 2009 2:30 PM :: Rank: 0 |
Author
|
|
|
Points: 545
Level: System Center Hero |
Thank you for your rating!
|
I hope to communicate this clearly, cookdown is rather complex problem and I'm not sure if I can explain well as english is my second language (and as our MVP personnels could prove, I like to use a white board a lot in such cases :))
Cookdown is based on the hash that is calculated from configuration sent to the starting module (data source) as XML string, module type etc.
This means that as long as variable value is not present in configuration, there is a big chance for cookdown to happen as expected by the author.
The difference between regular detection and on demand detection in your case is "hidden" in the internal module we use to start on demand detection workflow. It is data source (although not visible thru MP) but unfortunately this data source takes instance ID (equal to target instance id) as one part of configuration. That makes cookdown happen within target but not accros all managed entity -> configuration is unique per instance. Your regular detection seems to avoid such troubles as appears having NO dynamic (or otherwise instance dependent values) present in configuration for data source nor for following probe module.
does it make sense what I just said?
|
|
Reply
Report Abuse
| Your Reports Help Protect the Community |
|
The community depends on each member to help keep Answers a safe and positive place. Do your part by using the form below to report Q&A that violates the Community Guidelines.
|
Additional Detail(optional)
|
|
Report AbuseCancel
|
|
|
|
|
RE: OnDemand Detection and Cook Down
Posted: Fri, Jul 24, 2009 8:58 PM :: Rank: 1 |
Author
|
|
|
Points: 20757
Level: System Center Expert |
Thank you for your rating!
|
Marius, I must confess I do not quite follow. It may well be a lack of understanding of OnDemand Detection on my part (and not your English). Let me repeat what I think you are saying.
Are you saying with OnDemand Detection we do not have the same amount of control as in regular detection?
When you say ". That makes cookdown happen within target but not accros all managed entity -> configuration is unique per instance."....are you saying that if I was using output of a Data Source for multiple rules or monitors targeting an single instance it would cook down, but not in multi-instance scenarios?
If the above is correct, should I avoid configuring on demand detection for multi-instance monitoring situations?
I also wanted to say it is really great that you take time to speak with us like this. I definitely appreciate access to knowledgeable people like yourself.
|
|
Reply
Report Abuse
| Your Reports Help Protect the Community |
|
The community depends on each member to help keep Answers a safe and positive place. Do your part by using the form below to report Q&A that violates the Community Guidelines.
|
Additional Detail(optional)
|
|
Report AbuseCancel
|
|
|
|
|
RE: OnDemand Detection and Cook Down
Posted: Mon, Jul 27, 2009 4:50 PM :: Rank: 1 |
Author
|
|
|
Points: 545
Level: System Center Hero |
Thank you for your rating!
|
yes, I' saying that you do not have much control on how cookdown will be perfromed because on demand detection uses data source module (start module for on demand detection workflow) which is not cooked down accross entity but only for instance (where entity is a type and instance is instance of such type). So regardless that your "trigger only" probe is cooked down (or would be cooked down), fork happenned already with data source module which preceds this module in your monitor definition.
having multi instance scenario, there might be cases I would try to avoid on demand detection. Such cases would include script base probe (where script is not instance dependant but could for example just read computer registry etc) because such script will be executed for each instance ...
|
|
Reply
Report Abuse
| Your Reports Help Protect the Community |
|
The community depends on each member to help keep Answers a safe and positive place. Do your part by using the form below to report Q&A that violates the Community Guidelines.
|
Additional Detail(optional)
|
|
Report AbuseCancel
|
|
|
|
|
RE: OnDemand Detection and Cook Down
Posted: Sun, Aug 09, 2009 8:12 PM :: Rank: 1 |
Author
|
|
|
Points: 29130
Level: System Center Expert |
Thank you for your rating!
|
|
|
Reply
Report Abuse
| Your Reports Help Protect the Community |
|
The community depends on each member to help keep Answers a safe and positive place. Do your part by using the form below to report Q&A that violates the Community Guidelines.
|
Additional Detail(optional)
|
|
Report AbuseCancel
|
|
|
|
|
|
|
|
Quick Links
System Center Web sites
3rd Party / Partner Resources
Other System Center Resources
|
|
|
|
|
Top Contributors
|
|
Points: 29130
Level: System Center Expert
|
|
|
Points: 24529
Level: System Center Expert
|
|
|
Points: 20757
Level: System Center Expert
|
|
|
Points: 12514
Level: System Center Expert
|
|
|
Points: 11676
Level: System Center Expert
|
|
|
|
|
|
|
|
|
|
|
|