OnDemand Detection and Cook Down
Home  » MP Development  » OnDemand Detection and Cook Down

OnDemand Detection and Cook Down
Posted: Wed, Jul 22, 2009 6:53 PM :: Rank: 1
Author
Points: 1002
Level: System Center Specialist

I am seeing a huge impact on HealthService performance when I add OnDemandDetection to an MP that discovers many instances of a given class. Any time the HealthService is reloaded, every instance spawns an independent ProbeAction (a script in my case which returns a multi instance property bag). That isn't a problem when only 2 or 3 instances exist but with 20 to 200 that stalls the agent. If the agent survives that initial load however, OnDemandDetection works as expected.

Is there any workaround for this or do we currently have to live with the fact that using OnDemandDetection is only good if very few instances are discovered? I am looking for a way to force all of these initial OnDemandDetections into a single call of the workflow. Just as I can achieve it with a SyncTime in the scheduler of the ConditionDetections' datasource.

Any thoughts?

Raphael

Reply  
Report Abuse
RE: OnDemand Detection and Cook Down
Posted: Thu, Jul 23, 2009 11:17 AM :: Rank: 0
Author
Points: 545
Level: System Center Hero

First, there is a bug in SP1 where on demand triggering data source is not cooking down properly. This means you will have an on demand workflow started per state the monitor can change to so it is adding additional pressure to the runtime. Not sure if we are releasing fix, will look ...

Second, all releases up to date have this behavior, where all workflows related to monitor are started when monitor is registered for the first time. Regular detection workflows may cook-down if they use same data source module (meaning same configuration and such cook-down will possibly fork at some condition that decides what is monitor state). In R2, on demand cooks down data source correctly.

when you are using script as your probe in on demand detection, it gives you additional problem (even if proper cookdown is in place, at least two instances of script will run per instance and race to try to be the first to make a state change). This is actually a good test to see if your cookdown is working, have script raise an event for single instance case and count how many events you have.

but overall, you are correct, on demand gives you some burden, not sure if we are changing much in this space but I can surely try to file some work items for tracking. I guess we did not have much negative feedback which could be caused by the fact that even MS management packs are not using on demand detection a lot ...

Accepted Answer
Reply  
Report Abuse
RE: OnDemand Detection and Cook Down
Posted: Thu, Jul 23, 2009 1:02 PM :: Rank: 0
Author
Points: 1002
Level: System Center Specialist

Marius

Many thanks for the detailed answer. I should have mentioned that I am using R2 and that I was aware of the race condition problem which exists in SP1.

Just a question though: You mention two scripts being fired simultaneously when the monitor is registered. You mean one for the ConditionDetection and the other one for OnDemandDetection?

My theory is that when using several instances, cook down for OnDemand is not going to work since the monitors are not registered at the exact same time for all the instances. The OnDemandModule is run immediately - and a ConditionDetection in parallel - and indepemdently for each instance. Right?

The ConditionDetection for the regular monitor states I configured with a schedule that has a SyncTime. Such they cook down perfectly as I can easily verify by an event written by the script. Three monitors and up to several hundred instances always just spawn a single script. If I omit the SyncTime however, then the same race condition happens. I believe the schedule's time counter for each instance of an object starts counting when the instance is loaded - and that doesn't seem to be on the same timetick for all of them. Am I more or less on the right track?

Reply  
Report Abuse
RE: OnDemand Detection and Cook Down
Posted: Thu, Jul 23, 2009 2:30 PM :: Rank: 0
Author
Points: 545
Level: System Center Hero

I hope to communicate this clearly, cookdown is rather complex problem and I'm not sure if I can explain well as english is my second language (and as our MVP personnels could prove, I like to use a white board a lot in such cases :))

Cookdown is based on the hash that is calculated from configuration sent to the starting module (data source) as XML string, module type etc.

This means that as long as variable value is not present in configuration, there is a big chance for cookdown to happen as expected by the author.

The difference between regular detection and on demand detection in your case is "hidden" in the internal module we use to start on demand detection workflow. It is data source (although not visible thru MP) but unfortunately this data source takes instance ID (equal to target instance id) as one part of configuration. That makes cookdown happen within target but not accros all managed entity -> configuration is unique per instance. Your regular detection seems to avoid such troubles as appears having NO dynamic (or otherwise instance dependent values) present in configuration for data source nor for following probe module.

does it make sense what I just said?

Reply  
Report Abuse
RE: OnDemand Detection and Cook Down
Posted: Fri, Jul 24, 2009 8:58 PM :: Rank: 1
Author
Points: 20757
Level: System Center Expert

Marius, I must confess I do not quite follow. It may well be a lack of understanding of OnDemand Detection on my part (and not your English). Let me repeat what I think you are saying.

Are you saying with OnDemand Detection we do not have the same amount of control as in regular detection?

When you  say ". That makes cookdown happen within target but not accros all managed entity -> configuration is unique per instance."....are you saying that if I was using output of a Data Source for multiple rules or monitors targeting an single instance it would cook down, but not in multi-instance scenarios?

If the above is correct, should I avoid configuring on demand detection for  multi-instance monitoring situations?

I also wanted to say it is really great that you take time to speak with us like this. I definitely appreciate access to knowledgeable people like yourself.

Reply  
Report Abuse
RE: OnDemand Detection and Cook Down
Posted: Mon, Jul 27, 2009 4:50 PM :: Rank: 1
Author
Points: 545
Level: System Center Hero

yes, I' saying that you do not have much control on how cookdown will be perfromed because on demand detection uses data source module (start module for on demand detection workflow) which is not cooked down accross entity but only for instance (where entity is a type and instance is instance of such type). So regardless that your "trigger only" probe is cooked down (or would be cooked down), fork happenned already with data source module which preceds this module in your monitor definition.

having multi instance scenario, there might be cases I would try to avoid on demand detection. Such cases would include script base probe (where script is not instance dependant but could for example just read computer registry etc) because such script will be executed for each instance ...

 

Reply  
Report Abuse
RE: OnDemand Detection and Cook Down
Posted: Sun, Aug 09, 2009 8:12 PM :: Rank: 1
Author
Points: 29130
Level: System Center Expert

 Marius, I believe I have tested and observed successful cookdown of a script (in single data source module) outputting data to two workflows...a collection rule and a two-state monitor. 

Is Daniele correct in his assertion that this may not actually be so?

http://nocentdocent.wordpress.com/2009/08/08/on-demand-detections-caveats-a-cookdown-story/

Reply  
Report Abuse

Home  » MP Development  » OnDemand Detection and Cook Down
Tag Cloud
Quick Links
Top Contributors
Pete Zerger
Points: 29130
Level: System Center Expert
Simon Skinner
Points: 24529
Level: System Center Expert
Tommy Gunn
Points: 20757
Level: System Center Expert
Stefan Koell
Points: 12514
Level: System Center Expert
MadHatter
Points: 11676
Level: System Center Expert