As discussed in Part 1, it is almost critical to properly configure AVIcode if you want to get the most of it. AVIcode is great for the production environments from troubleshooting and monitoring perspective, but it may take some time to fine-tune AVIcode based on the application code analysis and based on the data you’ll see in AVIcode once you have installed it.
However, it is sometimes difficult to ignore the desire to start monitoring just everything. It’s the easiest approach, and it can be implemented quickly. Just add “all namespaces”, add some of the system namespaces on top of that, and you’ll get enough details to satisfy any development team. The problem is that you’ll probably render your application unusable by doing that since users will have to spend minutes for what they could do in a just few seconds before that.
Using the same example as in Part 1, let’s assume that there is a huge XML file (I used 5 MB file) that has about 70000 lines. That might look like an extreme example, but it actually takes only about 5 seconds to run the following code against such a file:
Imagine that you decided to monitor System.Xml namespace in this situation – after all, this code sample does most of the work in that namespace, so you might want to know where exactly. On my workstation, I had to watch 100% CPU usage for almost 2.5 minutes before I got this code executed with that kind of monitoring settings.
So, that’s where the problem is. Out of the box, AVIcode does not collect a lot of troubleshooting data, yet it does not introduce significant overhead. It is not that difficult, though, to seriously damage application performance by applying a configuration change that does not suite your environment.
Most of those dangers are there because for every instrumented function AVIcode has to do some processing no matter if there is going to be some data captured eventually or not. For example, if you add a namespace, that will make AVIcode instrument all the functions in that namespace. And, whenever your application enters a function from that namespace, there will be a callback to AVIcode. Which may be ok or not, but, apparently, it was far from ok in my example with System.Xml.
AVIcode has a number of performance optimizations which may help to overcome such problems. It does have “event throttling” functionality which allows AVIcode to ignore similar events if they are raised too often. AVIcode can also start to “almost” ignore namespaces if it decides that the namespace is too noisy (meaning that there are just too many calls to the function from that namespace). That does not always help, though.
The only real solution to that is testing. Once you have a configuration that provides enough data, you have to run through performance tests to ensure that your application users will not suffer from performance overhead. In many cases, it may be sufficient to have some manual tests done, but that will not really allow you to justify your configuration settings if you are asked to provide objective performance comparison of your application behaviour with and without AVIcode. And that’s where application performance base lining can help.
That sort of base lining should not rely on AVIcode, of cause. If there are performance counters implemented in your application, you might probably use them. In general, though, you can come up with a good performance base line using a very basic set of performance counters:
- %Processor Time
- Private Bytes
- Virtual Bytes
For ASP.NET applications, you might also use
- Requests /sec
- Request Execution Time
- Request Wait Time
- Requests In Application Queue
– You might add additional counters, but, even with these ones, you should be able to quickly find out if AVIcode monitoring affects application performance more than necessary. Once you have a base line of your application performance without AVIcode, do the same performance tests with AVIcode and compare performance counters collected during that test with the base line. Of cause, you may have to start fine-tuning your AVIcode configuration settings based on the results of those test, but, at the very least, your application users won’t have to suffer from the potential performance issues that may occur otherwise.
About the author
Alex Shlega works as a Senior Technical Consultant at VIACode Inc. Prior to this, Alex worked as a sustain engineering team manager at AVIcode.
See Alex’s blog for more AVIcode-related posts: http://gotchahunter.net/category/avicode/