Generic Performance Data Connector using .csv files







Generic Performance Data Connector

Using Character-Separated Input Files

By Andreas “Tenchuu” Zuckerhut

 

Preface

WARNING!!! Use this at your own risk. I will not take responsibility for any damage this Application may cause. Use it only in Lab Environments.

Also, this page looks just fine in Firefox, but not in Internet Explorer for some reason. It’s a .net Nuke Fail I guess.

I recently wanted to import Performance Data into Operations Manager using a .csv file, which was a few hours of figuring out how to do it, coding it, and debugging it.

And just like everything you face in your daily business, once you know how to do it, it’s pretty easy.

So, I thought, why should I not reinvent my little tool making it generic so others could benefit from my experience as well. Even though it took me quite a few hours of my spare time, I think the result is worth it.

So now I can proudly present my Generic Performance Connector using Inputfiles that takes 11 Arguments. Yea, I know it’s not user-friendly but still better than you having to develop it yourself 😉

So, this is my first attempt on releasing something I can call an Application, therefore I have not much experience on documenting something like this. If something confuses you, if you have any suggestions, feature requests, if you run into errors or need help on the configuration, send me an E-Mail: andreas.zuckerhut(at)gmail.com

Please also tell me if you are using it as I’d really like to get some Feedback on it as this is the only payment I will get from it.

 

General Information

Something technical at first, a Performance Data Value expects the following:

          A MonitoringObject

          An Object

          An Instance (which is optional)

          A CounterName

          A Value

          A DateTime (which is somewhat optional)

 

The MonitoringObject:

Like a Windows Computer is discovered, we will just discover a MonitoringObject ourselves or you can of course provide the information about an existing one. In other words, it’s an Instance.

 

An Object:

Like a CPU, it doesn’t matter if this exists or not, but we need to provide a Name for it.

 

An Instance:

Like a Core. As it is currently, providing the InstanceName is pretty redundant because the CounterName will have the Information about the Instance in all the cases I’ve seen so far. If you feel like I should implement this, please tell me how you’d like it.

 

The CounterName:

Like CPU Usage in %.

A Value:

The Value that is attached to the CounterName.

 

A DateTime:

The Time a Value was taken.

 

In the Console this looks like this:

 

Configuration

 

Configuring the Parameters:

The tool takes all the information using CommandLine Parameters which is maybe the most efficient way, especially when you plan to run it on a scheduled basis.

Below, you’ll find all the Parameters you need to provide including examples. Besides the obvious I had to implement additional parameters.

Any optional Parameter can be “” which is equal to nothing.

 

InputFile

This, of course is the file we are going to read during the import. Just provide the full path here. Example: “C:\dataset.csv”

 

ColumnSeparator

It tells the application how to split each line’s content. In usual case this is either a Comma or a Semicolon. Example: “;”

 

MonitoringObjectName

I made this variable. In case you want to attach the Data to a new MonitoringObject either use a custom name for it. Example: “Tenchuu”

 

Or you could provide a field in the Input-File. What this does is that it takes the Value of Column x of each Datarow and uses that Name for the MonitoringObject. Example: “5”

 

You could also provide an existing Object. In this case you need to provide the Management Pack, the Class and the Name of the Instance and split it up using “!”.

Example: A Windows Machine named “EVA01.Tenchuu.com” would look like: “Microsoft.Windows.Computer!EVA01.Tenchuu.com”

 

ObjectName

I made this variable. Either provide a Name. Example: “CPU”

 

Or you could provide a field in the Input-File. What this does is that it takes the Value of Column x of each Datarow and uses that Name for the MonitoringObject. Example: “5”

 

InstanceName

Not implemented yet as I don’t see a purpose for it so far. But you still need to provide it, at least as an empty Value. Example: “”

 

CounterName

The CounterName is based on the Column in which a Value stands. If you need to change it, change it in the Input File.

 

ValuePosition

Defines in which Column the Data Values are stored. This can be a list of course splitted by “!”. Example: “1!2!3!4!5”

 

DateTimePosition

The position of the DateTime Values in the Input File. At this point this can only be one Column since I didn’t see anything that would need more than one. Example: “3”

 

DateTimeFormat

The DateTime Format is the most critical part of the configuration. I figured that the most effective way to make this optional is to use the DateTime.ParseExact function (did it with dynamic Regex before which was a pretty ugly and complicated piece of code).

 

If you can’t figure how to translate your DateTime into the right Formatstring just send me an E-Mail: andreas.zuckerhut(at)gmail.com

A full list Documentation on this can be found here:

http://msdn.microsoft.com/en-gb/library/8kb3ddd4(VS.80).aspx

 

Basically we have:

Part                 Without leading Zero             With leading Zero

Month             M(1)                                       MM(01)

Day                 d                                             dd

Hour                h                                             hh       

Hour(24)         H                                             HH

Minute            m                                            mm    

Second            s                                              ss

 

AM/PM           t(A or P)                                  tt(AM or PM)

Year                y(09)                                       yy(09)                         yyyy(2009)

 

Example:

DateTime String: “10/1/2009 4:00:00 AM”

DateTime Format String: “M/d/yyyy h:m:s tt”

 

DateTime String: “10/1/2009 19:23:14.567”

DateTime Format String: “M/d/yyyy H:m:s.fff”

 

Special situation with the DateTime Format. I had a pretty stupid .csv file that had 2 different DateTime Formats, in case it was Midnight it used “10/1/2009” so the parsing failed.

Therefore you can provide more than one DateTime FormatString (splitting them with “!” again), the application will try which one works.

 

Example:

DateTime String: “10/1/2009 4:00:00 AM” and “10/1/2009”

DateTime Format String: “M/d/yyyy h:m:s tt!M/d/yyyy”

 

DecimalSymbol

Performance Data Values of the SDK are Double Values and the Decimal Symbol of C# applications rely on the local settings. This can lead to weird conversions.

Therefore provide the Decimal Symbol that is used in the Input File, the application will translate it so it’s used correctly.

Example: “,”

 

DigitGroupingSymbol

[OPTIONAL] If your Input File uses one, provide it, it’ll be removed since it serves absolutely no purpose.

Example: “.”

 

WaitForVerification

Unless you set it to false it is true for security reasons. It will pause the Application after each step and will wait for User Input to continue. Only set this to false if it’s really necessary.

If it’s false it will not pause and will continue.

 

Commandline put together:

.exe “InputFile” “ColumnSeparator” “MonitoringObjectName” “ObjectName” “InstanceName” “ValuePosition” “DateTimePosition” “DateTimeFormat” “DecimalSymbol” “DigitGroupingSymbol”

 

Program Logic

 

Each function returns some Output for Debugging and Logging purposes. If you want to redirect the Output into a file add >> “FullPathToFile” at the end of the Command.

Example:  “C:\data.csv” “,” “TestInstance” “TestObject” “” “2!3!4!5!6!7” “1” “M/d/yyyy H:m:s.fff” “.” “” “true”>> “C:\logfile.txt”

 

The Application takes the the Arguments that were provided, checks if they are valid and inserts them into static Variables that will be used throughout the entire Application. If there’s an error during the verification of the Arguments, the Application will exit.

 

It will then insert the Data from the Input File into a DataTable. Every Value will get the Quotes (“) removed since I didn’t see any purpose for them, also if there are empty Values like “” or “ “ they will be replaced with a “0”, otherwise the Verify will fail later. It will also run some checks, if they fail the Application will exit.

 

The DataTable will then be tested against the DateTimeFormat, DateTimePosition, DecimalSymbol and DigitGroupingSymbol.

It will output a Conversion Sample and if WaitForVerification is true it will pause after the check of the first Row and after the last Row.

 

The next step is the creation of a SCOM Connector with ID “7E0C0007-7E0C-0007-7E0C-00077E0C0007” as long as it doesn’t exist already.

Description = “This is an Inbound Connector for Generic Performance Data”

DisplayName = “Generic Performance Data Connector.”;

Name        = “Generic Performance Data Connector.”;

 

Once the Connector is up and initialized, the MonitoringObjects will either be discovered on the fly or their existence will be checked if you provided your own Class and Name for it. It will output the DisplayNames of the MonitoringObjects that are going to be discovered including a Warning that after you hit Enter it will insert them into Operations Manager and that it will start importing the PerformanceData.

 

At this point nothing should go wrong and it imports the PerformanceData into Operations Manager. If it fails at this point please inform me about it: andreas.zuckerhut(at)gmail.com

 

Practical Example

Configuring the Arguments

I tried to make this as fool-proof as possible therefore there may be redundant information here.

 

I personally started with this using .csv files from Gomez, but this is really not a good example therefore I’ll just use a .csv file that was created by Perfmon.

 

I just put the file data.csv to C:\ so my InputFile Parameter will be: “C:\data.csv”

So I open my .csv file (with Notepad, not with Excel as Excel starts converting Values and DateTimes) and identify the ColumnSeparator at first. I’ve highlighted it in the Screenshot above even though it should be pretty obvious.

As the ColumnSeparator is the second Argument my Arguments now will be: “C:\data.csv” “,”

 

Next Is the MonitoringObjectName. I will now use “TestInstance” so I can watch the Data afterwards and if something didn’t work out as I expected I can just delete my Management Pack again and get rid of the PerformanceData.

 

For this I will also have to import the Generic.xml ManagementPack that I provided in the Archive as it holds the Class for the Generic.PerformanceConnector Class we are using for Custom MonitoringObjects.

 

So my Arguments now are: “C:\data.csv” “,” “TestInstance”

 

As I don’t have ObjectNames in the Rows I’ll just use “TestObject”. My new Arguments are: “C:\data.csv” “,” “TestInstance” “TestObject”

 

As we don’t use the InstanceName we’ll just provide an empty Argument “” so my Arguments are: “C:\data.csv” “,” “TestInstance” “TestObject” “”

 

The CounterName is not an Argument as it will use each Value’s ColumnName. Therefore I want to fix my .csv file first and remove the Information I don’t like from the Columns.

 

The ValuePosition is actually a list of positions. The first ValuePosition I have in my .csv file is “2” which is “Total %Processor Time”. But, I also have PerformanceValues in 3, 4, 5, 6 and 7. Therefore I’ll have “2!3!4!5!6!7”

So my Arguments now are: “C:\data.csv” “,” “TestInstance” “TestObject” “” “2!3!4!5!6!7”

 

My DateTimePosition is 1 so I add “1”. And, I need to translate the DateTime into a Format String. Let’s have a closer look again.

So I have “Month/Day/Year Hour:Minute:Second.Milisecond”

And according to http://msdn.microsoft.com/en-gb/library/8kb3ddd4(VS.80).aspx this translates to “M/d/yyyy h:m:s.fff”, at least that’s what I thought.

Actually, we have a 19 there which means that this is a 24-hours time and these translate to “H” and not “h”. Therefore the correct one is “M/d/yyyy H:m:s.fff”

 

So my Arguments now are: “C:\data.csv” “,” “TestInstance” “TestObject” “” “2!3!4!5!6!7” “1” “M/d/yyyy H:m:s.fff”

 

The DecimalSymbol is a dot. Therefore “.” And my Arguments now are: “C:\data.csv” “,” “TestInstance” “TestObject” “” “2!3!4!5!6!7” “1” “M/d/yyyy H:m:s.fff” “.”

As I don’t have a DigitGroupingSymbol I’ll use an empty string “”.

My Arguments now are: “C:\data.csv” “,” “TestInstance” “TestObject” “” “2!3!4!5!6!7” “1” “M/d/yyyy H:m:s.fff” “.” “”

 

And finally, do I run it as a Scheduled Task? No, therefore I will set WaitForVerification to “true”.

 

And my final Arguments are:

“C:\data.csv” “,” “TestInstance” “TestObject” “” “2!3!4!5!6!7” “1” “M/d/yyyy H:m:s.fff” “.” “”

 

Final Preparations

I connect to the RMS now as my Connector will connect to localhost and if I didn’t do it by now, I’ll import the ManagementPack “Generic.xml”.

 

Once finished I’ll make sure again that my data.csv is under c: and I’ll open an elevated Command Prompt locating my Application:

 

…and inserting the Arguments. IMPORTANT: Do never prepare your Arguments in Word or copy and paste them from Outlook as the Quotes are not compatible and they’ll get recognized as string-parts. You won’t see any difference in the Command Prompt though and you’ll sit there totally confused.

 

So, what I do is checking it again in Notepad and run Replaces:

“C:\data.csv” “,” “TestInstance” “””2!3!4!5!6!7” “1” “M/d/yyyy H:m:s.fff” “.” “” “true”

 

Afterwards I paste them into the Command Prompt next to my .exe

 

 

Fire

Once started, I get a Configuration Summary and the Information on the Column Names and how my lines were split. (Or an ErrorInformation)

I’ll verify that information and hit Enter to continue.

 

I will then see a few lines with values, this is actually just there so I know that the Application is doing something. It’s also good for debugging.

 

What is interesting then is this block:

It shows how the DateTime and the Values will be converted. I had problems with my first attempts with the DecimalSymbol being mistaken for a DigitGroupingSymbol and “0,324” suddenly was “324.000”

Since this looks just perfect I hit Enter to Continue. It will then start trying to Convert every Value and DateTime to verify that the Data is valid.

 

If the Verification is fine, I’ll get here:

 

I hit Enter, and the Discovery of my MonitoringObjects will begin, in case it’s needed. Otherwise I will just get the Information that the Class and Name I provided is right (or wrong).

Afterwards it’ll output information on the new MonitoringObjects and a decent Warning that if I hit Enter that the DiscoveryData will be added to Operations Manager and that the actual Import of the PerformanceData will start.

 

Before this point, nothing (known) bad thing can happen. Afterwards, there’s no turning back.

 

Since I’m pretty fine with this I hit Enter and see that it’s doing something.

 

Once it is finished, I want to check out if my Data is actually there. For this I open the Operations Console, go to the Monitoring Tab, go to “Discovered Inventory” and change the Target to “Generic PerformanceConnector”

 

What I should see then is my MonitoringObject I just discovered, and of course I want to open the Performance View to check if my Data is actually there:

 

And since I just inserted Values with a 1-minute stepping I have to adjust my TimeRange so I can actually see something:

 

 

Don’t panic if the Data is not visible instantly, it may take a few minutes because Operations Manager has to do some calculations as well.

 

And that’s pretty much it.

 

Attachment

 

In the Attachment you’ll find the files needed to run this Application and also a copy of this documentation as well as the Sourcecode. I release this under some license where you are not allowed use this to make profit. 😉

 

C# Projectfiles

Visual Studio 2008 Project Files.

Executables

The Application itself including the Libraries and the .pdb file that provides specific Error Information. (Yea, my Application is so cool it even has Unexpected Errors.)

Documentation

This Documentation.

Generic.xml

The ManagementPack needed.

0 thoughts on “Generic Performance Data Connector using .csv files

  1. Andreas Zuckerhut Post author

    Which part of it 😉
    SDK of course, I didn’t really want to go too deep into the Program Logic as it is not that impressive I think. I just use a lot of ifs and elses to cover all the possible configurations I could think of. Took me quite a few hours of coding though.

  2. Pete Zerger

    Well done, your next bottle of whiskey is on me.

    As for Stefan, he gets coffee for no man! But he will setup the Coffee Management Pack to be sure you don’t run out. 😉

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.