Performance Analysis with Performance CoPilot, iPython and pandas
Introduction
One of many reasons to love Performance CoPilot, is the fact that it is a fully fledged framework to do performance analysis. It makes it extremely simple to extend and to build anything on top of it. In this post we shall explore how simple it is to analyze your performance data using iPython and pandas.
Setup
To start we will need some PCP archives which contain some collected metrics from a system. In this post I will use the data I collect on my home firewall and will try to analyze some of the data there in. To learn how to store performance metrics in an archive, take a look at pmlogger and the Quickstart guide. For this example I collected data over the course of a day with a 1 minute interval.
iPython and PCP
First of all you need to import a small python module that bridges PCP and pandas/numpy:
git clone https://github.com/mbaldessari/pcpinteractive.git cd pcpinteractive
Now let us start our iPython console, import our python module and load our archive:
At this point the data is fully parsed in memory and we can start analyzing it, using all the standard tools like pandas and matplotlib. Let's start by looking at how many metrics are present in the archive:
Pandas and PCP
Now we can get a pandas object out of a metric. Let's take incoming and outgoing network traffic expressed in bytes over time.
We can now graph the data obtained with a simple:
And we can also explore the data with the use of the describe()
method, but first
let's force the output to be in nonscientific notation as it is more readable for
network data:
Manipulate the data
Now let's see what is possible to do in terms of data manipulation: * Drop columns we do not care about:
Or, alternatively:

Resample data at lower intervals

Filter out all the zero columns

Show last element:

Select a smaller timeframe:

Get one column with:

Apply a function on the whole dataframe:

Sum all values for each column:

Calculate the mean for each column:

Find the time of day when the max values are reached

Select only the tun0 and eth0 devices:
Merge and group dataframes
Now let's merge the net_in
and the net_out
dataframes into a single one, in order
to try and do some analysis on both traffic directions at the same time.
Another very interesting aspect is the plethora of statistical functions that come for free through the use of pandas. For example, to find covariance() and correlation() we can use the following methods:
We can also group columns like the following:
Calculate the rolling mean of an interface and plot it:
Export data
Save the data in csv file or in excel format:
Other outputs like latex, sql, clipboard, hd5f and more are supported.
Conclusions
The versatility of PCP allows anyone to use many currently available frameworks (numpy, pandas, R, scipy) to analyze and display the collected performance data. There is some work to be done to make this process a bit simpler with an out of the box PCP installation.