Website Logo. Upload to /source/logo.png ; disable in /source/_includes/logo.html

Michele’s Blog

C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D

Performance Analysis With Performance Co-Pilot, iPython and Pandas


One of many reasons to love Performance Co-Pilot, is the fact that it is a fully fledged framework to do performance analysis. It makes it extremely simple to extend and to build anything on top of it. In this post we shall explore how simple it is to analyze your performance data using iPython and pandas.



To start we will need some PCP archives which contain some collected metrics from a system. In this post I will use the data I collect on my home firewall and will try to analyze some of the data there in. To learn how to store performance metrics in an archive, take a look at pmlogger and the Quickstart guide. For this example I collected data over the course of a day with a 1 minute interval.

iPython and PCP

First of all you need to import a small python module that bridges PCP and pandas/numpy:

git clone
cd pcpinteractive

Now let us start our iPython console, import our python module and load our archive:

$ ipython qtconsole --pylab=inline
In [1]: import pcparchive, pandas
In [2]: p = pcparchive.PcpArchive('testfiles/monday')
In [3]: p.parse()
Parsing archive: [########## 100.0%]

At this point the data is fully parsed in memory and we can start analyzing it, using all the standard tools like pandas and matplotlib. Let’s start by looking at how many metrics are present in the archive:

In [4]: metrics = p.get_metrics()
In [5]: len(metrics)
Out[5]: 253

Pandas and PCP

Now we can get a pandas object out of a metric. Let’s take incoming and outgoing network traffic expressed in bytes over time.

In [6]: net_in = p.get_pandas('')
In [7]: net_out = p.get_pandas('network.interface.out.bytes')

We can now graph the data obtained with a simple:

In [8]: net_in.plot()


And we can also explore the data with the use of the describe() method, but first let’s force the output to be in non-scientific notation as it is more readable for network data:

In [9]: pandas.set_option('display.float_format', lambda x: '%.2f' % x)
In [10]: net_in.describe()
                eth0    eth1    eth2            eth3             lo
count        1439.00 1439.00 1439.00         1439.00        1439.00   
mean  23209831415.26    0.00    0.00 257865786302.88 32743773553.20   
std     133904051.55    0.00    0.00   1677032669.74      887369.78   
min   23074135536.00    0.00    0.00 256117467223.00 32742236187.00   
25%   23077865299.00    0.00    0.00 256128768785.50 32743005086.50   
50%   23160237636.00    0.00    0.00 257410715707.00 32743771068.00   
75%   23319280110.50    0.00    0.00 259252325129.50 32744537961.50   
max   23470418350.00    0.00    0.00 260826247120.00 32745311229.00   

                 ppp0       redhat0           tun0  
count         1439.00       1439.00        1439.00  
mean  253307755506.55 6897358284.79 12675081585.52  
std     1648648131.02  163459557.15        8568.61  
min   251589569686.00 6726948179.00 12675070001.00  
25%   251600158007.50 6727362445.50 12675070001.00  
50%   252860366375.00 6859405494.00 12675088375.00  
75%   254670529194.50 7051678675.50 12675088375.00  
max   256217896469.00 7171608913.00 12675088375.00

Manipulate the data

Now let’s see what is possible to do in terms of data manipulation: * Drop columns we do not care about:

In [12]: del net_in['eth1']

Or, alternatively:

In [13]: net_in = net_in.drop('eth2', 1)
  • Resample data at lower intervals
In [14]: net.resample('10min')
  • Filter out all the zero columns
In [15]: net_clean = net.ix[:,(net != 0).any(axis=0)]
  • Show last element:
In [16]: net_clean.ix[-1]
  • Select a smaller timeframe:
In [17]: net_clean['2014-12-28 12:00':'2014-12-28 14:00']
  • Get one column with:
In [18]: net_in.eth4
In [19]: net_in['lo']
  • Apply a function on the whole dataframe:
In [20]: net_in * 5
  • Sum all values for each column:
In [21]: net_in.sum()
eth0        33398947406561
eth3       371068866489847
lo          47118290143065
ppp0       364509860173927
redhat0      9925298571818
tun0        18239442401565
dtype: int64
  • Calculate the mean for each column:
In [23]: net_in.mean()
  • Find the time of day when the max values are reached
In [24]: net_in.idxmax()
In [25]: net_in.idxmax(axis=1)
  • Select only the tun0 and eth0 devices:
In [26]: test = net_in.ix[:, ['tun0', 'eth0']]

Merge and group dataframes

Now let’s merge the net_in and the net_out dataframes into a single one, in order to try and do some analysis on both traffic directions at the same time.

In [27]: net = pandas.merge(net_in, net_out, left_index=True, right_index=True, how='left', suffixes=('_in', '_out'))

Another very interesting aspect is the plethora of statistical functions that come for free through the use of pandas. For example, to find covariance() and correlation() we can use the following methods:

In [28]: net.cov()
In [29]: net.corr()
In [30]: net.corrwith(net.eth0_in)
Out [31]:
eth0_in       1.000
eth3_in       0.993
lo_in         0.958
ppp0_in       0.993
redhat0_in    0.991
tun0_in       0.748
eth0_out      0.994
eth3_out      0.986
lo_out        0.958
ppp0_out      0.983
redhat0_out   0.978
tun0_out      0.748

We can also group columns like the following:

In [32]: mapping = { 'eth0_in': 'in', 'eth3_in': 'in', 'ppp0_in': 'in', 'tun0_in': 'in',
                     'redhat0_in': 'in', 'lo_in': 'in', 'eth0_out': 'out', 'eth3_out': 'out',
                     'ppp0_out': 'out', 'tun0_out': 'out', 'redhat0_out': 'out', 'lo_out': 'out'}
In [33]: test = net.groupby(mapping, axis=1)
In [34]: test.sum()

Calculate the rolling mean of an interface and plot it:

In [35]: pandas.rolling_mean(net.eth0_in, 250).plot()


Export data

Save the data in csv file or in excel format:

In [36]: net.to_csv('testfiles/out.csv')
In [37]: net.to_excel('testfiles/out.xls')

Other outputs like latex, sql, clipboard, hd5f and more are supported.


The versatility of PCP allows anyone to use many currently available frameworks (numpy, pandas, R, scipy) to analyze and display the collected performance data. There is some work to be done to make this process a bit simpler with an out of the box PCP installation.

Observing X11 Protocol Differences

I was trying to understand some oddities going with an X11 legacy application showing bad artifacts in one environment and working flawlessly in another environment. Since wireshark does not have any support for diffing two pcaps, I came up with the following steps:

  • Dump both working.pcap and nonworking.pcap into text files with full headers:
~/Devel/wireshark/tshark -r working.pcap -T text -V -Y "x11" > working.txt
~/Devel/wireshark/tshark -r notworking.pcap -T text -V -Y "x11" > notworking.txt
  • Prune the two text files with a script like the following:
def clean(source, dest):
    f = open(source, "r")
    x11_state = False
    output = []
    for line in f.readlines():
        if x11_state:
            if line.startswith(' '):
                x11_state = False
            if line.startswith('X11') and \
                    "Property" in line:
                x11_state = True
    o = open(dest, "w")
    for i in output:

if __name__ == '__main__':
    clean("working.txt", "working-trimmed.txt")
    clean("notworking.txt", "notworking-trimmed.txt")
  • At this point we can easily vimdiff the outputs obtained from above via vimdiff working-trimmed.txt notworking-trimmed.txt: vimdiff

Performance Co-Pilot and Arduino (Part 2)

After initially setting up Performance Co-Pilot and Arduino, I wanted to improve the data being displayed. As latency is quite important to me, I wanted to display that as well. I did not have too much time on my hands to code a new PMDA that collects that information, so I abused the pmdashping(1) for this purpose. The steps are simple:

  1. Go to /var/lib/pcp/pmdas/shping
  2. Create sample.conf with the following line:
1 /var/lib/pcp/pmdas/shping/
  1. Create /var/lib/pcp/pmdas/shping/
# This hack will break if latency > 254 ;)
ret=`ping -c 2 -n -q | grep ^rtt | cut -f5 -d\/ | cut -f1 -d\.`
exit $ret
  1. Launch ./Install and choose [2] ./sample.conf
  2. Now it is possible to abuse the shping.error metric to fetch that value:
$ pminfo -f shping.error
   inst [0 or ""] value 52

The last step was to fetch this via PMWEBAPI(3). This did not work until I realized, thanks to Fche’s suggestion that the issue was related to my inital context initialization. As a matter of fact there is a big difference between the following two:

  • /pmapi/context?local=ANYTHING – Creates a PM_CONTEXT_LOCAL PMAPI context.
  • /pmapi/context?hostname=STRING – Creates a PM_CONTEXT_HOST PMAPI context with the given host name and/or extended specification.

The man page of pmNewContext(3) explains this in more detail. Frank has added some more info to the PMWEBAPI(3) man page via the following commit, to make it a little bit more obvious. It’s still a pretty gross hack, but for the time being it’s enough for my needs.


Performance Co-Pilot and Arduino

Besides being an incredible nice toolkit to work with, Performance Co-Pilot is extremely simple to integrate with any application. Amongst an extensive API it allows to export any metric via JSON using the PMWEBAPI(3). I actived this feature on my Soekris firewall by installing PCP and running both the pmcd and the pmwebd services. Once pmwebd is active, querying any metric is quite simple:

  • In python the first step is to get a context:
req = requests.get(url=url + 'pmapi/context?local=foo')
resp = req.json()
ctx_local = resp['context']
  • With this context we can get info about a specific metric:
ctxurl = url + 'pmapi/' + str(ctx_local) + '/'
req = requests.get(url=ctxurl + '_fetch?names=network.interface.out.bytes')
resp = req.json()

The returned JSON is something along the following lines:

{u'timestamp': {u's': 1412888245, u'us': 443665}, u'values': [{u'instances': 
   [{u'instance': 0, u'value': 503730734}, {u'instance': 1, u'value': 17610637798}, ... 

Armed with an Arduino with a 328 Atmel onboard, an Ethernet shield and a 2x16 LCD I wanted to display my ADSL bandwith use. Instead of having to write a network parser for the PCP protocol (or worse SNMP), it’s simple to use the exported JSON data for this. Here are the results:


I’ll eventually clean up the C code I used for this and publish it somewhere.

Direction of Captured Packets

When capturing network traffic on an interface, it is usually pretty obvious which direction the packets are going. Let’s take a typical Linux machine that hosts some VMs over a linux bridge. The interfaces will look like this:

Physical   Linux     Linux      VM
  Nic      Bridge     Tap     Interface
--------  -------  ---------  --------
| eth0 |--| br0 |--| vnet0 |--| eth0 |
--------  -------  ---------  --------

When the VM does an ARP resolution we will see the following on the host’s eth0:

52  33.575036 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has  Tell 
53  33.577890 00:00:0c:4f:2a:30 -> 52:54:00:11:22:33    ARP 60 is at 00:00:0c:4f:2a:30     

In this case it is clear that, from eth0’s point of view, packet 52 is outgoing and 53 is the incoming reply. There are some situations though, where this is not completely obvious:

58  22.252109 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has  Tell
59  22.252202 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has  Tell
60  22.254918 00:00:0c:4f:2a:30 -> 52:54:00:11:22:33    ARP 60 is at 00:00:0c:4f:2a:30    

In the above example, we could assume that both 58 and 59 were outgoing packets, but we’d be wrong. Although size 42 suggests that it has not been padded to ethernet’s minimal frame size, frame number 59 is not really coming from the “external network”. One hint is that arp requests are sent with a second interval between each request, so it’d be unlikely that the VM is the creator of the second packet too. So where is 59 coming from? It turns out that with SR-IOV enabled, some cards’ onboard switch loops packets back. Why is that a problem? Glad you asked.

When the Linux bridge sees packet 59, it records the mac-address 52:54:00:11:22:33 as coming from eth0, and not from the locally connected vnet0 tunnel. When packet 60 arrives, the bridge will drop it because it believes the destination MAC address is on eth0.

Long story short, in order to troubleshoot these kinds of issues, I know of three ways to be able to see the direction of packets:


With a fairly recent tcpdump/libpcap you can specify the -P in|out|inout option and capture traffic in a specific direction. In a situation like the one described here, it will be a bit cumbersome as you will need two separate tcpdump instances, but it works.


netsniff-ng can do an incredible number of cool things. Amongst others, it shows the direction of packets by default:

< em1 60 1400412101s.907918291ns 
 [ Eth MAC (00:00:24:cc:27:40 => 2c:41:38:ab:99:e2), Proto (0x0806, ARP) ]
 [ Vendor (CONNECT AS => Hewlett-Packard Company) ]
 [ ARP Format HA (1 => Ethernet), Format Proto (0x0800 => IPv4), HA Len (6), Proto Len (4), Opcode (1 => ARP request) ]
 [ Chr .................. ]
 [ Hex  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ]

> em1 42 1400412101s.907936759ns 
 [ Eth MAC (2c:41:38:ab:99:e2 => 00:00:24:cc:27:40), Proto (0x0806, ARP) ]
 [ Vendor (Hewlett-Packard Company => CONNECT AS) ]
 [ ARP Format HA (1 => Ethernet), Format Proto (0x0800 => IPv4), HA Len (6), Proto Len (4), Opcode (2 => ARP reply) ]


pktdump is the most user-friendly of the three. It is not in Fedora, but I’ve build a COPR repo here. Here’s an example output:

# pktdump -i em1 -f 'arp'
Capturing packets on the 'em1' interface
[12:24:07] RX(em1) : ARP| REQUEST 00:00:24:CC:27:40 00:00:00:00:00:00
[12:24:07] TX(em1) : ARP| REPLY 2C:41:38:AB:99:E2 00:00:24:CC:27:40

It’s especially useful when trying to follow the route packets are taking in a complex multi-interface setup.

Navigating Source Code With VIM

Lately I’ve had the pleasure to have to read different code bases in a fairly short amount of time. So I spent a little bit of time checking out the available tools to navigate source code in a more efficient way than launching ‘grep’ all over a code base. So first things first, let’s make sure that on this system (Fedora 19) all the needed packages are installed:

$ rpm -q vim-enhanced cscope ctags vim-nerdtree

Let’s start to explore one tool at the time.


I use this plugin just to get a better feel of how files and directories are structured. Once you install ‘vim-nerdtree’, you only need to type “:NERDTree” from VIM to activate it:



Let’s now add the vim-taglist plugin to our VIM installation:

mkdir -p ~/.vim && wget -O ~/ \ \
 && cd ~/.vim && unzip ~/

Also add the following line to your ~/.vimrc:

" Enable Tlist toggle with F8
nnoremap <silent> <F8> :TlistToggle<CR>

Now we can see all the functions, macros and variables of a source file just by pressing F8:


Using Ctrl-] you will always go to the definition of a symbol which is pretty handy (use Ctrl-t to go backwards)


I also use cscope because ctags is not always too precise and it does not allow you to see which functions call a certain other functions which is often quite handy. It is slower than ctags to generate the index so it’s a bit more painful for bigger codebases but nothing too dramatic. Let’s start by creating the index for our codebase via:

cscope -R -b -q

Note: the above cscope command indexes all the files. You might want to make it smarter and index only the files you’re interested in. I do that with a script that takes all the files with extensions and skips certain directories I am not interested in (for example arch/ia64 of the Linux kernel)

Once the index ‘cscope.out’ is created we can use it within VIM. Let’s for example open VIM in the same directory as ‘cscope.out’ and look for “all the functions that call add_matched_proc()” via ‘:cs find c add_matched_proccscope


The cctree plugin is not currently packaged in Fedora so we need to install it by hand:

wget -O ~/.vim/plugin/cctree.vim

Once installed we need to load and parse the cscope DB. We do this with ‘:CCTreeLoadDB cscope.out’. Once this is done we can ask ourselves questions like “what is the callgraph of functions from main()”? We type “:CCTreeTraceForward main” and we get something like the following:


Do you have other useful plugins you use? Drop me a line ;)

In Memory of Ray Dassen (JHM)

It’s unfortunately very official now, our beloved Red Hat colleague and friend Ray Dassen has passed away. I still can’t believe it, as we had just spent a week in Brno together less than a month ago. I will always remember him. JHM++

Here is a little a little gem we (*) came up with some weeks ago while debugging some strace outputs.

Given a strace line like the following:

recvfrom(5, "r\33\205\200\0\1\0\1\0\0\0\0\tboxen1222\4corp\6foobar\3com\0\0\1\0\1\300\f\0\1\0\
1\0\0\16\20\0\4\n\351\360\26", 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("")}, [16]) = 59 

You can feed it to tshark with the following:

printf "\336\316\205\200\0\1\0\1\0\0\0\0\tboxen1222\4corp\6foobar\3com\0\0\1\0\1\300\f\0\1\0\1\0\0\16\20\0\4\n\351\360\26" | \
 od -Ax -tx1 -v  | text2pcap -d -u 53,6666 - /tmp/tmp.pcap && tshark -x -V -r /tmp/tmp.pcap

(*) By “we” I really mean Ray

Telecom Italia ADSL and IPv6

Surprisingly enough (to me at least), I discovered that my ISP actually does support IPv6. You simply need to configure your PPP connection with the following:

  • Username:
  • Password: IPV6@alice6

On my Debian-based Soekris firewall I use shorewall to manage the filtering rules. Eth4 is the internal ADSL modem and eth0 is the internal network. In order to distribute IPv6 to my home network I added the following to /etc/ppp/ipv6-up.d/dsl-provider (nb: it needs the ndisc6 package) :

# Get the /64 network we've been assigned
prefix=$(rdisc6 -1 -q ppp0)
# External interface gets $prefix::1
ip1=$(echo $prefix | sed 's/::\/64/::1\/128/')
ip -6 addr add ${ip1} dev ppp0
# Internal eth0 interface gets $prefix:ffff:ffff:ffff:fffe
ip2=$(echo $prefix | sed 's/::\/64/:ffff:ffff:ffff:fffe\/64/')
ip -6 addr flush scope global dev eth0
ip -6 addr add ${ip2} dev eth0

ip -6 r a default dev ppp0

# Customize and then restart radvd
cat > /etc/radvd.conf <<EOF
interface eth0 {
        AdvSendAdvert on;
        prefix $ip2 {
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr on;

/etc/init.d/radvd restart

Building Your Own FC SAN With Linux


With the Linux kernel 3.5 release the Fibre Channel SCSI Target support has been merged. This means that given a couple of QLogic cards and a Fibre Channel SAN switch you can actually build your own cheap FC SAN environment. I thought of giving it a shot and hit eBay to get:

  • Brocade Silkworm 3250 with 4 SFPs
  • 2 x QLogic ISP2432 HBAs
  • Fibre Optic cables

The small challenge was to actually reset the configuration of the Brocade switch as I kept receiving no output from the serial console. A quick google showed that the DB-9 serial cable needs to have pins 2,3 and 5 connected straight between the connectors. After a little soldering I finally got proper serial output and managed to reset it. Following this excellent guide I reset the password in no time.

Here is the info regarding my setup:

  • marquez – Fedora 19 (as of 20th of May). The target system containing the LUN. Its qla2xxx has WWN: 0x210000e08b943494
  • h2 – RHEL 6.4. The “client” system. WWN: 0x210000e08b947193
  • FC Switch – Fabric OS 4.2.2. WWN: 10:00:00:05:1e:35:d9:0a

FC Switch

Let’s start to configure the FC Switch. We’ll create a simple ZONE adding the WWN’s of the involved parties.

> cfgClear
> aliCreate HBA_MARQUEZ, 21:00:00:e0:8b:94:71:93
> aliCreate HBA_H1, 21:00:00:e0:8b:94:34:94
> zoneCreate "SAN", "HBA_MARQUEZ; HBA_H1"
> cfgDisable
 Updating flash ...
> cfgshow
 Defined configuration:
 alias: HBA_H1 21:00:00:e0:8b:94:34:94
 alias: HBA_MARQUEZ 21:00:00:e0:8b:94:71:93

Effective configuration:
 no configuration in effect

> cfgcreate SAN_CFG, "SAN" 
> cfgenable SAN_CFG
 zone config "SAN_CFG" is in effect
Updating flash ...

With the above we created a single zone called ‘SAN’. The two HBAs are members of this zone.


We need to disable the initiator mode from the qla2xxx driver:

# echo 'options qla2xxx qlini_mode="disabled"' > /usr/lib/modprobe.d/qla2xxx.conf
# rmmod qla2xxx
# modprobe qla2xxx

(Note that it’s better to put it under /usr/lib… as most likely qla2xxx will be loaded from your initramfs. Don’t forget to rebuild the initram with ‘dracut -f’. Then we install targetcli and start configuring the LUN and the ACLs on the LUN:

# yum install -y targetcli
# targetcli
/> qla2xxx/ info
Fabric module name: qla2xxx
ConfigFS path: /sys/kernel/config/target/qla2xxx
Allowed WWN types: naa
Allowed WWNs list: naa.210000e08b943494
Fabric module features: acls
Corresponding kernel module: tcm_qla2xxx

/> /qla2xxx create naa.210000e08b943494
Created target naa.210000e08b943494.
/> /backstores/block create lun1 /dev/sdb
Created block storage object lun1 using /dev/sdb.
/> /qla2xxx/naa.210000e08b943494/luns create /backstores/block/lun1
Created LUN 0.
/> /qla2xxx/naa.210000e08b943494/acls create 210000e08b947193

Note that if there are any errors in the FC Switch configuration we will see errors like ‘SNS scan failed — assuming zero-entry result.’ in your messages.


We can just issue a LIP or rescan the scsi device and we will see the LUN:

# echo 1 > /sys/class/fc_host/host9/issue_lip

# tail -f /var/log/messages
May 20 22:49:24 h2 kernel: qla2xxx [0000:40:00.0]-505f:9: Link is operational (2 Gbps).
May 20 22:49:27 h2 kernel: scsi 9:0:0:0: Direct-Access     LIO-ORG  lun1             4.0  PQ: 0 ANSI: 5
May 20 22:49:27 h2 kernel: sd 9:0:0:0: Attached scsi generic sg2 type 0
May 20 22:49:27 h2 kernel: sd 9:0:0:0: [sdb] 293046768 512-byte logical blocks: (150 GB/139 GiB)
May 20 22:49:27 h2 kernel: sd 9:0:0:0: [sdb] Write Protect is off
May 20 22:49:27 h2 kernel: sd 9:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
May 20 22:49:27 h2 kernel: sdb: sdb1
May 20 22:49:27 h2 kernel: sd 9:0:0:0: [sdb] Attached SCSI disk

At this point we can use /dev/sdb as a normal block device.