Website Logo. Upload to /source/logo.png ; disable in /source/_includes/logo.html

Michele’s Blog

C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D

Terminal Calendaring Application

As an avid mutt-kz user, I always found it quite annoying to have to use a web browser or my phone to check out my work calendar or upcoming birthdays. I have slowly started to use khal which is shaping up to be a very nice calendaring application for use within terminals:

khal

For Fedora users I have created a COPR repo. As root, simply run:

dnf copr enable mbaldessari/khal

and then launch:

dnf install khal

This will install the following packages: python-atomicwrites, vdirsyncer and khal Once installed, we need to tell vdirsyncer where to fetch the caldav entries from. My ~/.vdirsyncer/config is as follows and contains my birthday list from Facebook and my work calendar:

[general]
status_path = ~/.vdirsyncer/status/
 
[pair my_calendar]
a = my_calendar_local
b = my_calendar_remote
 
[storage my_calendar_local]
type = filesystem
path = ~/.calendar/work/
fileext = .ics
 
[storage my_calendar_remote]
type = caldav
url =  https://foo.bar.corp/dav/michele@foo.com/Calendar/
username = michele
password = 123

[pair my_fb_birthdays]
a = my_fb_birthdays_local
b = my_fb_birthdays_remote

[storage my_fb_birthdays_local]
type = filesystem
path = ~/.calendar/fb_birthdays/
fileext = .ics

[storage my_fb_birthdays_remote]
type = http
url = https://www.facebook.com/ical/b.php?uid=123&key=123

At this point you can run vdirsyncer sync and the entries will be fetched and stored locally. Note that if you get SSL validation errors you most likely need to import your company Root CA:

# cp MY-CA-root-cert.pem /etc/pki/ca-trust/source/anchors/
# update-ca-trust extract

Once vdirsyncer has completed fetching all the entries, it is time to display them in khal or in its interactive cousing ikhal. Here is my ~/.config/khal/khal.conf file:

[calendars]
 
[[fb]]
path = ~/.calendar/fb_birthdays/
color = yellow

[[work]]
path = ~/.calendar/work/
color = dark red
 
[locale]
local_timezone= Europe/Berlin
default_timezone= Europe/Berlin
timeformat= %H:%M
dateformat= %d.%m.
longdateformat= %d.%m.%Y
datetimeformat= %d.%m. %H:%M
longdatetimeformat= %d.%m.%Y %H:%M

That’s it. There is still some work todo before I can ditch other calendaring applications completely. Namely:

Let me know if you have any issues with the copr repo.

Performance Analysis With Performance Co-Pilot, iPython and Pandas

Introduction

One of many reasons to love Performance Co-Pilot, is the fact that it is a fully fledged framework to do performance analysis. It makes it extremely simple to extend and to build anything on top of it. In this post we shall explore how simple it is to analyze your performance data using iPython and pandas.

panda

Setup

To start we will need some PCP archives which contain some collected metrics from a system. In this post I will use the data I collect on my home firewall and will try to analyze some of the data there in. To learn how to store performance metrics in an archive, take a look at pmlogger and the Quickstart guide. For this example I collected data over the course of a day with a 1 minute interval.

iPython and PCP

First of all you need to import a small python module that bridges PCP and pandas/numpy:

git clone https://github.com/mbaldessari/pcpinteractive.git
cd pcpinteractive

Now let us start our iPython console, import our python module and load our archive:

$ ipython qtconsole --pylab=inline
In [1]: import pcparchive, pandas
In [2]: p = pcparchive.PcpArchive('testfiles/monday')
In [3]: p.parse()
Parsing archive: [########## 100.0%]

At this point the data is fully parsed in memory and we can start analyzing it, using all the standard tools like pandas and matplotlib. Let’s start by looking at how many metrics are present in the archive:

In [4]: metrics = p.get_metrics()
In [5]: len(metrics)
Out[5]: 253

Pandas and PCP

Now we can get a pandas object out of a metric. Let’s take incoming and outgoing network traffic expressed in bytes over time.

In [6]: net_in = p.get_pandas('network.interface.in.bytes')
In [7]: net_out = p.get_pandas('network.interface.out.bytes')

We can now graph the data obtained with a simple:

In [8]: net_in.plot()

netpcpmatplot

And we can also explore the data with the use of the describe() method, but first let’s force the output to be in non-scientific notation as it is more readable for network data:

In [9]: pandas.set_option('display.float_format', lambda x: '%.2f' % x)
In [10]: net_in.describe()
Out[11]: 
                eth0    eth1    eth2            eth3             lo
count        1439.00 1439.00 1439.00         1439.00        1439.00   
mean  23209831415.26    0.00    0.00 257865786302.88 32743773553.20   
std     133904051.55    0.00    0.00   1677032669.74      887369.78   
min   23074135536.00    0.00    0.00 256117467223.00 32742236187.00   
25%   23077865299.00    0.00    0.00 256128768785.50 32743005086.50   
50%   23160237636.00    0.00    0.00 257410715707.00 32743771068.00   
75%   23319280110.50    0.00    0.00 259252325129.50 32744537961.50   
max   23470418350.00    0.00    0.00 260826247120.00 32745311229.00   

                 ppp0       redhat0           tun0  
count         1439.00       1439.00        1439.00  
mean  253307755506.55 6897358284.79 12675081585.52  
std     1648648131.02  163459557.15        8568.61  
min   251589569686.00 6726948179.00 12675070001.00  
25%   251600158007.50 6727362445.50 12675070001.00  
50%   252860366375.00 6859405494.00 12675088375.00  
75%   254670529194.50 7051678675.50 12675088375.00  
max   256217896469.00 7171608913.00 12675088375.00

Manipulate the data

Now let’s see what is possible to do in terms of data manipulation: * Drop columns we do not care about:

In [12]: del net_in['eth1']

Or, alternatively:

In [13]: net_in = net_in.drop('eth2', 1)
  • Resample data at lower intervals
In [14]: net.resample('10min')
  • Filter out all the zero columns
In [15]: net_clean = net.ix[:,(net != 0).any(axis=0)]
  • Show last element:
In [16]: net_clean.ix[-1]
  • Select a smaller timeframe:
In [17]: net_clean['2014-12-28 12:00':'2014-12-28 14:00']
  • Get one column with:
In [18]: net_in.eth4
In [19]: net_in['lo']
  • Apply a function on the whole dataframe:
In [20]: net_in * 5
  • Sum all values for each column:
In [21]: net_in.sum()
Out[22]: 
eth0        33398947406561
eth3       371068866489847
lo          47118290143065
ppp0       364509860173927
redhat0      9925298571818
tun0        18239442401565
dtype: int64
  • Calculate the mean for each column:
In [23]: net_in.mean()
  • Find the time of day when the max values are reached
In [24]: net_in.idxmax()
In [25]: net_in.idxmax(axis=1)
  • Select only the tun0 and eth0 devices:
In [26]: test = net_in.ix[:, ['tun0', 'eth0']]

Merge and group dataframes

Now let’s merge the net_in and the net_out dataframes into a single one, in order to try and do some analysis on both traffic directions at the same time.

In [27]: net = pandas.merge(net_in, net_out, left_index=True, right_index=True, how='left', suffixes=('_in', '_out'))

Another very interesting aspect is the plethora of statistical functions that come for free through the use of pandas. For example, to find covariance() and correlation() we can use the following methods:

In [28]: net.cov()
In [29]: net.corr()
In [30]: net.corrwith(net.eth0_in)
Out [31]:
eth0_in       1.000
eth3_in       0.993
lo_in         0.958
ppp0_in       0.993
redhat0_in    0.991
tun0_in       0.748
eth0_out      0.994
eth3_out      0.986
lo_out        0.958
ppp0_out      0.983
redhat0_out   0.978
tun0_out      0.748

We can also group columns like the following:

In [32]: mapping = { 'eth0_in': 'in', 'eth3_in': 'in', 'ppp0_in': 'in', 'tun0_in': 'in',
                     'redhat0_in': 'in', 'lo_in': 'in', 'eth0_out': 'out', 'eth3_out': 'out',
                     'ppp0_out': 'out', 'tun0_out': 'out', 'redhat0_out': 'out', 'lo_out': 'out'}
In [33]: test = net.groupby(mapping, axis=1)
In [34]: test.sum()

Calculate the rolling mean of an interface and plot it:

In [35]: pandas.rolling_mean(net.eth0_in, 250).plot()

rollingmean

Export data

Save the data in csv file or in excel format:

In [36]: net.to_csv('testfiles/out.csv')
In [37]: net.to_excel('testfiles/out.xls')

Other outputs like latex, sql, clipboard, hd5f and more are supported.

Conclusions

The versatility of PCP allows anyone to use many currently available frameworks (numpy, pandas, R, scipy) to analyze and display the collected performance data. There is some work to be done to make this process a bit simpler with an out of the box PCP installation.

Observing X11 Protocol Differences

I was trying to understand some oddities going with an X11 legacy application showing bad artifacts in one environment and working flawlessly in another environment. Since wireshark does not have any support for diffing two pcaps, I came up with the following steps:

  • Dump both working.pcap and nonworking.pcap into text files with full headers:
1
2
~/Devel/wireshark/tshark -r working.pcap -T text -V -Y "x11" > working.txt
~/Devel/wireshark/tshark -r notworking.pcap -T text -V -Y "x11" > notworking.txt
  • Prune the two text files with a script like the following:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
def clean(source, dest):
    f = open(source, "r")
    x11_state = False
    output = []
    for line in f.readlines():
        if x11_state:
            if line.startswith(' '):
                output.append(line)
            else:
                x11_state = False
        else:
            if line.startswith('X11') and \
                    "Property" in line:
                output.append(line)
                x11_state = True
            else:
                continue
    o = open(dest, "w")
    for i in output:
        o.write(i)

if __name__ == '__main__':
    clean("working.txt", "working-trimmed.txt")
    clean("notworking.txt", "notworking-trimmed.txt")
  • At this point we can easily vimdiff the outputs obtained from above via vimdiff working-trimmed.txt notworking-trimmed.txt: vimdiff

Performance Co-Pilot and Arduino (Part 2)

After initially setting up Performance Co-Pilot and Arduino, I wanted to improve the data being displayed. As latency is quite important to me, I wanted to display that as well. I did not have too much time on my hands to code a new PMDA that collects that information, so I abused the pmdashping(1) for this purpose. The steps are simple:

  1. Go to /var/lib/pcp/pmdas/shping
  2. Create sample.conf with the following line:
1
8.8.8.8 /var/lib/pcp/pmdas/shping/ping.sh
  1. Create /var/lib/pcp/pmdas/shping/ping.sh:
1
2
3
4
#!/bin/sh
# This hack will break if latency > 254 ;)
ret=`ping -c 2 -n -q acksyn.org | grep ^rtt | cut -f5 -d\/ | cut -f1 -d\.`
exit $ret
  1. Launch ./Install and choose [2] ./sample.conf
  2. Now it is possible to abuse the shping.error metric to fetch that value:
1
2
3
$ pminfo -f shping.error
shping.error
   inst [0 or "8.8.8.8"] value 52

The last step was to fetch this via PMWEBAPI(3). This did not work until I realized, thanks to Fche’s suggestion that the issue was related to my inital context initialization. As a matter of fact there is a big difference between the following two:

  • /pmapi/context?local=ANYTHING – Creates a PM_CONTEXT_LOCAL PMAPI context.
  • /pmapi/context?hostname=STRING – Creates a PM_CONTEXT_HOST PMAPI context with the given host name and/or extended specification.

The man page of pmNewContext(3) explains this in more detail. Frank has added some more info to the PMWEBAPI(3) man page via the following commit, to make it a little bit more obvious. It’s still a pretty gross hack, but for the time being it’s enough for my needs.

arduino-pcp

Performance Co-Pilot and Arduino

Besides being an incredible nice toolkit to work with, Performance Co-Pilot is extremely simple to integrate with any application. Amongst an extensive API it allows to export any metric via JSON using the PMWEBAPI(3). I actived this feature on my Soekris firewall by installing PCP and running both the pmcd and the pmwebd services. Once pmwebd is active, querying any metric is quite simple:

  • In python the first step is to get a context:
1
2
3
req = requests.get(url=url + 'pmapi/context?local=foo')
resp = req.json()
ctx_local = resp['context']
  • With this context we can get info about a specific metric:
1
2
3
ctxurl = url + 'pmapi/' + str(ctx_local) + '/'
req = requests.get(url=ctxurl + '_fetch?names=network.interface.out.bytes')
resp = req.json()

The returned JSON is something along the following lines:

{u'timestamp': {u's': 1412888245, u'us': 443665}, u'values': [{u'instances': 
   [{u'instance': 0, u'value': 503730734}, {u'instance': 1, u'value': 17610637798}, ... 

Armed with an Arduino with a 328 Atmel onboard, an Ethernet shield and a 2x16 LCD I wanted to display my ADSL bandwith use. Instead of having to write a network parser for the PCP protocol (or worse SNMP), it’s simple to use the exported JSON data for this. Here are the results:

arduino-pcp

I’ll eventually clean up the C code I used for this and publish it somewhere.

Direction of Captured Packets

When capturing network traffic on an interface, it is usually pretty obvious which direction the packets are going. Let’s take a typical Linux machine that hosts some VMs over a linux bridge. The interfaces will look like this:

Physical   Linux     Linux      VM
  Nic      Bridge     Tap     Interface
--------  -------  ---------  --------
| eth0 |--| br0 |--| vnet0 |--| eth0 |
--------  -------  ---------  --------

When the VM does an ARP resolution we will see the following on the host’s eth0:

52  33.575036 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has 192.168.0.254?  Tell 192.168.1.70 
53  33.577890 00:00:0c:4f:2a:30 -> 52:54:00:11:22:33    ARP 60   192.168.0.254 is at 00:00:0c:4f:2a:30     

In this case it is clear that, from eth0’s point of view, packet 52 is outgoing and 53 is the incoming reply. There are some situations though, where this is not completely obvious:

58  22.252109 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has 192.168.0.254?  Tell 192.168.1.70
59  22.252202 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has 192.168.0.254?  Tell 192.168.1.70
60  22.254918 00:00:0c:4f:2a:30 -> 52:54:00:11:22:33    ARP 60   192.168.0.254 is at 00:00:0c:4f:2a:30    

In the above example, we could assume that both 58 and 59 were outgoing packets, but we’d be wrong. Although size 42 suggests that it has not been padded to ethernet’s minimal frame size, frame number 59 is not really coming from the “external network”. One hint is that arp requests are sent with a second interval between each request, so it’d be unlikely that the VM is the creator of the second packet too. So where is 59 coming from? It turns out that with SR-IOV enabled, some cards’ onboard switch loops packets back. Why is that a problem? Glad you asked.

When the Linux bridge sees packet 59, it records the mac-address 52:54:00:11:22:33 as coming from eth0, and not from the locally connected vnet0 tunnel. When packet 60 arrives, the bridge will drop it because it believes the destination MAC address is on eth0.

Long story short, in order to troubleshoot these kinds of issues, I know of three ways to be able to see the direction of packets:

tcpdump

With a fairly recent tcpdump/libpcap you can specify the -P in|out|inout option and capture traffic in a specific direction. In a situation like the one described here, it will be a bit cumbersome as you will need two separate tcpdump instances, but it works.

netsniff-ng

netsniff-ng can do an incredible number of cool things. Amongst others, it shows the direction of packets by default:

< em1 60 1400412101s.907918291ns 
 [ Eth MAC (00:00:24:cc:27:40 => 2c:41:38:ab:99:e2), Proto (0x0806, ARP) ]
 [ Vendor (CONNECT AS => Hewlett-Packard Company) ]
 [ ARP Format HA (1 => Ethernet), Format Proto (0x0800 => IPv4), HA Len (6), Proto Len (4), Opcode (1 => ARP request) ]
 [ Chr .................. ]
 [ Hex  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ]

> em1 42 1400412101s.907936759ns 
 [ Eth MAC (2c:41:38:ab:99:e2 => 00:00:24:cc:27:40), Proto (0x0806, ARP) ]
 [ Vendor (Hewlett-Packard Company => CONNECT AS) ]
 [ ARP Format HA (1 => Ethernet), Format Proto (0x0800 => IPv4), HA Len (6), Proto Len (4), Opcode (2 => ARP reply) ]

pktdump

pktdump is the most user-friendly of the three. It is not in Fedora, but I’ve build a COPR repo here. Here’s an example output:

# pktdump -i em1 -f 'arp'
Capturing packets on the 'em1' interface
[12:24:07] RX(em1) : ARP| REQUEST 00:00:24:CC:27:40 192.168.0.254 00:00:00:00:00:00 foo.int.
[12:24:07] TX(em1) : ARP| REPLY 2C:41:38:AB:99:E2 foo.int 00:00:24:CC:27:40 192.168.0.254

It’s especially useful when trying to follow the route packets are taking in a complex multi-interface setup.

Navigating Source Code With VIM

Lately I’ve had the pleasure to have to read different code bases in a fairly short amount of time. So I spent a little bit of time checking out the available tools to navigate source code in a more efficient way than launching ‘grep’ all over a code base. So first things first, let’s make sure that on this system (Fedora 19) all the needed packages are installed:

$ rpm -q vim-enhanced cscope ctags vim-nerdtree
 vim-enhanced-7.4.027-2.fc19.x86_64
 cscope-15.8-4.fc19.x86_64
 ctags-5.8-10.fc19.x86_64
 vim-nerdtree-4.2.0-9.fc19.noarch

Let’s start to explore one tool at the time.

NerdTree

I use this plugin just to get a better feel of how files and directories are structured. Once you install ‘vim-nerdtree’, you only need to type “:NERDTree” from VIM to activate it:

nerdtree

ctags

Let’s now add the vim-taglist plugin to our VIM installation:

mkdir -p ~/.vim && wget -O ~/vimtaglist.zip \
 http://vim.sourceforge.net/scripts/download_script.php?src_id=19574 \
 && cd ~/.vim && unzip ~/vimtaglist.zip

Also add the following line to your ~/.vimrc:

" Enable Tlist toggle with F8
nnoremap <silent> <F8> :TlistToggle<CR>

Now we can see all the functions, macros and variables of a source file just by pressing F8:

ctags

Using Ctrl-] you will always go to the definition of a symbol which is pretty handy (use Ctrl-t to go backwards)

cscope

I also use cscope because ctags is not always too precise and it does not allow you to see which functions call a certain other functions which is often quite handy. It is slower than ctags to generate the index so it’s a bit more painful for bigger codebases but nothing too dramatic. Let’s start by creating the index for our codebase via:

cscope -R -b -q

Note: the above cscope command indexes all the files. You might want to make it smarter and index only the files you’re interested in. I do that with a script that takes all the files with extensions and skips certain directories I am not interested in (for example arch/ia64 of the Linux kernel)

Once the index ‘cscope.out’ is created we can use it within VIM. Let’s for example open VIM in the same directory as ‘cscope.out’ and look for “all the functions that call add_matched_proc()” via ‘:cs find c add_matched_proccscope

CCTree

The cctree plugin is not currently packaged in Fedora so we need to install it by hand:

wget -O ~/.vim/plugin/cctree.vim http://www.vim.org/scripts/download_script.php?src_id=18112

Once installed we need to load and parse the cscope DB. We do this with ‘:CCTreeLoadDB cscope.out’. Once this is done we can ask ourselves questions like “what is the callgraph of functions from main()”? We type “:CCTreeTraceForward main” and we get something like the following:

cctree

Do you have other useful plugins you use? Drop me a line ;)

In Memory of Ray Dassen (JHM)

It’s unfortunately very official now, our beloved Red Hat colleague and friend Ray Dassen has passed away. I still can’t believe it, as we had just spent a week in Brno together less than a month ago. I will always remember him. JHM++

Here is a little a little gem we (*) came up with some weeks ago while debugging some strace outputs.

Given a strace line like the following:

recvfrom(5, "r\33\205\200\0\1\0\1\0\0\0\0\tboxen1222\4corp\6foobar\3com\0\0\1\0\1\300\f\0\1\0\
1\0\0\16\20\0\4\n\351\360\26", 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("
10.10.1.1")}, [16]) = 59 

You can feed it to tshark with the following:

printf "\336\316\205\200\0\1\0\1\0\0\0\0\tboxen1222\4corp\6foobar\3com\0\0\1\0\1\300\f\0\1\0\1\0\0\16\20\0\4\n\351\360\26" | \
 od -Ax -tx1 -v  | text2pcap -d -u 53,6666 - /tmp/tmp.pcap && tshark -x -V -r /tmp/tmp.pcap

(*) By “we” I really mean Ray

Telecom Italia ADSL and IPv6

Surprisingly enough (to me at least), I discovered that my ISP actually does support IPv6. You simply need to configure your PPP connection with the following:

  • Username: adsl@alice6.it
  • Password: IPV6@alice6

On my Debian-based Soekris firewall I use shorewall to manage the filtering rules. Eth4 is the internal ADSL modem and eth0 is the internal network. In order to distribute IPv6 to my home network I added the following to /etc/ppp/ipv6-up.d/dsl-provider (nb: it needs the ndisc6 package) :

#!/bin/bash
# Get the /64 network we've been assigned
prefix=$(rdisc6 -1 -q ppp0)
# External interface gets $prefix::1
ip1=$(echo $prefix | sed 's/::\/64/::1\/128/')
ip -6 addr add ${ip1} dev ppp0
# Internal eth0 interface gets $prefix:ffff:ffff:ffff:fffe
ip2=$(echo $prefix | sed 's/::\/64/:ffff:ffff:ffff:fffe\/64/')
ip -6 addr flush scope global dev eth0
ip -6 addr add ${ip2} dev eth0

ip -6 r a default dev ppp0

# Customize and then restart radvd
cat > /etc/radvd.conf <<EOF
interface eth0 {
        AdvSendAdvert on;
        prefix $ip2 {
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr on;
        };
};
EOF

/etc/init.d/radvd restart