Website Logo. Upload to /source/logo.png ; disable in /source/_includes/logo.html

Michele's Blog

C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D

Direction of Captured Packets

When capturing network traffic on an interface, it is usually pretty obvious which direction the packets are going. Let’s take a typical Linux machine that hosts some VMs over a linux bridge. The interfaces will look like this:

Physical   Linux     Linux      VM
  Nic      Bridge     Tap     Interface
--------  -------  ---------  --------
| eth0 |--| br0 |--| vnet0 |--| eth0 |
--------  -------  ---------  --------

When the VM does an ARP resolution we will see the following on the host’s eth0:

52  33.575036 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has 192.168.0.254?  Tell 192.168.1.70 
53  33.577890 00:00:0c:4f:2a:30 -> 52:54:00:11:22:33    ARP 60   192.168.0.254 is at 00:00:0c:4f:2a:30     

In this case it is clear that, from eth0’s point of view, packet 52 is outgoing and 53 is the incoming reply. There are some situations though, where this is not completely obvious:

58  22.252109 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has 192.168.0.254?  Tell 192.168.1.70
59  22.252202 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has 192.168.0.254?  Tell 192.168.1.70
60  22.254918 00:00:0c:4f:2a:30 -> 52:54:00:11:22:33    ARP 60   192.168.0.254 is at 00:00:0c:4f:2a:30    

In the above example, we could assume that both 58 and 59 were outgoing packets, but we’d be wrong. Although size 42 suggests that it has not been padded to ethernet’s minimal frame size, frame number 59 is not really coming from the “external network”. One hint is that arp requests are sent with a second interval between each request, so it’d be unlikely that the VM is the creator of the second packet too. So where is 59 coming from? It turns out that with SR-IOV enabled, some cards’ onboard switch loops packets back. Why is that a problem? Glad you asked.

When the Linux bridge sees packet 59, it records the mac-address 52:54:00:11:22:33 as coming from eth0, and not from the locally connected vnet0 tunnel. When packet 60 arrives, the bridge will drop it because it believes the destination MAC address is on eth0.

Long story short, in order to troubleshoot these kinds of issues, I know of three ways to be able to see the direction of packets:

tcpdump

With a fairly recent tcpdump/libpcap you can specify the -P in|out|inout option and capture traffic in a specific direction. In a situation like the one described here, it will be a bit cumbersome as you will need two separate tcpdump instances, but it works.

netsniff-ng

netsniff-ng can do an incredible number of cool things. Amongst others, it shows the direction of packets by default:

< em1 60 1400412101s.907918291ns 
 [ Eth MAC (00:00:24:cc:27:40 => 2c:41:38:ab:99:e2), Proto (0x0806, ARP) ]
 [ Vendor (CONNECT AS => Hewlett-Packard Company) ]
 [ ARP Format HA (1 => Ethernet), Format Proto (0x0800 => IPv4), HA Len (6), Proto Len (4), Opcode (1 => ARP request) ]
 [ Chr .................. ]
 [ Hex  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ]

> em1 42 1400412101s.907936759ns 
 [ Eth MAC (2c:41:38:ab:99:e2 => 00:00:24:cc:27:40), Proto (0x0806, ARP) ]
 [ Vendor (Hewlett-Packard Company => CONNECT AS) ]
 [ ARP Format HA (1 => Ethernet), Format Proto (0x0800 => IPv4), HA Len (6), Proto Len (4), Opcode (2 => ARP reply) ]

pktdump

pktdump is the most user-friendly of the three. It is not in Fedora, but I’ve build a COPR repo here. Here’s an example output:

# pktdump -i em1 -f 'arp'
Capturing packets on the 'em1' interface
[12:24:07] RX(em1) : ARP| REQUEST 00:00:24:CC:27:40 192.168.0.254 00:00:00:00:00:00 foo.int.
[12:24:07] TX(em1) : ARP| REPLY 2C:41:38:AB:99:E2 foo.int 00:00:24:CC:27:40 192.168.0.254

It’s especially useful when trying to follow the route packets are taking in a complex multi-interface setup.

Navigating Source Code With VIM

Lately I’ve had the pleasure to have to read different code bases in a fairly short amount of time. So I spent a little bit of time checking out the available tools to navigate source code in a more efficient way than launching ‘grep’ all over a code base. So first things first, let’s make sure that on this system (Fedora 19) all the needed packages are installed:

$ rpm -q vim-enhanced cscope ctags vim-nerdtree
 vim-enhanced-7.4.027-2.fc19.x86_64
 cscope-15.8-4.fc19.x86_64
 ctags-5.8-10.fc19.x86_64
 vim-nerdtree-4.2.0-9.fc19.noarch

Let’s start to explore one tool at the time.

NerdTree

I use this plugin just to get a better feel of how files and directories are structured. Once you install ‘vim-nerdtree’, you only need to type “:NERDTree” from VIM to activate it:

nerdtree

ctags

Let’s now add the vim-taglist plugin to our VIM installation:

mkdir -p ~/.vim && wget -O ~/vimtaglist.zip \
 http://vim.sourceforge.net/scripts/download_script.php?src_id=19574 \
 && cd ~/.vim && unzip ~/vimtaglist.zip

Also add the following line to your ~/.vimrc:

" Enable Tlist toggle with F8
nnoremap <silent> <F8> :TlistToggle<CR>

Now we can see all the functions, macros and variables of a source file just by pressing F8:

ctags

Using Ctrl-] you will always go to the definition of a symbol which is pretty handy (use Ctrl-t to go backwards)

cscope

I also use cscope because ctags is not always too precise and it does not allow you to see which functions call a certain other functions which is often quite handy. It is slower than ctags to generate the index so it’s a bit more painful for bigger codebases but nothing too dramatic. Let’s start by creating the index for our codebase via:

cscope -R -b -q

Note: the above cscope command indexes all the files. You might want to make it smarter and index only the files you’re interested in. I do that with a script that takes all the files with extensions and skips certain directories I am not interested in (for example arch/ia64 of the Linux kernel)

Once the index ‘cscope.out’ is created we can use it within VIM. Let’s for example open VIM in the same directory as ‘cscope.out’ and look for “all the functions that call add_matched_proc()” via ‘:cs find c add_matched_proccscope

CCTree

The cctree plugin is not currently packaged in Fedora so we need to install it by hand:

wget -O ~/.vim/plugin/cctree.vim http://www.vim.org/scripts/download_script.php?src_id=18112

Once installed we need to load and parse the cscope DB. We do this with ‘:CCTreeLoadDB cscope.out’. Once this is done we can ask ourselves questions like “what is the callgraph of functions from main()”? We type “:CCTreeTraceForward main” and we get something like the following:

cctree

Do you have other useful plugins you use? Drop me a line ;)

In Memory of Ray Dassen (JHM)

It’s unfortunately very official now, our beloved Red Hat colleague and friend Ray Dassen has passed away. I still can’t believe it, as we had just spent a week in Brno together less than a month ago. I will always remember him. JHM++

Here is a little a little gem we (*) came up with some weeks ago while debugging some strace outputs.

Given a strace line like the following:

recvfrom(5, "r\33\205\200\0\1\0\1\0\0\0\0\tboxen1222\4corp\6foobar\3com\0\0\1\0\1\300\f\0\1\0\
1\0\0\16\20\0\4\n\351\360\26", 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("
10.10.1.1")}, [16]) = 59 

You can feed it to tshark with the following:

printf "\336\316\205\200\0\1\0\1\0\0\0\0\tboxen1222\4corp\6foobar\3com\0\0\1\0\1\300\f\0\1\0\1\0\0\16\20\0\4\n\351\360\26" | \
 od -Ax -tx1 -v  | text2pcap -d -u 53,6666 - /tmp/tmp.pcap && tshark -x -V -r /tmp/tmp.pcap

(*) By “we” I really mean Ray

Telecom Italia ADSL and IPv6

Surprisingly enough (to me at least), I discovered that my ISP actually does support IPv6. You simply need to configure your PPP connection with the following:

  • Username: adsl@alice6.it
  • Password: IPV6@alice6

On my Debian-based Soekris firewall I use shorewall to manage the filtering rules. Eth4 is the internal ADSL modem and eth0 is the internal network. In order to distribute IPv6 to my home network I added the following to /etc/ppp/ipv6-up.d/dsl-provider (nb: it needs the ndisc6 package) :

#!/bin/bash
# Get the /64 network we've been assigned
prefix=$(rdisc6 -1 -q ppp0)
# External interface gets $prefix::1
ip1=$(echo $prefix | sed 's/::\/64/::1\/128/')
ip -6 addr add ${ip1} dev ppp0
# Internal eth0 interface gets $prefix:ffff:ffff:ffff:fffe
ip2=$(echo $prefix | sed 's/::\/64/:ffff:ffff:ffff:fffe\/64/')
ip -6 addr flush scope global dev eth0
ip -6 addr add ${ip2} dev eth0

ip -6 r a default dev ppp0

# Customize and then restart radvd
cat > /etc/radvd.conf <<EOF
interface eth0 {
        AdvSendAdvert on;
        prefix $ip2 {
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr on;
        };
};
EOF

/etc/init.d/radvd restart

Building Your Own FC SAN With Linux

silkworm

With the Linux kernel 3.5 release the Fibre Channel SCSI Target support has been merged. This means that given a couple of QLogic cards and a Fibre Channel SAN switch you can actually build your own cheap FC SAN environment. I thought of giving it a shot and hit eBay to get:

  • Brocade Silkworm 3250 with 4 SFPs
  • 2 x QLogic ISP2432 HBAs
  • Fibre Optic cables

The small challenge was to actually reset the configuration of the Brocade switch as I kept receiving no output from the serial console. A quick google showed that the DB-9 serial cable needs to have pins 2,3 and 5 connected straight between the connectors. After a little soldering I finally got proper serial output and managed to reset it. Following this excellent guide I reset the password in no time.

Here is the info regarding my setup:

  • marquez – Fedora 19 (as of 20th of May). The target system containing the LUN. Its qla2xxx has WWN: 0x210000e08b943494
  • h2 – RHEL 6.4. The “client” system. WWN: 0x210000e08b947193
  • FC Switch – Fabric OS 4.2.2. WWN: 10:00:00:05:1e:35:d9:0a

FC Switch

Let’s start to configure the FC Switch. We’ll create a simple ZONE adding the WWN’s of the involved parties.

> cfgClear
> aliCreate HBA_MARQUEZ, 21:00:00:e0:8b:94:71:93
> aliCreate HBA_H1, 21:00:00:e0:8b:94:34:94
> zoneCreate "SAN", "HBA_MARQUEZ; HBA_H1"
> cfgDisable
 Updating flash ...
> cfgshow
 Defined configuration:
 zone: SAN HBA_MARQUEZ; HBA_H1
 alias: HBA_H1 21:00:00:e0:8b:94:34:94
 alias: HBA_MARQUEZ 21:00:00:e0:8b:94:71:93

Effective configuration:
 no configuration in effect

> cfgcreate SAN_CFG, "SAN" 
> cfgenable SAN_CFG
 zone config "SAN_CFG" is in effect
Updating flash ...

With the above we created a single zone called ‘SAN’. The two HBAs are members of this zone.

Target

We need to disable the initiator mode from the qla2xxx driver:

# echo 'options qla2xxx qlini_mode="disabled"' > /usr/lib/modprobe.d/qla2xxx.conf
# rmmod qla2xxx
# modprobe qla2xxx

(Note that it’s better to put it under /usr/lib… as most likely qla2xxx will be loaded from your initramfs. Don’t forget to rebuild the initram with ‘dracut -f’. Then we install targetcli and start configuring the LUN and the ACLs on the LUN:

# yum install -y targetcli
# targetcli
/> qla2xxx/ info
Fabric module name: qla2xxx
ConfigFS path: /sys/kernel/config/target/qla2xxx
Allowed WWN types: naa
Allowed WWNs list: naa.210000e08b943494
Fabric module features: acls
Corresponding kernel module: tcm_qla2xxx

/> /qla2xxx create naa.210000e08b943494
Created target naa.210000e08b943494.
/> /backstores/block create lun1 /dev/sdb
Created block storage object lun1 using /dev/sdb.
/> /qla2xxx/naa.210000e08b943494/luns create /backstores/block/lun1
Created LUN 0.
/> /qla2xxx/naa.210000e08b943494/acls create 210000e08b947193

Note that if there are any errors in the FC Switch configuration we will see errors like ‘SNS scan failed — assuming zero-entry result.’ in your messages.

Initiator

We can just issue a LIP or rescan the scsi device and we will see the LUN:

# echo 1 > /sys/class/fc_host/host9/issue_lip

# tail -f /var/log/messages
May 20 22:49:24 h2 kernel: qla2xxx [0000:40:00.0]-505f:9: Link is operational (2 Gbps).
May 20 22:49:27 h2 kernel: scsi 9:0:0:0: Direct-Access     LIO-ORG  lun1             4.0  PQ: 0 ANSI: 5
May 20 22:49:27 h2 kernel: sd 9:0:0:0: Attached scsi generic sg2 type 0
May 20 22:49:27 h2 kernel: sd 9:0:0:0: [sdb] 293046768 512-byte logical blocks: (150 GB/139 GiB)
May 20 22:49:27 h2 kernel: sd 9:0:0:0: [sdb] Write Protect is off
May 20 22:49:27 h2 kernel: sd 9:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
May 20 22:49:27 h2 kernel: sdb: sdb1
May 20 22:49:27 h2 kernel: sd 9:0:0:0: [sdb] Attached SCSI disk

At this point we can use /dev/sdb as a normal block device.

The Hat Is Red

TL;DR: Long time no blog. Changed jobs beginning 2012.

List All Your Forwarders With Powershell

I clearly lied about on my previous post about being the last Microsoft-based post. Here’s what I did to check all our forwarders in our forest:

$domain = "corp.local"
$myForest = [System.DirectoryServices.ActiveDirectory.Forest]::GetCurrentForest()
$dc_list = $myforest.Sites | % { $_.Servers } | Select Name

foreach ($dc in $dc_list) {
    $DCName = $dc.Name
    $Server = Get-WMIObject -Computer $DCName -Namespace "root\MicrosoftDNS" -Class "MicrosoftDNS_Server"
    $Forwarders = $Server.Forwarders
    Write-Host $DCName ":" $Forwarders
}

Lync Contact Groups and Powershell

This will be likely one of my last Windows posts (more on that later). At work we did upgrade our OCS Server to Lync. Besides the annoyance of changing product name every two releases, this new version removed the possibility that existed in OCS to provision the contact groups for the ocs client. It was a WMI interface and it was quite handy.

Especially in cases where the rollout is not company-wide and users aren’t sure who has Lync and who doesn’t, having groups provisioned administratively is very handy. So I spent a couple of evenings solving this topic and it turned out that it’s not really that difficult ;)

After poking a bit in the DB I noticed some handy stored procedures that would be quite handy. So I cooked up a script that takes a folder and for every csv file stored in that folder (one samaccount per line), it creates a group named as the file and it populates it with the users contained in that file. I.e.: the file IT-Dept.csv that contains users jon123 and joe456, will create a group called “IT-Dept” which contains both users and only for those two users.

Disclaimer: this script is still buggy, it’s coded horrendously and it’s a very quick hack. Backup your db before toying with it. I won’t have time to clean it up or refine it so feel free to tweak it, put it on github etc ;)

Here it is: sync-lync-rosters.ps1

Microsoft PKI and Powershell

I need to check among all issued certificates from a Microsoft PKI if there is a disabled or non valid Active Directory account. Since powershell seems to be the future for scripting on Microsoft platform, I cooked up the following script: check-ad-pki.ps1 It uses the powershell PKI module which can be found here.

It’s my first stab at something with powershell, so I do appreciate any feedback on this. If anyone has better approaches to check the issue of certificates for a user that is now disabled or expired, that would be cool too ;)