Direction of Captured Packets

When capturing network traffic on an interface, it is usually pretty obvious which direction the packets are going. Let's take a typical Linux machine that hosts some VMs over a linux bridge. The interfaces will look like this:

Physical   Linux     Linux      VM
  Nic      Bridge     Tap     Interface
--------  -------  ---------  --------
| eth0 |--| br0 |--| vnet0 |--| eth0 |
--------  -------  ---------  --------

When the VM does an ARP resolution we will see the following on the host's eth0:

52  33.575036 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has 192.168.0.254?  Tell 192.168.1.70 
53  33.577890 00:00:0c:4f:2a:30 -> 52:54:00:11:22:33    ARP 60   192.168.0.254 is at 00:00:0c:4f:2a:30

In this case it is clear that, from eth0's point of view, packet 52 is outgoing and 53 is the incoming reply. There are some situations though, where this is not completely obvious:

58  22.252109 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has 192.168.0.254?  Tell 192.168.1.70
59  22.252202 52:54:00:11:22:33 -> ff:ff:ff:ff:ff:ff    ARP 42   Who has 192.168.0.254?  Tell 192.168.1.70
60  22.254918 00:00:0c:4f:2a:30 -> 52:54:00:11:22:33    ARP 60   192.168.0.254 is at 00:00:0c:4f:2a:30

In the above example, we could assume that both 58 and 59 were outgoing packets, but we'd be wrong. Although size 42 suggests that it has not been padded to ethernet's minimal frame size, frame number 59 is not really coming from the "external network". One hint is that arp requests are sent with a second interval between each request, so it'd be unlikely that the VM is the creator of the second packet too. So where is 59 coming from? It turns out that with SR-IOV enabled, some cards' onboard switch loops packets back. Why is that a problem? Glad you asked.

When the Linux bridge sees packet 59, it records the mac-address 52:54:00:11:22:33 as coming from eth0, and not from the locally connected vnet0 tunnel. When packet 60 arrives, the bridge will drop it because it believes the destination MAC address is on eth0.

Long story short, in order to troubleshoot these kinds of issues, I know of three ways to be able to see the direction of packets:

tcpdump

With a fairly recent tcpdump/libpcap you can specify the -P in|out|inout option and capture traffic in a specific direction. In a situation like the one described here, it will be a bit cumbersome as you will need two separate tcpdump instances, but it works.

netsniff-ng

netsniff-ng can do an incredible number of cool things. Amongst others, it shows the direction of packets by default:

< em1 60 1400412101s.907918291ns 
 [ Eth MAC (00:00:24:cc:27:40 => 2c:41:38:ab:99:e2), Proto (0x0806, ARP) ]
 [ Vendor (CONNECT AS => Hewlett-Packard Company) ]
 [ ARP Format HA (1 => Ethernet), Format Proto (0x0800 => IPv4), HA Len (6), Proto Len (4), Opcode (1 => ARP request) ]
 [ Chr .................. ]
 [ Hex  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ]

> em1 42 1400412101s.907936759ns 
 [ Eth MAC (2c:41:38:ab:99:e2 => 00:00:24:cc:27:40), Proto (0x0806, ARP) ]
 [ Vendor (Hewlett-Packard Company => CONNECT AS) ]
 [ ARP Format HA (1 => Ethernet), Format Proto (0x0800 => IPv4), HA Len (6), Proto Len (4), Opcode (2 => ARP reply) ]

pktdump

pktdump is the most user-friendly of the three. It is not in Fedora, but I've build a COPR repo here. Here's an example output:

# pktdump -i em1 -f 'arp'
Capturing packets on the 'em1' interface
[12:24:07] RX(em1) : ARP| REQUEST 00:00:24:CC:27:40 192.168.0.254 00:00:00:00:00:00 foo.int.
[12:24:07] TX(em1) : ARP| REPLY 2C:41:38:AB:99:E2 foo.int 00:00:24:CC:27:40 192.168.0.254

It's especially useful when trying to follow the route packets are taking in a complex multi-interface setup.