High Speed Packet Capture Using Packet MMAP technique


Traditional packet capture approaches uses libpcap - the packet capture library. The capture process using libpcap for high speed network traffic is very inefficient because of the following reasons:

The Ring Buffer Structure

A ring buffer contains many blocks onto which the packets are written. Each block is a continuous region of physical memory. A block contains many frames. Block sizes are always multiple of frame sizes. A frame is the smallest unit. A frame contains two parts - tpacket_hdr, data buffer. The tpacket_hdr is a struct which contains details about the packet and the status of this frame. It contains meta information about the packet like timestamp, snaplen of capture and most importantly, the offset to next packet header. It is defined in the header file linux/if_packet.h The data buffer is what which actually contains the packet that is sent over the network interface.

Memory Mapping and using circular ring buffer

Mapping is done by calling the conventional mmap function. The socket is configured to write the packets in a memory mapped ring buffer. Once the set up is ready, the user reads packets from the memory mapped buffer. This helps in avoiding packet copies and reducing the number of system calls for accessing the packets. The user reads the packets block wise. Once the packet starts arriving into the ring, the user can either continously keep checking for new packets or leave the work to operating system by using poll() utility.


Ring buffer associated with the socket can have three configurations using the setsockopt() function. A socket can have be of any of the three TPACKET versions. Version 1 is the default ring buffer version. A few advantages of using TPACKET_V3 is that polling works on a per block basis instead of per ring basis as in the earlier version. It is said that TPACKET_V3 utilizes 15 - 20% less CPU usage and has a higher packet capture rate.

For more details about other versions, one can read about them here

Setting up Packet Capture Interface

These are the main steps in setting up the capture process: - Create a socket - Allocate circular ring buffer - Memory map the allocatd ring buffer to user process - Wait for incoming packets and capture them - Process the packets as your requirements - Close the capture socket and deallocate all resources

Sockets are created using the socket() utility. We create an AF_PACKET socket. Once a socket it created, we set the TPACKET version as 3. The ring buffer is set using a call to setsockopt() function with the option PACKET_RX_RING and ring properties. Once the socket and the ring buffer is setup, we create a memory mapped region with the socket file descriptor and ring size as input arguments. The ring size is the number of blocks in the ring multiplied by size of block. These values are configurable. The socket is then binded to an interface and in case we are using multiple sockets for capture, we can use PACKET_FANOUT option to distributed packets across sockets.

The kernel initializes all blocks to TP_STATUS_KERNEL. It keeps a pointer to one of the blocks in the ring buffer (Starting at 0) and when the kernel receives a packet it puts it in the buffer and updates the status with at least the TP_STATUS_USER flag. Then the user can read the packet and once the packet is read the user must reset the status fields to TP_STATUS_KERNEL, so the kernel can again use that frame buffer. The kernel fills a block, and it increments (modulo the number of block) the block pointer.

To check for new packets, the user can use the poll() function which takes in the socket file descriptor (everything is a file in UNIX) and returns true when there are new packets. Otherwise, the user could also use their own implementation to check for new packets in the socket. An implementation of the packet capture technique can be found here