Experiments with writing C eBPF code

Posted on 05/12/202208/04/2023 by Bernard Pietraga

This blog post is not affiliated with or otherwise sponsored by The Linux Foundation. eBPF logo is used following The Linux Foundation Brand Guidelines.

I sometimes work with high performance systems, space where nanoseconds matter. EBPF has a great use there. But mostly I work with higher layers, compiled probes, things like Cilium. Another place where EBPF shines is thread detection, Falco has ebpf-probe. I started to wonder how this thing is done on a lower level. You can write and extend EBPF programs yourself.

If you don’t know what the eBPF is, the official Linux Foundation page has great explanation -> What is eBPF?.

In short eBPF is a virtual machine that runs in the Linux kernel. It is designed to enable the execution of custom programs that can be used to monitor and manipulate network traffic.

For introduction into writing eBPF this series of Collabora blog posts by Adrian Ratiu is reasonable start.

While I prefer to work with higher level libraries for creating EBPF programs like Rust ecosystem redbpf this blogpost will focus on good’ol C. This way there are less layers to comprehend.

Why EBPF is fast?

EBPF is a technology that allows programs to be attached to sockets in the Linux kernel in order to monitor and filter network traffic.

EBPF programs can process network traffic in real-time, without the overhead of transitioning between user space and kernel space.

The programs are written in a low-level, machine-friendly language that is optimized for performance. This allows EBPF programs to be highly efficient and perform complex operations on network packets without sacrificing speed.The code gets compiled using a specialized compiler that generates optimized machine code for the Linux kernel.

Overall, EBPF is fast because it is designed for high-performance real-time processing of network traffic (but not limited to), and it takes advantage of low-level language, compiler, and kernel optimization to achieve this goal.

What is EBPF probe?

An EBPF probe is a program that is attached to a socket in the Linux kernel in order to monitor and filter network traffic. EBPF probes are designed to be fast and efficient, allowing them to process network traffic in real-time without sacrificing performance. They are typically used to monitor and filter network traffic in a variety of scenarios, including traffic mirroring, packet inspection, and intrusion detection.

Some downsides of going this route

Like everything in engineering this soltion eBPF comes with some downsides. I want to state before everything will become too nice and rosey.

EBPF bytecode is JIT
Requires elevated priviliges – see CAP_BPF in the Linux capabilies man page and bpf, this goes along with other capabilies, depending on your usecase like CAP_PERFMON, CAP_SYS_RESOURCE, CAP_SYS_PTRACE.
Probes run in kernel space. While it is fast it also introduces new issues potential depending on your threat model. Keep a note that eBPF in Kernel VM has limited instruction set it still can be exploited. See spectre attack. In my case I was fine with this.
Maintaining this solution varies from maintaing typical container workloads and is tricky to get right. You can still include code in container.

What doing here?

Here are the basic steps for creating traffic mirror EBPF probe

Install the necessary software and tools, including a C compiler.
Write a C program that specifies the conditions under which the probe should be triggered, and the actions that the probe should take when triggered.
Compile the C program to BPF bytecode.
Load the EBPF object file into the Linux kernel using the bpftool utility.
Use the bpftool utility to attach the EBPF probe to the desired socket in the Linux kernel.
Test if it works
Profit

Getting build environment ready

For this you need Linux, preferably in virtual machine for devlopment purposes. It needs to have BPF complied into kernel. You will most likely have it if you run modern kernel.

Next we need toolchain to build the EBPF bytecode which will be loaded into in Kernel VM. In this case clang compiler and libbpf. A lot of distribtions serve packages in their repository.

For development usecase the bcc is handy. Here is bcc install guide. It provides nice frontend to run programs from python.

I for production usecases I prefer to use the libbpf. It has BPF CO-RE (Compile Once – Run Everywhere) which means you do not need the Clang or LLVM runtime on the machine running probe which is different from bcc.

More about protability can be found on this blog post by Andrii Nakryiko’s

If you want to go with pure C without libbpf not bcc there is this awesome blogpost.

How to compile with LLVM and attach probes

To use eBPF to probe localhost traffic on port 3000 and mirror it to port 5000 we need small chunk of C code. Once you have written the eBPF program, you would then need to compile it and attach it to the desired network interface using the bpf command.

The ebpf program code is created using C. You can compile it using LLVM.

clang -Wall -arch bpf -c program-code.c -o bpf-objectfile.o

After you compile the eBPF program to bytecode objectfile, you can for example attach it to the desired network interface using the ip link set command:

sudo ip link set dev lo xdp obj bpf-objectfile.o

For ease of testing here we will use local BCC setup where we can just create python heredoc with C code and include it in python to compile and run. Make sure to run the code with sudo or needed linux capabilies, EBPF requres elevated priviliges.

Example programs

Here is a simple example of an EBPF programs. Just save them to files and include in python BCC compiler. The project provides great starting point examples:
https://github.com/iovisor/bcc/tree/master/examples

Here are some of my experiments with BPF. I based parts of my code on the examples above, everything has GPL license.

The code won’t compile in latest 6.0 kernel. It doesn’t have the uapi/linux/bpf.h.

No 1 – File creation in bpf

Let’s start with simple one. Program creating file in root folder, and saving hello world.

The create_file.c

#include 
#include 

#define FILE_PATH "/root/new_file.txt"

int ebpf_probe(struct CTXTYPE *ctx) {
  // Create the new file
  int fd = bpf_fs_open(FILE_PATH, O_WRONLY | O_CREAT, 0644);
  if (fd >= 0) {
    // Write some data to the file
    char data[] = "Hello World!\n";
    bpf_fs_write(fd, data, sizeof(data));
    bpf_fs_close(fd);
  }

  return 0;
}

And python runner file create_file.py.

from bcc import BPF
b = BPF(src_file="create_file.c")
b.load_func("create_file", BPF.SYSCALL)

You can run it with sudo python3 create_file.py

No 2 – HTTP hello world

Now funny one, if the network request is on port 9000, respond with HTTP 200 with hello world.

hello_world.c

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

char LICENSE[] SEC("license") = "GPL";

// BEGIN GPL CODE COPY
// https://android.googlesource.com/platform/external/bcc/+/HEAD/examples/networking/http_filter/http-parse-complete.c
struct Key {
        u32 src_ip;               //source ip
        u32 dst_ip;               //destination ip
        unsigned short src_port;  //source port
        unsigned short dst_port;  //destination port
};
// END GPL CODE COPY

int ebpf_probe(struct __sk_buff *skb) {
  u8 *cursor = 0;

  struct ethernet_t *ethernet = cursor_advance(cursor, sizeof(*ethernet));
  u16 ethertype = ethernet->type;

  if (ethertype == ETH_P_IP) {
    struct Key *ip = cursor_advance(cursor, sizeof(*ip));
    u16 src_port = ip->src_port;

    if (src_port == htons(9000)) {
      // Build the "200 OK" HTTP response
      char response[] = "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\nHello World!
";
      bpf_skb_store_bytes(ip, 0, response, sizeof(response), 0);
    }
  }

  return 0;
}

hello_world.py

from bcc import BPF
from pyroute2 import IPRoute

ip = IPRoute()
idx = ip.link_lookup(ifname="eth0")[0]

# load BPF program

b = BPF(src_file="hello_world.c")
fn = b.load_func("ebpf_probe", BPF.SOCKET_FILTER)
b.attach_raw_socket(fn, "eth0")

No 3 – Packet mirror

Let us mirror all the request going to port 3000 to port 5000. Just create a Python file and mount to the appropriate network interface.

from bcc import BPF
from pyroute2 import IPRoute

ip = IPRoute()
idx = ip.link_lookup(ifname="eth0")[0]

# define BPF program
prog = """
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

BPF_HASH(counts, u32, u64);

// BEGIN GPL CODE COPY
// https://android.googlesource.com/platform/external/bcc/+/HEAD/examples/networking/http_filter/http-parse-complete.c
struct Key {
        u32 src_ip;               //source ip
        u32 dst_ip;               //destination ip
        unsigned short src_port;  //source port
        unsigned short dst_port;  //destination port
};
// END GPL CODE COPY

int ebpf_probe(struct __sk_buff *skb) {
  u8 *cursor = 0;

  struct ethernet_t *ethernet = cursor_advance(cursor, sizeof(*ethernet));
  u16 ethertype = ethernet->type;

  if (ethertype == ETH_P_IP) {
    struct Key *ip = cursor_advance(cursor, sizeof(*ip));
    u16 src_port = ip->src_port;

    if (src_port == htons(3000)) {
      bpf_clone_redirect(ip, 5000, 0);
    }
  }

  return 0;
}
"""

# load BPF program

b = BPF(text=prog)
fn = b.load_func("ebpf_probe", BPF.SOCKET_FILTER)
b.attach_raw_socket("eth0", fn, 0)

No 4 – IP V4 address counter for incoming requests

Finally EBPF probe monitors incoming network traffic and tracks the number of packets received from each IP address.

If a device’s IP address receives more than one packet, the program prints out that it here that the address has exceeded a threshold.

#include 
#include 
#include 
#include 
#include 

BPF_HASH(counts, u32, u64);

int ebpf_probe(struct CTXTYPE *ctx) {
  u8 *cursor = 0;

  struct ethernet_t *ethernet = cursor_advance(cursor, sizeof(*ethernet));
  u16 ethertype = ethernet->type;

  if (ethertype == ETH_P_IP) {
    struct ip_t *ip = cursor_advance(cursor, sizeof(*ip));
    u32 src = ip->src;

    counts.increment(src, 1);

    u64 *value = counts.lookup(src);
    if (value && *value > 10) {
      bpf_trace_printk("IP address %d.%d.%d.%d has received more than 10 packets\n",
        src & 0xff, (src >> 8) & 0xff, (src >> 16) & 0xff, (src >> 24) & 0xff);
    }
  }

  return 0;
}