Getting Started with eBPF in Go

In this guide, we'll walk you through building a new eBPF-powered Go application from scratch. We'll introduce the toolchain, write a minimal eBPF C example and compile it using bpf2go. Then, we'll put together a Go application that loads the eBPF program into the kernel and periodically displays its output.

The application attaches an eBPF program to an XDP hook that counts the number of packets received by a physical interface. Filtering and modifying packets is a major use case for eBPF, so you'll see a lot of its features being geared towards it. However, eBPF's capabilities are ever-growing, and it has been adopted for tracing, systems and application observability, security and much more.

eBPF C program

Dependencies

To follow along with the example, you'll need:

• Linux kernel version 5.7 or later, for bpf_link support
• LLVM 11 or later ¹ (clang and llvm-strip)
• libbpf headers ²
• Linux kernel headers ³
• Go compiler version supported by ebpf-go's Go module

Let's begin by writing our eBPF C program, as its structure will be used as the basis for generating Go boilerplate.

Click the annotations in the code snippet for a detailed explanation of the individual components.

// (1)!
//go:build ignore

#include <linux/bpf.h> // (2)!
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY); // (3)!
    __type(key, __u32);
    __type(value, __u64);
    __uint(max_entries, 1);
} pkt_count SEC(".maps"); // (4)!

// count_packets atomically increases a packet counter on every invocation.
SEC("xdp") // (5)!
int count_packets() {
    __u32 key    = 0; // (6)!
    __u64 *count = bpf_map_lookup_elem(&pkt_count, &key); // (7)!
    if (count) { // (8)!
        __sync_fetch_and_add(count, 1); // (9)!
    }

    return XDP_PASS; // (10)!
}

char __license[] SEC("license") = "Dual MIT/GPL"; // (11)!

1. When putting C files alongside Go files, they need to be excluded by a Go build tag, otherwise go build will complain with C source files not allowed when not using cgo or SWIG. The Go toolchain can safely ignore our eBPF C files.

2. Include headers containing the C macros used in the example. Identifiers such as __u64 and BPF_MAP_TYPE_ARRAY are shipped by the Linux kernel, with __uint, __type, SEC and BPF helper definitions being provided by libbpf.

3. Declare a BPF map called pkt_count, an Array-type Map holding a single u64 value. See man bpf or the online bpf man pages for an overview of all available map types.
   
   For this example, we went with an array since it's a well-known data structure you're likely familiar with. In BPF, arrays are preallocated and zeroed, making them safe and ready to use without any initialization.

4. The Map definition is placed in the .maps ELF section, which is where ebpf-go expects to find it.

5. In BPF, not all programs are equal. Some act on raw packets, some execute within the context of kernel or user space functions, while others expect to be run against an __sk_buff. These differences are encoded in the Program Type. libbpf introduced a set of conventions around which ELF sections correspond to which type. In this example, we've chosen xdp since we'll attach this program to the XDP hook later.

6. There's only one possible element in pkt_count since we've specified a max_entries value of 1. We'll always access the 0th element of the array.

7. Here, we're asking the BPF runtime for a pointer to the 0th element of the pkt_count Map.
   
   bpf_map_lookup_elem is a BPF helper declared in docs.h. Helpers are small pieces of logic provided by the kernel that enable a BPF program to interact with its context or other parts of the kernel. Discover all BPF helpers supported by your kernel using man bpf-helpers or the online bpf-helpers man pages.

8. All Map lookups can fail. If there's no element for the requested key in the Map, count will hold a null pointer. The BPF verifier is very strict about checking access to potential null pointers, so any further access to count needs to be gated by a null check.

9. Atomically increase the value pointed to by count by 1. It's important to note that on systems with SMP enabled (most systems nowadays), the same BPF program can be executed concurrently.
   
   Even though we're loading only one 'copy' of our Program, accompanied by a single pkt_count Map, the kernel may need to process incoming packets on multiple receive queues in parallel, leading to multiple instances of the program being executed, and pkt_count effectively becoming a piece of shared memory. Use atomics to avoid dirty reads/writes.

10. XDP allows for dropping packets early, way before it's passed to the kernel's networking stack where routing, firewalling (ip/nftables) and things like TCP and sockets are implemented. We issue the XDP_PASS verdict to avoid ever interfering with the kernel's network stack.

11. Since some BPF helpers allow calling kernel code licensed under GPLv2, BPF programs using specific helpers need to declare they're (at least partially) licensed under GPL. Dual-licensing is possible, which we've opted for here with Dual MIT/GPL, since ebpf-go is MIT-licensed.

Create an empty directory and save this file as counter.c. In the next step, we'll set up the necessary bits to compile our eBPF C program using bpf2go.

Compile eBPF C and generate scaffolding using bpf2go

With the counter.c source file in place, create another file called gen.go containing a //go:generate statement. This invokes bpf2go when running go generate in the project directory.

Aside from compiling our eBPF C program, bpf2go will also generate some scaffolding code we'll use to load our eBPF program into the kernel and interact with its various components. This greatly reduces the amount of code we need to write to get up and running.

package main

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go counter counter.c

Using a dedicated file for your package's //go:generate statement(s) is neat for keeping them separated from application logic. At this point in the guide, we don't have a main.go file yet. Feel free to include it in existing Go source files if you prefer.

Before using the Go toolchain, Go wants us to declare a Go module. This command should take care of that:

% go mod init ebpf-test
go: creating new go.mod: module ebpf-test
go: to add module requirements and sums:
    go mod tidy
% go mod tidy

We also need to manually add a dependency on bpf2go since it's not explicitly imported by a .go source file:

% go get github.com/cilium/ebpf/cmd/bpf2go
go: added github.com/cilium/ebpf v0.11.0
go: added golang.org/x/exp v0.0.0-20230224173230-c95f2b4c22f2
go: added golang.org/x/sys v0.6.0

Now we're ready to run go generate:

% go generate
Compiled /home/timo/getting_started/counter_bpfel.o
Stripped /home/timo/getting_started/counter_bpfel.o
Wrote /home/timo/getting_started/counter_bpfel.go
Compiled /home/timo/getting_started/counter_bpfeb.o
Stripped /home/timo/getting_started/counter_bpfeb.o
Wrote /home/timo/getting_started/counter_bpfeb.go

bpf2go built counter.c into counter_bpf*.o behind the scenes using clang. It generated two object files and two corresponding Go source files based on the contents of the object files. Do not remove any of these, we'll need them later.

Let's inspect one of the generated .go files:

type counterPrograms struct {
    CountPackets *ebpf.Program `ebpf:"count_packets"`
}

Neat! Looks like bpf2go automatically generated a scaffolding for interacting with our count_packets Program from Go. In the next step, we'll explore how to load our program into the kernel and put it to work by attaching it to an XDP hook!

The Go application

Finally, with our eBPF C code compiled and Go scaffolding generated, all that's left is writing the Go code responsible for loading and attaching the program to a hook in the Linux kernel.

Click the annotations in the code snippet for some of the more intricate details. Note that we won't cover anything related to the Go standard library here.

package main

import (
    "log"
    "net"
    "os"
    "os/signal"
    "time"

    "github.com/cilium/ebpf/link"
    "github.com/cilium/ebpf/rlimit"
)

func main() {
    // Remove resource limits for kernels <5.11.
    if err := rlimit.RemoveMemlock(); err != nil { // (1)!
        log.Fatal("Removing memlock:", err)
    }

    // Load the compiled eBPF ELF and load it into the kernel.
    var objs counterObjects // (2)!
    if err := loadCounterObjects(&objs, nil); err != nil {
        log.Fatal("Loading eBPF objects:", err)
    }
    defer objs.Close() // (3)!

    ifname := "eth0" // Change this to an interface on your machine.
    iface, err := net.InterfaceByName(ifname)
    if err != nil {
        log.Fatalf("Getting interface %s: %s", ifname, err)
    }

    // Attach count_packets to the network interface.
    link, err := link.AttachXDP(link.XDPOptions{ // (4)!
        Program:   objs.CountPackets,
        Interface: iface.Index,
    })
    if err != nil {
        log.Fatal("Attaching XDP:", err)
    }
    defer link.Close() // (5)!

    log.Printf("Counting incoming packets on %s..", ifname)

    // Periodically fetch the packet counter from PktCount,
    // exit the program when interrupted.
    tick := time.Tick(time.Second)
    stop := make(chan os.Signal, 5)
    signal.Notify(stop, os.Interrupt)
    for {
        select {
        case <-tick:
            var count uint64
            err := objs.PktCount.Lookup(uint32(0), &count) // (6)!
            if err != nil {
                log.Fatal("Map lookup:", err)
            }
            log.Printf("Received %d packets", count)
        case <-stop:
            log.Print("Received signal, exiting..")
            return
        }
    }
}

1. Linux kernels before 5.11 use RLIMIT_MEMLOCK to control the maximum amount of memory allocated for a process' eBPF resources. By default, it's set to a relatively low value. See Resource Limits for a deep dive.

2. counterObjects is a struct containing nil pointers to Map and Program objects. A subsequent call to loadCounterObjects populates these fields based on the struct tags declared on them. This mechanism saves a lot of repetition that would occur by checking a Collection for Map and Program objects by string.
   
   As an added bonus, counterObjects adds type safety by turning these into compile-time lookups. If a Map or Program doesn't appear in the ELF, it won't appear as a struct field and your Go application won't compile, eliminating a whole class of runtime errors.

3. Close all file descriptors held by objs right before the Go application terminates. See Object Lifecycle for a deep dive.

4. Associate the count_packets (stylized in the Go scaffolding as CountPackets) eBPF program with eth0. This returns a Link abstraction.

5. Close the file descriptor of the Program-to-interface association. Note that this will stop the Program from executing on incoming packets if the Link was not Link.Pined to the bpf file system.

6. Load a uint64 stored at index 0 from the pkt_count Map (stylized in the Go scaffolding as PktCount). This corresponds to the logic in counter.c.

Save this file as main.go in the same directory alongside counter.c and gen.go.

Building and running the Go application

Now main.go is in place, we can finally compile and run our Go application!

% go build && sudo ./ebpf-test
2023/09/20 17:18:43 Counting incoming packets on eth0..
2023/09/20 17:18:47 Received 0 packets
2023/09/20 17:18:48 Received 4 packets
2023/09/20 17:18:49 Received 11 packets
2023/09/20 17:18:50 Received 15 packets

Generate some traffic on eth0 and you should see the counter increase.

Iteration Workflow

When iterating on the C code, make sure to keep generated files up-to-date. Without re-running bpf2go, the eBPF C won't be recompiled, and any changes made to the C program structure won't be reflected in the Go scaffolding.

% go generate && go build && sudo ./ebpf-test

What's Next?

Congratulations, you've just built your (presumably) first eBPF-powered Go app! Hopefully, this guide piqued your interest and gave you a better sense of what eBPF can do and how it works.

With XDP, we've only barely scratched the surface of eBPF's many use cases and applications. For more easily-accessible examples, see the main repository's examples/ folder. It demonstrates use cases like tracing user space applications, extracting information from the kernel, attaching eBPF programs to network sockets and more.

Follow our other guides to continue on your journey of shipping a portable eBPF-powered application to your users.

---

1. Use clang --version to check which version of LLVM you have installed. Refer to your distribution's package index to finding the right packages to install, as this tends to vary wildly across distributions. Some distributions ship clang and llvm-strip in separate packages. ↩

2. For Debian/Ubuntu, you'll typically need libbpf-dev. On Fedora, it's libbpf-devel. ↩

3. On AMD64 Debian/Ubuntu, install linux-headers-amd64. On Fedora, install kernel-devel.

   On Debian, you may also need ln -sf /usr/include/asm-generic/ /usr/include/asm since the example expects to find <asm/types.h>. ↩

---

Last updated 2023-11-24
Authored by Timo Beckers