Testing eBPF Program Compatibility Across Kernels with LVH and GitHub Actions
GitHub Actions Meets Little VM Helper and eBPF
When you’re building production-grade eBPF, you have to make sure your program works on every kernel version you claim to support.
And the time-consuming (and slightly masochistic) approach is to manually spin up VMs for every kernel version your eBPF program needs to support, then test it on each one individually.
Quite unpractical, right?
But now what if I tell you, that you can automate all of this using GitHub Actions and Little VM Helper (LVH).
🚀 Special thanks to Jatin Kumar, Software Engineer at StepSecurity, for putting together this deep and practical guest post for eBPFChirp!
In an earlier post we talked about writing portable eBPF.
In short, we embedded BTF information for all the different kernels into our binary, and conceptually it should work on all of them.
But as to whether you want to truly assure yourself this is the case and that your eBPF program actually works on all these different kernels, the only way is actually running them — at least for the kernel versions your program is bound to support.
And ideally we’d want to do this fast, so we can iterate the changes in case there are any failures along the way.
With a Little VM Helper (LVH) this can be nicely automatized in the CI/CD pipelines.
In short, LVH is a VM management tool, aimed for testing and development of features that depend on the kernel, such as eBPF.
It is maintained by Cilium and used by projects like Cilium, Tetragon, and pwru.
While it may sound complicated to set up — quite the opposite.
LVH has a Github Action that can easily be reused in your workflows to launch a (little) VM with a desired Linux Kernel image.
Here’s the example GitHub Repository — fork it any try it running it yourself.
As you can see at the top of the script, you define the list of Kernel images, that get passed down to the image-version parameter of the LVH GitHub Action step.
But where can you find the available images?
These kernel images (used by LVH) are built using another open-source cilium project named little-vm-helper-images and are stored in this registry.
As of now 3 variants of images are available — namely:
base — the stripped-down root image LVH uses for tasks such as Tetragon unit-test VMs.
kind — the image LVH uses when spinning up KinD clusters in Cilium’s end-to-end connectivity tests.
complexity-test — the image Cilium’s “Datapath BPF Complexity” CI workflow boots to run eBPF verifier complexity checks.
In essence, the three images mostly vary just in the userspace dependencies.
Base includes just minimal core utilities, kind builds on that with containerd, Kubernetes and common networking helpers (so tools like dig are present), while complexity-test layers in a full build tool-chain plus tracing, perf and BPF-debugging packages for verifier-stress tests.
And that’s it - you are ready to ship some eBPF 🐝
⏪ Did you miss the previous issues? I'm sure you wouldn't, but JUST in case:
I hope you find this resource helpful. Keep an eye out for more updates and developments in eBPF in next week's newsletter.
Until then, keep 🐝-ing!
Warm regards, Teodor and Jatin