Patching, Instrumenting & Debugging Linux Kernel Modules

So not long ago I found myself having to test a fix in a Linux networking module as part of the coordinated vulnerability disclosure I posted about recently.

Maybe my Google-fu wasn't on point, but it wasn't immediately clear what the best approach was, so hopefully this post can provide some direction for anyone interested in quickly patching, or instrumenting, Linux kernel modules.

Now, if we're talking about patching and instrumentation in the Linux kernel, I'd be remiss not to at least touch on some debugging basics as well, right? So hopefully between those three topics we should be able to cover some good ground in this post!

❗

This post ended up being quite long, so if you like a narrative and hearing the why behind the how, please continue! But for brevity I've also included the essentials my repo over at sam4k/linux-kernel-resources.

Preamble
- Kernel Module?
Getting Setup
Getting Stuck In
FAQ
Postamble

Preamble

This post is written in the context of kernel security research, which might deviate from other use cases, so bear that in mind when reading this post.

When finding a vuln, or looking into an existing bug, I'll want to set up a representative environment to play around with it. This basically just means setting up an Ubuntu VM (representative of a typical in-the-wild box) with a vulnerable kernel version.

The only real hard requirement I assume, is that you're doing your kernel stuff in a VM; as this'll make debugging the kernel a lot easier down the line.

Kernel Module?

In the early, early days (<1995) the Linux kernel was truly monolthic. Any functionality needed to be built into the base kernel at build time and that was that.

Since then, Loadable Kernel Modules (LKMs) have improved the flexibility of the Linux kernel, allowing features to be implemented as modules which can either be built into the base kernel or built as separate, loadable modules.

These can be loaded into, and unloaded from, kernel memory on demand without requiring a reboot or having to rebuild the kernel. Nowadays LKMs are used for device drivers, filesystem drivers, network drivers etc.

Getting Setup

Alright, let's get things setup shall we? In this section I'll talk about how to get to a position where we're able to make changes to a kernel module, rebuild it and install it.

There are probably a lot of different ways to do this - some quicker, some hackier and some context specific. While I'll touch on some shortcuts in the next section, in my experience the easiest way to avoid a headache is just starting from a fresh kernel build.

So that's what we're going to do! Buckle up, let's see if I can keep this brief. First I'll quickly cover how to build the kernel and then move onto patching specific modules.

Building The Kernel

First things first, make sure you grab the necessary dependencies for building the kernel:

$ sudo apt-get install git fakeroot build-essential ncurses-dev xz-utils libssl-dev bc flex libelf-dev bison dwarves

With that sorted, download the kernel version you're wanting to play with from kernel.org:

$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.17.tar.xz

If you're not sure what kernel version to go for, just pick one closest to your current environment, which you can check via the cmd uname -r; don't worry about patch versions or anything past the first two number, we ain't got no time for that.

Next let's extract the kernel source into our current dir and cd into it after:

$ tar -xf linux-5.17.tar.xz && cd linux-5-17

Now we need to configure our kernel. The kernel configuration is stored in a file named .config, in the root of the kernel source tree (aka where we just cd'd into).

On Debian-based distros you should be able to find your config at /boot/config-$(uname -r) or similar; on my Arch box it's compressed at /proc/config.gz:

$ cp /boot/config-$(uname -r) .config

This file contains all the configuration options for your kernel; if you want to play around with these you can use make menuconfig to tweak your config. Speaking of tweaking your config, you may want to make some changes:

On Ubuntu, you'll likely encounter some key related issues if you try to build using their config, so set the following config values in your .config: CONFIG_SYSTEM_TRUSTED_KEYS="", CONFIG_SYSTEM_REVOCATION_KEYS=""
Given there may be patching and debugging involved down the line, it might be worth taking the opportunity to enable debugging symbols with CONFIG_DEBUG_INFO=Y && CONFIG_GDB_SCRIPTS=Y ; you can enable this easily by using the helper ./scripts/config -e DEBUG_INFO -e GDB_SCRIPTS

With .config ready, let's crack on. By using oldconfig instead of menuconfig we can avoid the ncurses interface and just update the kernel configuration using our .config (it just means we may get some prompts during the make process for new options):

$ make oldconfig

Now we're ready to start building the kernel, and depending on your system and the .config we've copied over, this can take a while, so fire all CPU cores:

$ make -j$(nproc)

Next up, we can start installing our freshly built kernel. First up are the modules, which will typically be installed to /lib/modules/<kernel_vers>. So, to install our modules we'll go ahead and run:

$ sudo make modules_install

Finally we'll install the kernel itself; the follow command will do all the housekeeping required to let us select the new kernel from our bootloader:

$ sudo make install

And voila! Just like that we've built our Linux kernel from source, nabbing the config from our current environment, and we're ready to do some tinkering!

Module Patching

Okay, now we have a clean environment to work with and can start tinkering! Because we've built the kernel from source, we know we're building our patched modules in the exact same development environment as the kernel we're installing them into.

While the initial build can be lengthy, it's straightforward and we avoid the headache of out-of-tree module taints, signing issues and other finicky version-mismatch related issues.

Instead, we can make whatever changes we intend to make to our module and then run much the same commands we did during the initial install, only targeting our patched module(s). For example, for CVE-2022-0435 I tested a patches in net/tipc/monitor.c, so to rebuild and install my patched module I'd simply run:

$ make M=net/tipc
$ sudo make M=net/tipc modules_install

I'm then able to go ahead and re/load tipc and we're good to go! Easy as that.

Shortcuts & Alternatives

As some of you may already be painfully aware, building a full-featured kernel can actually take some time, especially in a VM with limited resources.

Minimal Configs

So to speed things up dramatically, if you're familiar with the module(s) you're going to be looking at, a more efficient approach is to start from a minimal config and enable the bare minimum features required for your testing environment.

For example $ make defconfig will generate a minimal default config for your arch, and then you can use $ make menuconfig to make further adjustments.

Skip Building Altogether

Depending on your requirements, you can just avoid building altogether:

if you just want to do some debugging, you could pull debug symbols from your distribution repo (see section on symbols below)
you may be able to fetch source from your distro repos, where you can then patch and build modules from there
if you don't need to worry about module signing/taint, and you're happy to get messy, there's hackier ways to do all this too

Getting Stuck In

Now that we've got our kernel dev environment setup, it's time to get stuck in! I'll briefly touch on generating patches, because why not, and instrumentation (though I'm not as familiar with this topic) before finally covering how we can debug kernel modules.

Patch Diffs

Disclaimer, if you want to submit any patches to the kernel formally, then definitely check out this comprehensive kernel doc on the various dos & donts of submitting patches.

That said, we're just playing around here! Plus I don't think it actually mentions the command in that particular doc. Anyway, I digress, we can run the following commands to generate a simple patch diff between two files:

$ diff -u monitor.c monitor_patched.c 
--- monitor.c   2021-03-11 13:19:18.000000000 +0000
+++ monitor_patched.c 2022-04-06 19:25:27.449661568 +0100
@@ -503,8 +503,10 @@
        /* Cache current domain record for later use */
        dom_bef.member_cnt = 0;
        dom = peer->domain;
-       if (dom)
+       if (dom) {
+               printk("printk debugging ftw!\n")
                memcpy(&dom_bef, dom, dom->len);
+       }
 
        /* Transform and store received domain record */
        if (!dom || (dom->len < new_dlen)) {

Where -u tells diff to use the unified format, which provides us with 3 lines of unified context (this is the standard, but N lines of context can be specified with -u N).

This unified format provides a line-by-line comparison of the given files, letting us know what's changed from one to another:

Line 2 is part of the patch header, prefixed with ---, and tells us the original file, date created and timezone offset from UTC (thanks @kfazz01!)
Line 3 is also part of the header, prefixed with +++, and tells us the new file, date created andtimezone offset from UTC (thanks @kfazz01!)
Line 4, encapsulated by @@, defines the start of "hunk" (group) of changes in our diff; sticking to - for original and + for new, -503,8 tells us this hunk is starting from line 503 in monitor.c and shows 8 lines. +503,10 means the hunk also starts from line 503 in monitor_patched.c but shows 10 lines (which checks out as we removed 1 and added 3).
Lines 5-7 & 13-15 are our 3 lines of unified context, just to give us some idea of what's going on around the lines we've changed
Lines 8-12 then are, by process of elimination, the lines we've changed. Changing things up, now - prefixes lines we've removed (i.e in monitor.c but no longer in monitor_patched.c) and + prefixes lined we've added to monitor_patched.c

So there's a quick ramble on patch diffs. It's as easy as that. We can also do diffs on entire directly/globs of files:

$ diff -Naur net/tipc/ net/tipc_patched/

Where -N treats missing files as empty, -a treats all files as text, -r recursively compares subdirs and -u is the same as before.

If we want to save these patches and apply them down the line, we can redirect the output into a file and then apply it to the original:

$ diff -u monitor.c monitor_patched.c > monitor.patch
$ patch -p0 < monitor.patch 
patching file monitor.c

When we pass patch a patch file, it expects and argument -pX where X defines how many directory levels to strip from our patch header. Our was like --- monitor.c, so we include -p0 as there's 0 dir levels to strip!

Instrumentation

Memes aside, printf() does the job in your own C projects, printk() is just the kernel-land equivalent^[1] and sometime's a cheeky printk("here") is all you need.

Using the patching approach we mentioned above, sometimes the easiest way to debug or trace execution isn't to set up some complication framework but simply to sprinkle in some printk()'s and rebuild your module and voila!

And well, that's the extent of my practical kernel instrumentation knowledge. But I'd feel bad making a whole section just to meme printk(), so while I can't expand on them fully, here are a couple of other avenues for kernel instrumentation:

kprobes

kprobes enable you to dynamically break into any kernel routine and collect debugging and performance information non-disruptively. You can trap at almost any kernel code address, specifying a handler routine to be invoked when the breakpoint is hit — kernel.org/doc

kprobes provide a fairly comprehensive API for your instrumentation needs, however the flip side is that is does require some light kernel development skills (perhaps a good intro task to kernel development??) to get stuck in.

ftrace

ftrace, or function tracer, is "an internal tracer designed to help out developers and designers of systems to find what is going on inside the kernel [...] although ftrace is typically considered the function tracer, it is really a frame work of several assorted tracing utilities." ^[2].

ftrace is actually quite interesting, as unlike similarly named (but not to be confused) tools like strace, there is no usermode binary to interact with the kernel component. Instead, users interact with the tracefs file system.

For the sake of brevity, if you're interested in checking out ftrace, here is an introductory guide by Gaurav Kamathe on opensource.com:

eBPF??

Okay, this might be a bit of a rogue one. Quick disclaimer being I've unfortunately not found the time, despite it being high up on my list, to properly play with eBPF. So touch any statements RE eBPF features with a pinch of salt!

That said, to summarise (I think I got this bit right), eBPF is a kernel feature introduced in 4.x that allows privileged usermode applications to run sandboxed code in the kernel.

I'm particularly interested in seeing the limits of its application, particularly in spaces such as detection, rootkits and debugging; for something original focused around networking.

Although, RE instrumentation & debugging, I'm not sure how much extra mileage eBPF would be able to provide. The eBPF bytecode runs in a sandboxed environment within the kernel, and as far as I'm aware can't alter kernel data.

That said, from a instrumentation perspective we can still do some interesting tracing. For example, we can attach to one of our kprobes and read function args & ret values.

Anyway, perhaps just some food-for-thought, but I'll stop rambling! I'll drop a couple of links below to existing publications on eBPF instrumentation/debugging ^[3].

The reason it's printk(), and not the classic printf() we usually find in C, as the C standard library isn't available in kernel mode; so the k in printk() let's us know we're using the kernel-land implementation.
https://www.kernel.org/doc/Documentation/trace/ftrace.txt
Debugging Linux issues with eBPF (USENIX LISA18)
Kernel analysis using eBPF

Debugging

Working with something as complex as the Linux kernel, you'll inevitably find yourself resonating with the above gif, and that's alright! That said, getting a smooth debugging workflow setup can go a long ways to alleviating the confusion.

Setting up good debugging environment means you can set breakpoints, allowing you to pause kernel execution at moments of interest, as well as inspect, and even change, registers and memory! There's also scope for scripting various elements of this process too.

GDB Debugging Stub

Remember about 2000 words ago I mentioned the only real assumption I was going to make is that you're doing your kernel testing/shenanigans in a VM?

It turns out that trying to debug the kernel you're running is... tricky. So besides snapshots and various other QoL features, a big pro to using VMs is the ability to remotely debug them at the kernel-level from our host (or another guest) using a debugger^[1].

The debugger in question, gdb, or the GNU Project debugger^[1], is a portable debugger that runs on many UNIX-like systems and is basically the defacto Linux kernel debugger (@ me).

Thanks to gdbstubs^[2], sets of files included by the virtualisation software (VMWare, QEMU etc.) in guests, we're able to remotely debug our guest kernel with much the same functionality we'd expect from userland debugging: breakpoints, viewing/setting registers and memory etc. etc.^[3]

I'll use this opportunity to plug GEF (GDB Enhanced Features) cos let's not forget gdb is like 36 years old and your boy needs some colours up in his CLI. Beyond just colours, gef has a great suite of quality-of-life features that just make the debugging workflow easier.

❗

Note that future GDB snippets will be using GEF, definitely not in an attempt to convert you, so don't be scared by the `gef➤` prompt; it's all the same program.

Anyway, enough rambling, let's take a look at getting kernel debugging setup on our VM:

Enable the gdbstub on your guest^[4]; typically this will listen on an interface:port you specify on the host. E.g. QEMU by default listens on localhost:1234.
Now on your host, or another guest that can reach the listening interface on your host, you can spin up and gdb^[5] and connect:

$ gdb
...
gef➤ target remote :1234 
gef➤ # you can omit localhost, so just :1234 works too

And just like that, you're now remotely debugging the Linux kernel - awesome, right? Except if you've just fired up gdb and connected like the snippet above, you're probably seeing something like this:

gef➤  target remote :12345
Remote debugging using :12345
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0xffffffffa703f9fe in ?? ()
[ Legend: Modified register | Code | Heap | Stack | String ]
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── registers ────
[!] Command 'context' failed to execute properly, reason: 'NoneType' object has no attribute 'all_registers'
gef➤  info reg
...
rip            0xffffffffa703f9fe  0xffffffffa703f9fe

Huh, so we've connected and it looks like we've trapped execution at 0xffffffffa703f9fe, but gdb has no idea where we are... This does not bode well for a productive debugging session; so let's look at how to fix that!

vmlinux, symbols & kaslr

So although our gdb has managed to make contact with the gdbstub on our guest, it's far from omnipotent. It can interact with memory and read the registers, as it understands the architecture, however it doesn't know about the kernel's functions and data structures.

Unfortunately for us that's the whole reason we're doing kernel debugging, to debug the kernel! Luckily though, it's fairly simple to tell gdb everything it needs to know.

If you've you've read my Linternals: The (Modern) Boot Process [0x02], you'll know that there's file called vmlinux containing the decompressed kernel image as a statically linked ELF. Just like debugging a userland binary, we can load this vmlinux into gdb and it's able to interpret it without any dramas.

Importantly, though, just like userland debugging we want to make sure we load a vmlinux with debugging symbols included, there's a couple options for this:

If you're building from source, just include CONFIG_DEBUG_INFO=y and optionally CONFIG_GDB_SCRIPTS=y and you'll find your vmlinux with debug symbols in your build root (see compiling/README.md for more info on building)
- ./scripts/config -e DEBUG_INFO -e GDB_SCRIPTS will enable these in your config with minimal fiddling
If you're running a distro kernel, you can check your distro's repositories to see if you can pull debug symbols
- On Ubuntu, if you update your sources and keyring [1], you can pull the debug symbols by running $ sudo apt-get install linux-image-$(uname -r)-dbgsym and should find your vmlinux @ /usr/lib/debug/boot/vmlinux-$(uname-r)

And just like that, we're done! jk, there's one more common gotcha (that I always forget) and that's KASLR: Kernel Address Space Layout Randomization. As it sounds, this randomizes where the kernel image is loaded into memory at boot time; so the address gdb reads from the vmlinux will naturally be wrong...

You can either add nokaslr to your boot options, typically via grub menu at boot
Or by editing /etc/default/grub and including nokaslr in GRUB_CMDLINE_LINUX_DEFAULT

After that we really are ready, and can repeat the steps from before, remember to also load our vmlinux with gdb:

$ gdb vmlinux
...
gef➤  target remote :12345 
... 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── threads ────
[#0] Id 1, stopped 0xffffffff81c3f9fe in native_safe_halt (), reason: SIGTRAP
[#1] Id 2, stopped 0xffffffff81c3f9fe in native_safe_halt (), reason: SIGTRAP
[#2] Id 3, stopped 0xffffffff81c3f9fe in native_safe_halt (), reason: SIGTRAP
[#3] Id 4, stopped 0xffffffff81c3f9fe in native_safe_halt (), reason: SIGTRAP
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── trace ────
[#0] 0xffffffff81c3f9fe → native_safe_halt()
[#1] 0xffffffff81c3fc4d → arch_safe_halt()
[#2] 0xffffffff81c3fc4d → acpi_safe_halt()
[#3] 0xffffffff81c3fc4d → acpi_idle_do_entry(cx=0xffff88810187d864)
[#4] 0xffffffff816e4201 → acpi_idle_enter(dev=<optimized out>, drv=<optimized out>, index=<optimized out>)
[#5] 0xffffffff8198e56d → cpuidle_enter_state(dev=0xffff888105a61c00, drv=0xffffffff8305dfa0 <acpi_idle_driver>, index=0x1)
[#6] 0xffffffff8198e88e → cpuidle_enter(drv=0xffffffff8305dfa0 <acpi_idle_driver>, dev=0xffff888105a61c00, index=0x1)
[#7] 0xffffffff810e7fa2 → call_cpuidle(next_state=0x1, dev=0xffff888105a61c00, drv=0xffffffff8305dfa0 <acpi_idle_driver>)
[#8] 0xffffffff810e7fa2 → cpuidle_idle_call()
[#9] 0xffffffff810e80c3 → do_idle()
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
gef➤

Awesome! Now gdb knows exactly where we our, and gef provides us lots of useful information in it's ctx menu, which you can always pop up with the ctx command.

I've cut it off for brevity but we can at a glance see sections (might need to scroll right for the headings) for registers, stack, code, threads and trace!

On top of that, as I'll touch on in Misc GDB Tips below, we're able to explore all the kernel structures and more thanks to the symbols we now have.

Loadable Modules

As a quick aside, you might find out that some symbols for certain modules are missing, despite doing all that vmlinux faff above. This is because not all modules are compiled into the kernel, some are compiled as loadable modules.

This means that the modules are only loaded into memory when they're needed, e.g. via modprobe. We can check if a module is loaded in our .config:

CONFIG_YOUR_MODULE=y defines an in-kernel module
CONFIG_YOUR_MODULE=m defines a loadable kernel module

For loadable modules, we need to do a couple of extra steps, in addition to those above, in order to let gdb know about these symbols:

Copy the module's your_module.ko from your debugging target; try /lib/modules/$(uname -r)/kernel/
On your debugging target, find out the base address of the module; try sudo grep -e "^your_module" /proc/modules
In your gdb session, you can now load in the module by (gdb) add-symbol-file your_module.ko 0xAddressFromProc - voila!

Sorted! Now the symbols from your_module should be available in gdb! Just remember that even with KASLR disabled, this address can be different each time you load the module, but you only need to grab the your_module.ko once at least.

Misc GDB Tips

Oof, well this post is already careening towards 4000 words (and I did this voluntarily, for fun?!), so I think I'll just link to my repository where you can find some useful gdb/gef commands for debugging the Linux kernel!

Other Stuff

As we're transitioning into a speedrun, congratulations to anyone who read the whole thing, I'll attempt to quickly touch on some other useful debugging resources:

drgn: remember earlier, when I said debugging the kernel your using can be tricky? Well drgn is an extremely programmable debugger, written in python (and not 36 years ago), that among other things allows you to do live introspection on your kernel. I still need to explore this more, but I wouldn't see it as a replacement for gdb for example, but a different tool for different goals.
strace: ah yes, our old friend, strace(1). The system call tracing utility can be useful for complimenting your kernel debugging by tracing the interactions between your poc/userland interface/program and the kernel. With minimal faff you can hone in on what kernel functions you may want to focus your debugging endeavours on.
procfs: another reminder about the various introspection available via /proc/; you saw earlier that we made use of /proc/modules. There's plenty to explore here.
man pages: don't sleep on the man pages! Although there isn't generally pages on kernel internals, the syscall section (2) can help with understanding some of the interactions that go on
source: due to word count concerns, oops, and the fact I never really use it, I haven't included adding source into gdb but that doesn't mean you can't have it up for reference! I always try to have a copy of source handy to explore, not to mention the documentation that's usually available somewhere in the kernel too

https://www.sourceware.org/gdb/
https://sourceware.org/gdb/onlinedocs/gdb/Remote-Stub.html
Future post idea? Dive into some debugging internals
https://github.com/sam4k/linux-kernel-resources/tree/main/debugging#gdb--vm
If your guest is a different architecture to your host, gdb needs to needs to know about it, so you'll need to install and use gdb-multiarch

FAQ

So this is a little bit of an experiment, and maybe more suited to the GitHub repo, but if anyone has any questions feel free to @ me on Twitter and I'll keep try keep this FAQ updated. Also, if anyone has any suggestions for FAQs, I'm happy to add those too :)

Postamble

Talk about feature creep, eh? We certainly covered a lot of ground in this post: from building the kernel to patching modules to setting up our debugging environment.

Hopefully some of this (or all!) have been useful, and maybe helped demystify things. As I briefly mentioned in the intro, I've included all the essentials in a github repository, which I'll continue to update with any useful Linux kernel resources/demos/shenanigans.

I think by nature of the work we do, as programmers and "hackers", a lot of times we find ourselves creating hacky solutions and shortcuts, then through some twisted process of natural selection some of these make their way into our workflow.

Though, perhaps because we consider them too niche or too messy, we often don't share these solutions or quick tricks and so the cycle continues. Is this necessarily a bad thing? Of course not! I love to tinker and believe me, I have many a bash script that should never see the light of day, but perhaps there's also a few that would help others if they did.

So really, this post is just a culmination of my own hacky, messy natural selection that has occurred during my time working on kernel stuff, so don't @ me if it's horribly wrong (DM me instead, pls help me), but hopefully there's some takeaways here that will inspire others to tinker and perhaps save some time in the process.

Obligatory @ me for any suggestions, corrections or questions!

exit(0);