Initially I was going to write the next instalment of the Linternals: Virtual Memory series after getting back from HITB2022SIN, but after a number of offline and online conversations it seems like this could help a number of you out, so let's give it a go!
My aim for this post is to provide some insights into getting into Linux kernel vulnerability research and exploit development (VRED), although I'm sure some of this will be transferable to similar areas.[1]
Sounds fairly straightforward, right? Well, much like the process of writing a kernel exploit, diving into this can also open-ended and confusing. There are many approaches and a wealth of resources out there, with no clearly defined path to follow.
Is this post going to pave that clearly defined path? Probably not. We all learn in different ways, have different experiences, motivations and goals. Hopefully, however, I can help demystify this topic a bit for you and give you the tools necessary to pave the right path for you.
Contents
- For a more general post on demystifying security research, I absolutely recommend a post of the same title by Alex Plaskett here, which touches on similar themes
Overview
As I mentioned above, linux vred is a complex and constantly evolving topic. So as you might imagine, trying to write an accessible, usable introduction to this topic has it's own challenges. But we gotta try!
The first thing I want to cover is mindset. Yeah, I get it, sounds wishy-washy and inactionable, but I think it will help to talk a bit about some useful mindset tips for approaching work like this and avoiding burnout.
Then I'll move onto talking about approaches you can take to begin your journey down the rabbit hole that is linux vred and hone your skills. Again, worth highlighting here that these are just suggestions from my experiences and are non-exhaustive.
I'll briefly touch on my workflow and some of the tooling I find useful, again this is really personal preference, but may be helpful as a starting point. Plus I always find it interesting to hear what cool tools and workflows other people use!
Finally I'll wrap things up with a list of resources, this will be far from exhaustive as well, but hopefully I'll get a decent amount of stuff in there!
Mindset
Motivation
At risk of sounding like one of those YouTube motivational speakers, one of the first things you want to understand is your motivation for getting into this.
Whatever your motivation, it's important to go into this with the understanding that this is a long journey, you (probably) won't be pwning kernels overnight! In fact, you'll never understand everything. There will be many "failures" and hurdles along the way.
But that's okay! Actually, it's more than okay, that means you're (probably) doing it right! Though, I'd be lying if I said this cycle of learning and "failure" with the occasional success wasn't a magnet for burnout and motivational humps.
However, by understanding your motivations and goals, as well as what you're getting into, these motivational humps can be more manageable and infrequent.
In terms of managing these humps, try where possible to prioritise working on things you enjoy and are interested in. Not only will it be better for your mental health, but you'll also likely find yourself more productive.
Due to the open-ended and exploratory nature of vred, you're not gonna have a good time trying to innovate and seek out solutions if you're completely unmotivated to do so. For the same reason, having some structure and milestones associated with tasks also helps prevent feelings of aimless drifting or getting overwhelmed.
Like I said though, these humps aren't always avoidable and are managed differently by different people, so I won't pretend to know the answers. For example, a common recommendation, and one I use, is to remember to context switch!
If you've been bashing your head against the keyboard for some months, neck-deep in C source code trying to find a particular primitive, sometimes it can help to take a pause. Go write that Python tool you've been meaning to. No, you won't forget everything. In fact, you may come back with a fresh perspective and clear mind.
Curiosity
Curiosity may have killed the cat, but it's a security researcher's best friend. Especially starting out, it can be tempting to rush to popping that shell.
Trust me, I've been guilty of it many a time. You're just starting out and trying a kernel CTF and you just want to get that flag to prove you can do it, right? So you Google some techniques and you copy and paste some code, tweak some stuff and keep iterating until you get it.
But as Emerson said, "It's not the destination, it's the journey". More important than popping the shell, is understanding how you popped it. The former may be a win here, but it's that deeper understanding which will net you future wins.
Be curious! Ask questions! Take your time. If you don't quite understand this technique you've seen, spend some time playing around with it until you do. If something isn't working, spend some time getting to the root cause rather than jumping straight to another approach.
This fundamental understanding you'll develop by being curious is a lot more flexible and applicable to future projects than a surface level awareness of potential techniques or approaches.
Perseverance
I've touched on this a few times now: kernel VRED is both complex and open-ended. Not only is there no clear path to winning, sometime's there is no path at all.
There might not be a bug in that module you've been looking at or a way to elevate your privileges with that heap overflow. Again, that's okay, it's normal!
Being able to persevere in the face of regular hurdles and dead-ends is key. An important aspect of this is defining "success" and "failure". I've thrown the F word around a few times so far, and been mindful to put it in quotes.
Just because you've spent months searching for a bug in a kernel module and come up with nothing, doesn't mean you've failed. During that time you've likely deepened your understanding of the kernel, improved your workflow, come up with tooling etc.
All of these are things which can help you "win" going forward, so yes while perseverance is key when you hit these roadblocks and dead-ends, also try not to just see them as failures!
It's also worth noting, the flip side of this is knowing when to call it quits. Later in in the workflow section, I talk about having a gameplan for approaching vred tasks. Such that when you've exhausted your gameplan, you know it's time to move on.
Ego
"your idea or opinion of yourself, especially your feeling of your own importance and ability" - Cambridge Dictionary on "ego"
Ego plays a big role in our industry, and fortunately is something that is spoken about more these days. And no I'm not talking about inflated egos (yet), but imposter syndrome.
In the beginning, you may come into this field finding things extremely daunting and overwhelming. After all, the kernel is huge and complicated and there's so many super smart people out there publishing some amazing work!
For many of us, this feeling never goes away. Myself included! I recently did my first conference talk at HITB2022SIN, and I was anxious for weeks in the build up despite the topic being something I worked on for months and was super familiar with.
Part of this was to do with public speaking, but part was worrying about the quality and validity of my work in the eyes of peers. What if it was all horribly wrong?!
So this section is just to reassure that if you feel this, it's okay, you're not alone! While this is common, try not to let it get on top of you! My main advice here would be that the only person you should be comparing yourself with is yourself a year or so ago[1] :)
The flip side to this, of course, is that I think it's good to maintain a level of humility. This is a field that is constantly evolving and you'll never know it all. Furthermore, due to the complexity of some this stuff, you might not have a complete understanding. This is all okay, just be open, and happy even, to adjust that understanding.
- Totally arbitrary number of course, as you may have taken a break and been working in other areas, but you get the gist of what I mean
Approaches
Alright, let's move onto some hands on advice! Hopefully now I've instilled some of mindset involved in getting into kernel vred stuff, time to put it to good use!
As has been a running theme here, there's many different approaches to get stuck into this and we all approach learning in different ways. I've tried to provide a variety of options here, though this is far from an exhaustive list.
Feel free to experiment, mix-and-match and see what works best for you! To throw in my 10 cents: I have found hands-on projects by far the best method to develop a working understanding of new stuff, supplementing this with some reading.
Reading
Okay, so the bread-and-butter for learning about kernel vred stuff is going to be reading; there's a wealth of blog posts and publications out there on a range of topics.
Not sure what else to say about this, other than that the hardest part here is curating and finding these readings. Contributors can vary from hobbyists, professional research and academic research - all being hosted in different places by different people.
Beyond the customary "use Twitter" for your infosec needs, I've also included a link in the resources below to a great repo called Linux Kernel Exploitation maintained by @andreyknvl which contains a pretty thorough list of reading materials.
Coming into this, the amount of materials out there may be overwhelming. I'd just suggest starting with stuff immediately relevant to what you're working on/interested in. E.g. if you want to try write a local priv esc, then read some recent LPE write-ups.
Also remember curiosity and perseverance. Some/most/all of this stuff may be utter gibberish at first, and that's fine. Especially with VRED write-ups, each bug and exploit will have it's own specific nuances which will be foreign to even experienced folks reading them for the first time.
Just remember to take your time to pause and follow up each bit you don't understand, even if it leads you down another rabbit hole, until you can piece it together.
Also another disclaimer that not everyone who takes the time to share their work is a NYT best seller, graphics designer or native English speaker!
Videos
If you're more of a visual learner, the options are a bit more limited but not non-existent. Besides my GIFs and occasional diagrams, there is a reasonable amount of recorded conference talks available on YouTube.
Again, the problem here becomes trying to find which conferences to checkout for content, because some of these may not index well and may not have a tonne of views. In the Resources section below, I'll include a list of con channels to get you started.
I'm sure there's probably some great content creators out there pumping out videos, but as that's not my preferred media I'm afraid I can't help much there. If you know of any I can plug here who make vids on Linternals / VRED then @ me pls.
Projects
I feel like theory can only get you so far and if you're interested in doing some kernel vred, you're going to need to get your hands dirty at some point anyway!
By getting some hands on, you're able to put into practice the techniques and understanding you've gained from your research. Furthermore, sometimes the best way to understand something in the kernel is to get in the debugger and take a peak yourself.
However, it's one thing to be told "just get some hands on experience!" and another to actually know where to start, especially if you're completely new to this.
As a result, I'll include some ideas and starting points for potential projects here. You'll find the more you get into things, the more ideas you'll have for your own tooling or experiments as you go on:
- A core part of kernel vred is, of course, understanding the kernel, so one project idea could be try and write your own kernel driver and play around with some features (reading input for userspace via IOCTLs, allocating memory etc.)
- Follow along with exploit write-ups! Find a local privilege escalation write-up you like (maybe with source available) and try follow it along and get it running in a VM; again taking the time to understand the how's and why's of what's going on
- Taking this a step further, you could try the above without source or even without a write-up by looking at some CVE's. Alternatively, piggy-backing off of idea 1 you could write your own vulnerable driver and exploit that :)
- CTFs are of course another popular way to test your kernel vred mettle, and I'll provide some links in the resources to some below.
- Tooling! Writing tooling to improve your kernel vred workflow or even just to explore kernel internals can be great way to develop that fundamental understanding. Don't worry if you don't have ideas right now, trust me you will!
- Posting your own write-ups or analysis! When I started this blog, I actually never intended for anyone to see it, it was just a way to motivate myself to look into various topics and refine my understanding on them via writing accessible posts
Workflow
Now onto the less glamorous, but just as fundamental part: workflow. I appreciate this is highly preference based, so this is more for reference and because I also find it interesting to hear about other people's workflows.
Your workflow is something that will likely constantly evolve, refined over an iterative process of discovering new tools and deeper understanding of your own preferences, strengths and weaknesses. Don't be afraid to try new things! :)
Tooling
For my IDE, I use "a configuration framework for GNU Emacs" called Doom. It's very easy to setup (and tweak) and the default settings are pretty good. I actually found this project thanks to a great talk, "Kernel Hacking Like It's 2020" by Russell Currey.
If you're interested in finding out more about Doom Emacs, there's a cool playlist on YouTube to get you started by Zaiste Programming.
Another cornerstone of my workflow is virtualisation. Whenever I'm writing up a new exploit or doing some testing, I'll be spinning up a representative target VM[1]. My tool of choice here is QEMU; I find it to be lightweight and very flexible (and it's free and open-source!). Â
The last part of the tooling trifecta for me: the debugger. Perhaps unsurprisingly I'm regularly neck-deep in gdb[2]. Despite being quite literally older than me, it still holds up. That said, addons like hugys's GEF (GDB Enhanced Features) makes life easier.
Organisation
AKA Documentation. Yep, I said it. But no, I'm not talking about carefully curated and margin-tweaked executive reports or several hundred page long technical specifications.
I can't stress enough how much future you will thank yourself if you get into the habit of documenting your work early on. It doesn't have to be anything fancy, I just use markdown + git. Just make sure there's some semblance of order and that it's going to be easy for you to hunt down and refer back to later.
You will accumulate a lot of knowledge during your research and you won't be able to retain all of it, nor will all of it be immediately useful. But having it neatly documented and easy to reference means that when you have to go back to it, you can. It also just helps to reinforce knowledge and understanding too.
Whether it's coming back to a kernel module you've previously done work on and want a refresher, or if you found a heap shaping primitive in a previous CTF that would be a perfect fit for the one you're working on now - having notes to refer back to is a life saver.
Staying Up-To-Date
Another useful part of your workflow to consider is keeping up-to-date with the latest kernel gossip. It seems like every week there's a new write-up or poc dropping and it can be a lot to keep up with, especially when they're from all over the place.
When I first asked a colleague how they found all these papers and write-ups, they replied Twitter and I scoffed. Surely not? But lo and behold, several years later I can confirm Twitter is still probably the best means to find this kind of content. It is what it is.
Alternatively of course, you can try to curate your own feed (e.g. via RSS or Atom) from the sources themselves and use a reader to catch updates.
Another sources beyond blogs and news sites is of course mailing lists. Yep, they're still a thing. The one I mainly keep an eye on is oss-security which is where you'll find public disclosures for linux kernel stuff if they went through the CVD process.
Furthermore, if you want to get granular and you're looking for specific information don't be afraid to dive into commit history or the lkml.
Having a Gameplan
We're almost there, I promise! The last, but certainly not the least, aspect of the workflow I want to talk about is having a gameplan for approaching vred projects.
Whether it's vulnerability research or exploit development, we're dealing with inherently complex and open-ended problems, which may have no solution at all.
By approaching these problems with a structured methodology, we're able to breakdown what can seem a daunting and overwhelming task into manageable chunks. Also, if we get to the end of it and don't find that bug or pop that shell, we at least know we've tried our best and can take what we've learned and move onto the next task.
So instead of just diving into the problem and following each lead, I'd recommended figuring out a gameplan that works for you and trying to approach these problems in and ordered, methodical way.
Again, this is something that will vary from person to person, depending on how you work. It will also likely evolve over time as you do more of this kind of work, and that's okay :)
For more concrete examples, I talk about this in my talk "E’rybody Gettin’ TIPC: Demystifying Remote Linux Kernel Exploitation". The recording isn't up yet, but you can see the slides here.
- Setting Up A Virtualised (Linux) Empire on Apple Silicon
- Patching, Instrumenting & Debugging Linux Kernel Modules
Resources
I'm already 3000 words deep so this resources section will be a work in process and is far from exhaustive. If you have additions, feel free to @ me or DM me and I'll get them in.
CTFs
- HTB has some kernel pwn challenges to practice your skills with
- smallkirby/kernelpwn seems like a decent curation of some kernel pwn challenges, with a section for beginners too :)
Reading Materials
- No need to reinvent the wheel, absolutely check out the awesome repo Linux Kernel Exploitation, maintained by @andreyknvl, containing a wealth of papers and write-ups
- linux-insides by 0xAX is a great low-level dive into some linux internals and was an inspiration for my own linternals series :)
- LWN.net
- sam4k/linux-kernel-resources is my attempt to curate some useful kernel tidbits related to compiling, debugging, instrumenting and patching the linux kernel
Tools
- bootlin's elixr cross referencer for linux source; great for browsing different kernel versions with references & defs in the browser
- Doom Emacs, my current IDE setup
- hugsy/gef (GDB Enhanced Features), an addon for GDB to improve RE/xdev workflow
Video Materials
- Black Hat (YouTube)
- DEFCONConference (Youtube)
- Hack In The Box Security Conference (YouTube)
- OffensiveCon (YouTube)
Wrapping Up
Oof, that was a long one, huh? Unlike my other posts, this one has covered a particularly subjective topic. Typically the content of my posts is derived from some objective source like the Linux kernel, however this one has ultimately been the culmination of my own experiences, understanding and journey into linux vred.
That said, I hope that at least some of the insights I've shared today have been useful for you. Not everything I've talked about will apply to everyone, but fingers crossed there's some helpful nuggets of information in there for each of you.
As always, I love to talk about this stuff and it means a lot to be able to help inspire and motivate people on their linux-y vred-y journeys. If you have any questions, suggestions or corrections then feel free to @ me or DM me on Twitter :)
Change History
I have a feeling this will see some updates and additions, so stay tuned here for any updates to the post.
exit(0);