Note: This post was originally written in 2019 and has been lightly refreshed for clarity. The content and examples are unchanged — just the presentation got a coat of paint.

This is part two of the AFL kernel fuzzing series. Part one covered the theoretical background and the modular kcov(4) AFL module. This post covers the practical side: porting AFL to work with NetBSD, validating coverage quality with a benchmark, and running FFS fuzzing with a pre-seeded corpus.

Porting AFL to NetBSD

AFL is primarily a user-space fuzzer. Its standard mode compiles the target with injected instrumentation via afl-as. For kernel fuzzing that does not apply directly — the instrumentation lives in the kernel, not in a binary AFL controls.

The approach here replaces AFL’s coverage source. Instead of reading data from compiled instrumentation, the fuzzing wrapper reads coverage data from kcov(4) via the AFL module described in Part 1. AFL sees the same 64 KB shared memory hash map it expects; it just does not know the counts came from the kernel.

One design decision: rather than modifying the binary being fuzzed, the wrapper runs as a shared library. AFL loads it, the library executes the mount/unmount cycle, and AFL mutates the filesystem image between cycles. This approach is more portable than patching the target binary and easier to adapt for remote setups where the kernel under test runs in a VM.

The port is on GitHub. The relevant changes are in the coverage source — the rest of AFL runs unchanged.

Validating Coverage with a Benchmark

Before fuzzing real targets, it is worth checking that coverage guidance actually helps. A coverage-blind fuzzer brute-forces the input space; a coverage-guided one should find interesting inputs faster. The test: can the fuzzer crack a six-byte secret?

The benchmark is a kernel character device — lottery_dev — that reads input and compares it against a hardcoded six-byte value. If the bytes match, the device panics; otherwise it returns zero. The number of possible inputs is 2⁴⁸ = 281,474,976,710,656. Brute force is not a strategy.

For comparison, a simple C program with the same six-byte comparison took AFL a few hours to crack on a laptop (running at roughly 10,000–40,000 exec/s for a userspace target — hardware and target complexity determine the actual rate). The coverage feedback guides the fuzzer toward the comparison boundary, progressively matching more bytes with each generation.

Running the same fuzzer against lottery_dev for nearly a week produced nothing. Something was wrong.

The Coverage Noise Problem

Running the wrapper with raw kcov(4) output (no AFL module, just collecting data) showed the problem immediately. After expanding the coverage buffer to fit all entries, the top twenty addresses by occurrence count were:

1544 /usr/netbsd/src/sys/uvm/uvm_page.c:847
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:869
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:890
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:880
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:858
1281 /usr/netbsd/src/sys/arch/amd64/compile/obj/GENERIC/./machine/cpu.h:70
1281 /usr/netbsd/src/sys/arch/amd64/compile/obj/GENERIC/./machine/cpu.h:71
 478 /usr/netbsd/src/sys/kern/kern_mutex.c:840
 456 /usr/netbsd/src/sys/arch/x86/x86/pmap.c:3046
 438 /usr/netbsd/src/sys/kern/kern_mutex.c:837
...

30,578 entries total. UVM page management and machine-dependent code dominated the trace. Every write to the character device dragged in the memory allocator, the page tables, the mutex subsystem. AFL saw thousands of constantly-changing entries from subsystems that had nothing to do with lottery_dev, and no signal from the code it was supposed to be exploring.

The fix should be __attribute__((no_instrument_function)) on the high-noise functions, telling GCC not to inject coverage hooks for them. The attribute exists, but GCC 7 — the version in NetBSD at the time — does not implement it reliably for all cases.

GCC 8

The solution was to bring up GCC 8 as the NetBSD kernel compiler. After fixing build warnings, the basic kernel boots. Getting kcov(4) fully functional under GCC 8 with working no_instrument_function support is still in progress — the next report will cover those results.

Fuzzing FFS

Given the coverage limitations, the approach for filesystem fuzzing shifts: rather than relying on coverage guidance to find the comparison boundary, we seed AFL with a valid, properly structured filesystem image and let it mutate from there.

Michal Zalewski’s posts on AFL mutation strategy and on JPEG recovery explain why this works: given structured input, AFL performs operations that preserve enough of the structure to keep the target engaged while breaking invariants one field at a time.

The Mount Wrapper

The fuzzing loop needs to mount a filesystem image, optionally interact with it, and unmount cleanly. In shell that looks like:

# Expose file from tmpfs as a block device
vndconfig vnd0 /tmp/rand.tmp

# Mount
mount /dev/vnd0 /mnt

# Do something
echo "FFS mounted!" > /mnt/test

# Teardown
umount /mnt
vndconfig -u vnd0

Shell is too slow for a fuzzer. The wrapper is compiled as a shared library using vnd(4) ioctls and direct syscalls:

struct ufs_args ufs_args;

rv = run_config(VND_CONFIG, dev, fpath);
if (rv)
    printf("VND_CONFIG failed: rv: %d\n", rv);

if (mount(FS_TYPE, fs_name, mntflags, &ufs_args, 8) == -1) {
    printf("Mount failed: %s", strerror(errno));
} else {
    /* filesystem is mounted — exercise it here */

    if (unmount(fs_name, 0) == -1)
        printf("Umount failed!\n");
}

rv = run_config(VND_UNCONFIG, dev, fpath);

vnd(4) exposes a regular file as a block device via opendisk(3) and the VND_CONFIG ioctl. mount(2) takes the ufs_args structure. The complete wrapper is on GitHub.

Build it as a shared object so AFL can use its fork server model:

gcc -fPIC -lutil -g -shared ./wrapper_mount.c -o wrapper_mount.so

Seeding the Corpus

Starting with a zeroed file produces a valid block device but not a valid filesystem. AFL will spend cycles on inputs the kernel rejects immediately. A proper corpus starts with a real FFS image:

dd if=/dev/zero of=./in/test1 bs=10k count=8
vndconfig vnd0 ./in/test1
newfs /dev/vnd0
vndconfig -u vnd0

Now ./in/test1 is a valid FFS image that mounts cleanly. AFL mutates from that baseline.

Running

./afl-fuzz -k -i ./in -o ./out -- /mypath/wrapper_mount.so @@

The @@ is replaced by AFL with the path to the mutated input file. After seventeen seconds:

                  american fuzzy lop 2.35b (wrapper_mount.so)

┌─ process timing ─────────────────────────────────────┬─ overall results ─────┐
│        run time : 0 days, 0 hrs, 0 min, 17 sec       │  cycles done : 0      │
│   last new path : none seen yet                      │  total paths : 1      │
│ last uniq crash : none seen yet                      │ uniq crashes : 0      │
│  last uniq hang : none seen yet                      │   uniq hangs : 0      │
├─ cycle progress ────────────────────┬─ map coverage ─┴───────────────────────┤
│  now processing : 0 (0.00%)         │    map density : 17.28% / 17.31%       │
│ paths timed out : 0 (0.00%)         │ count coverage : 3.53 bits/tuple       │
├─ stage progress ────────────────────┼─ findings in depth ────────────────────┤
│  now trying : trim 512/512          │ favored paths : 1 (100.00%)            │
│ stage execs : 15/160 (9.38%)        │  new edges on : 1 (100.00%)            │
│ total execs : 202                   │ total crashes : 0 (0 unique)           │
│  exec speed : 47.74/sec (slow!)     │   total hangs : 0 (0 unique)           │
├─ fuzzing strategy yields ───────────┴───────────────┬─ path geometry ────────┤
│   bit flips : 0/0, 0/0, 0/0                         │    levels : 1          │
│  byte flips : 0/0, 0/0, 0/0                         │   pending : 1          │
│ arithmetics : 0/0, 0/0, 0/0                         │  pend fav : 1          │
│  known ints : 0/0, 0/0, 0/0                         │  own finds : 0         │
│  dictionary : 0/0, 0/0, 0/0                         │  imported : n/a        │
│       havoc : 0/0, 0/0                              │ stability : 23.66%     │
│        trim : n/a, n/a                              ├────────────────────────┘
┴─────────────────────────────────────────────────────┘             [cpu:  0%]

47 executions per second. Slow — mount cycles are expensive — but functional. The stability at 23.66% reflects the coverage noise described above; AFL is seeing inconsistent paths on identical inputs because the UVM and mutex entries vary between runs.

This is a proof of concept, not a production setup. The fuzzer is running inside the same kernel it is fuzzing. If the kernel panics, the fuzzer stops. That is a known limitation, addressed in the next installment with a remote VM setup.

What’s Next

Three things are on the list for the next report:

  1. GCC 8 and no_instrument_function — filter UVM and machine-dependent code from the coverage trace to give AFL meaningful signal.
  2. Remote AFL setup — run the fuzzer from a controller outside the VM; the kernel crashes and reboots, and the fuzzer continues.
  3. Bug analysis — the FFS run found a few issues. Part 3 digs into the root cause of one: an invisible mount point caused by a corrupted root inode.