In Part 4, we tried to recompile 191 decompiled C files and hit 2,896 unresolved symbols. We learned why firmware linking is fundamentally different from userspace linking, and concluded that binary patching – the four-byte NOP from Part 3 – was the pragmatic fix.

But there’s a question we haven’t asked: what did the vendor do?

If there’s a newer firmware version, comparing old and new tells you exactly what changed – and whether the vendor agreed with your diagnosis. In this post, we’ll put two firmware versions side by side, decompile the same function from each, and see how a real fix differs from a four-byte patch. Then we’ll turn to a second technique: using call frequency statistics to automatically identify core utility functions in firmware where every name starts with FUN_.

Two Binaries, One Function

Suppose the vendor ships a firmware update. We now have two flat binaries:

$ ls -la firmware*.bin
-rw-r--r-- 1 user user  8496 Apr 25 firmware.bin       # v2.4.1-rc3 (buggy)
-rw-r--r-- 1 user user  8712 Apr 25 firmware_v2.bin    # v3.0.0 (fixed)

The new version is 216 bytes larger. That is suspicious – a NOP patch does not add bytes. Whatever changed, it was not a one-instruction fix.

The first thing we do with any new firmware version is the same thing we did in Part 2: strings. But this time, we diff the output:

$ diff <(strings firmware.bin | sort) <(strings firmware_v2.bin | sort)
< bridge routes deferred to bridge handler
---
> bridge
> bridge entries registered
> forwarding
> forwarding table updated
> registering route modules
< route: bridge flag set, skipping
> route: registering bridge entry
< status
< uptime
< v2.4.1-rc3
---
> v3.0.0

Several things jump out:

  1. Version changed: v2.4.1-rc3 to v3.0.0 – a major version bump, not a point release
  2. New strings: “forwarding”, “bridge”, “registering route modules” – these look like module names
  3. Changed messages: “route: bridge flag set, skipping” became “route: registering bridge entry” – the bridge path is not skipping routes anymore
  4. Removed string: “bridge routes deferred to bridge handler” is gone – the old deferral logic was removed entirely

And two lines that look like removed commands but are not: status and uptime disappear from the v3.0.0 output entirely. Did the vendor remove the status and uptime CLI commands? No – nm firmware_v2.elf still lists both handlers. What disappeared is the standalone string literal. In v2.4.1-rc3, the command name "status" happens to be a suffix of the help text "Show system status", and "uptime" a suffix of "Show system uptime". The linker’s mergeable string sections do suffix merging: if one string is a tail of another, the shorter one gets folded into the longer one’s bytes instead of getting its own copy. In v3.0.0 that merge just happened to trigger for these two commands, so strings – which only reports null-terminated runs – no longer sees "status" or "uptime" as independent strings. It is a linker artifact of that specific build, not a vendor change. Cross-version string diffs will occasionally show noise like this; the fix is to check nm before concluding a command was removed.

The string diff alone tells a story: the route engine was restructured from “check flags, skip bridge” to “register modules, dispatch independently.” Let’s confirm this by decompiling both versions.

Side-by-Side Decompilation

Here’s iterate_active_routes from the old firmware (v2.4.1-rc3). We saw this in Part 3 – the function that contains our priority inversion bug:

/* v2.4.1-rc3 — iterate_active_routes() — flag-checking loop */
void iterate_active_routes(int stack_idx)
{
    int i;
    int programmed = 0;
    int skipped = 0;

    for (i = 0; i < MAX_ROUTES; i++) {
        route_entry_t *entry = &g_route_table_entries[i];

        if (entry->flags == 0)
            continue;

        if (entry->flags & ROUTE_FLAG_BRIDGE) {        /* ← checks bridge FIRST */
            log_msg(MOD_ROUTE, "route: bridge flag set, skipping");
            skipped++;
            continue;                                    /* ← never reaches active check! */
        }

        if (entry->flags & ROUTE_FLAG_ACTIVE) {
            log_msg(MOD_ROUTE, "route: programming forwarding entry");
            programmed++;
        }
    }

    if (programmed == 0)
        log_msg(MOD_ROUTE, "WARNING: no routes programmed for stack");
    if (skipped > 0)
        log_msg(MOD_ROUTE, "bridge routes deferred to bridge handler");
}

The structure is a single loop with inline flag checks. The if/else ordering means BRIDGE (0x04) is checked before ACTIVE (0x01), and the continue means routes with both flags never reach the active check.

Now here’s the same function from v3.0.0:

/* v3.0.0 — iterate_active_routes() — table-driven dispatch */

typedef void (*route_handler_t)(route_entry_t *entry, int *count);

typedef struct {
    const char *name;
    route_handler_t handler;
    uint32_t route_mask;
} route_module_t;

static void handle_forwarding_routes(route_entry_t *entry, int *count) {
    if (entry->flags & ROUTE_FLAG_ACTIVE) {
        log_msg(MOD_ROUTE, "route: programming forwarding entry");
        (*count)++;
    }
}

static void handle_bridge_routes(route_entry_t *entry, int *count) {
    if (entry->flags & ROUTE_FLAG_BRIDGE) {
        log_msg(MOD_ROUTE, "route: registering bridge entry");
        (*count)++;
    }
}

const route_module_t g_route_modules[] = {
    { "forwarding", handle_forwarding_routes, ROUTE_FLAG_ACTIVE },
    { "bridge",     handle_bridge_routes,     ROUTE_FLAG_BRIDGE },
};

void iterate_active_routes(int stack_idx)
{
    int i, m;
    int counts[NUM_ROUTE_MODULES];

    for (m = 0; m < NUM_ROUTE_MODULES; m++)
        counts[m] = 0;

    for (i = 0; i < MAX_ROUTES; i++) {
        route_entry_t *entry = &g_route_table_entries[i];

        if (entry->flags == 0)
            continue;

        /* Dispatch to EVERY registered module — no flag priority */
        for (m = 0; m < NUM_ROUTE_MODULES; m++) {
            g_route_modules[m].handler(entry, &counts[m]);
        }
    }

    if (counts[0] == 0)
        log_msg(MOD_ROUTE, "WARNING: no routes programmed for stack");
    if (counts[0] > 0)
        log_msg(MOD_ROUTE, "forwarding table updated");
    if (counts[1] > 0)
        log_msg(MOD_ROUTE, "bridge entries registered");
}

This is a complete architectural rewrite. The function no longer checks flags inline. Instead:

  1. Route handlers are registered in a module table (array of route_module_t)
  2. Each handler is a separate function that decides independently whether to process an entry
  3. The dispatch loop calls every handler for every non-empty entry
  4. A route with both ACTIVE and BRIDGE flags gets processed by both handlers

There’s no if/else between the bridge check and the active check. There’s no continue that skips one path. The two handlers run independently, in separate functions, dispatched from a data table. The priority inversion that existed in v2.4.1-rc3 is structurally impossible in v3.0.0.

What Changed and Why It Matters

Let’s make the comparison precise:

Aspect v2.4.1-rc3 (buggy) v3.0.0 (fixed)
Algorithm Single loop with inline flag checks Table-driven module dispatch
Flag handling if/else chain: BRIDGE before ACTIVE Each handler checks its own flag independently
Bridge + Active route Bridge check fires, continue skips active Both handlers run, both succeed
Data structures None (flags checked inline) route_module_t table with function pointers
Handler isolation None (all logic in one function) Each handler is a separate function
New route types Requires adding else if branch Requires adding table entry

The v2.4.1-rc3 bug was in the ordering of two if statements inside one function. Our NOP patch in Part 3 fixed the symptom by disabling the bridge branch. The vendor fixed the class of bug by making the ordering irrelevant.

This is a common pattern in firmware evolution. The first version implements something quickly with inline checks. A bug is found. The fix isn’t “swap two lines” – it’s “restructure the function so that the bug category can’t exist.” If you see a major version bump and a function that went from 20 lines to 60 lines, this is probably what happened.

For a reverse engineer, cross-version comparison gives you three things:

  1. Confirmation: The vendor changed iterate_active_routes, confirming our bug diagnosis from Part 3
  2. Understanding: The vendor didn’t just fix the symptom – they rewrote the dispatch architecture
  3. Future-proofing: When analyzing the new version, we know to look for the module table instead of inline flag checks

The general technique: when you have two firmware versions and a known bug location, decompile the same function from both, and diff. The changes tell you exactly what the vendor considered broken and how they chose to fix it.

Call Graph Analysis: Naming the Unnamed

Let’s shift to a different problem. In Part 2, we found 35 functions by scanning for MIPS prologues. In Part 4, we saw what Ghidra produces when it decompiles them: port_validate renamed to FUN_80000c84, calling a still-unidentified helper, FUN_800002e4, twice along the way. Meaningful names replaced by addresses.

If you have debug symbols, nm gives you every name instantly. But vendor firmware ships stripped. You’re staring at 35 functions (or 4,426 in the real project) and need to figure out which is which.

Here’s the insight: not all functions are equal. Some functions are called once. Others are called from everywhere. The ones called from everywhere are utilities – and their call frequency alone tells you what they are.

Let’s count how many times each function is called as a JAL target in our firmware:

$ mipsel-linux-gnu-objdump -d firmware.elf \
    | grep -P '\tjal\t' \
    | sed 's/.*jal\s*[0-9a-f]*//' \
    | sort | uniq -c | sort -rn | head -10
    111 FUN_800002e4
     15 FUN_80000790
     15 FUN_80000530
     12 FUN_80000b74
      6 FUN_800002c4
      4 FUN_80000fec
      4 FUN_80000aa8
      2 FUN_80000d00
      2 FUN_80000c84
      2 FUN_80000280

(I’ve replaced the actual symbol names with Ghidra-style FUN_ addresses – in a real stripped binary, this is what you’d see. Note the \tjal\t filter: a plain grep 'jal' also matches jalr, the indirect-call-through-register instruction, and pollutes the count with a bogus entry. That distinction matters beyond cleanliness – v3.0.0’s table-driven dispatch calls each route handler through a function pointer, which compiles to jalr, not jal. A JAL-only call count is structurally blind to every call the new dispatch table makes; that dispatch traffic simply will not show up in this kind of analysis, no matter how the grep is tuned.)

The distribution is wildly uneven. FUN_800002e4 is called 111 times – more than all other functions combined. That’s not business logic. That’s infrastructure. Let’s figure out what it is.

From Counts to Names

FUN_800002e4 (111 calls): Called from nearly every function in the binary. It takes a single pointer argument. In the string cross-reference analysis from Part 2, every call to this function passes a string address. A function called 111 times that takes a string? That’s uart_puts – the serial output function.

FUN_80000790 (15 calls) and FUN_80000530 (15 calls): Both called frequently, both appear in the same functions as uart_puts. One takes a 32-bit integer and outputs hex characters (look for the 0123456789ABCDEF string reference). The other outputs decimal digits. These are uart_puthex and uart_putdec.

FUN_80000b74 (12 calls): Called by every subsystem initialization function. Takes two arguments: an integer (always a small constant like 1, 2, 3, 4) and a string pointer. The strings are log messages (“system starting”, “initializing ports”). This is log_msg – the logging function with a module ID and message string.

FUN_80000aa8 (4 calls): Takes three arguments: an integer, a string that ends in .c, and another integer. The strings are source filenames: route.c, flash.c, diag.c. A function that takes (code, "file.c", line_number) is an assert handler. This is fw_assert.

Five functions identified from call counts alone:

Call Frequency Analysis — Top Functions
========================================

Address       Calls  Identification      Evidence
──────────────────────────────────────────────────────────
FUN_800002e4    111  uart_puts           Every function, takes string arg
FUN_80000790     15  uart_putdec         Takes int, outputs decimal digits
FUN_80000530     15  uart_puthex         Takes int, refs "0123456789ABCDEF"
FUN_80000b74     12  log_msg             Takes (module_id, string), called by init fns
FUN_80000aa8      4  fw_assert           Takes (code, "file.c", line), halts on fatal
FUN_800002c4      6  uart_putchar        Takes single char, called by uart_puts

In a 35-function firmware, we just named 6 functions (17%) from call frequency and string cross-references. Those 6 functions account for 163 of 184 total calls (89%). Name the infrastructure and you understand most of the binary’s call graph.

The Heuristics at Scale

In our sample firmware with 35 functions, this is a nice exercise. In real vendor firmware with thousands of functions, it’s essential. The real project had 4,426 functions. Call graph analysis identified:

  • fw_assert at the top with 497 inbound calls – unmistakable as the assert handler
  • log_msg with 470 callers – the logging function with severity and module ID
  • irq_disable/irq_enable with 345/277 callers – interrupt lock pairs (always appear together)
  • mem_alloc with 330 callers – the firmware’s memory allocator
  • spinlock_acquire/release with 240/204 callers – synchronization primitives

The heuristics that work at scale:

  1. Highest call count + string argument → output function (print, log, puts)
  2. Called with "file.c" + integer → assert or error handler
  3. Always called in pairs (enable/disable, acquire/release) → lock/unlock primitives
  4. Called at the start of many functions → initialization or lock acquisition
  5. Called at the end of many functions → cleanup or lock release

Once you’ve named the top 10-15 utility functions, Ghidra’s decompiled output transforms. Instead of:

FUN_c0060934(0x8000, "psc_port.c", 342);
FUN_c0098e38(1, 0x53, 3, 0, 0, 0, 0);
FUN_c01714e0();

You read:

fw_assert(FATAL, "route.c", 342);
log_msg(ERROR, MOD_ROUTE, MSG_BRIDGE_FLAG);
irq_disable();

The code hasn’t changed. Your understanding of it has.

The Full Picture

Over five posts, we’ve built a complete firmware reverse engineering toolkit:

  1. Part 1: Understood the flat binary format – no headers, no symbols, raw instructions at byte zero
  2. Part 2: Found strings, traced cross-references, discovered hidden CLI commands
  3. Part 3: Decompiled the route engine, found the priority inversion bug, patched four bytes
  4. Part 4: Tried to recompile, hit 2,896 unresolved symbols, learned why firmware linking is hard
  5. Part 5: Compared two firmware versions to understand architectural fixes, used call graphs to name functions automatically

Every technique came from a real project. Every pattern is something we found in production firmware. The domain was changed (network gateway instead of the actual device), but the methods are universal. Cross-version comparison works on any firmware with multiple releases. Call graph analysis works on any binary with function calls.

The firmware still doesn’t want you to read it. But now you have five ways to read it anyway.

What’s Next: A New Series

This series used an 8 KB sample firmware to keep things approachable. But real firmware is bigger, more complex, and full of patterns we haven’t touched yet – cryptographic algorithm identification from lookup tables, struct recovery from raw memory offsets, hardware peripheral mapping from MMIO access patterns, and the kind of runtime architecture (RTOS schedulers, pool allocators, rule engines) that makes embedded devices work.

For the next series – “Crypto Tables, Struct Ghosts, and Other Secrets Hidden in Firmware” – we’ve built a 38 KB firmware with AES-128 encryption, SHA-256 hashing, a cooperative RTOS, a three-mode pool allocator, and seven MMIO hardware blocks. We’ll also finally introduce radare2 and compare it head-to-head against Ghidra and Capstone.

If you’ve followed along this far, you have the foundation. The next series goes deeper.

References