In Part 3, we patched a four-byte bug in the firmware’s route engine and recalculated the CRC. Four bytes – problem solved. But as we noted at the end: what if the fix was not that simple? What if we needed to restructure a function, add error handling, or change a data structure?

You cannot do that with a hex editor. You would need to recompile from source.

We have Ghidra’s decompiled C. We have the function names, the string references, the module structure. We have gcc and a linker script. How hard can it be?

Very.

The Illusion of Having Source Code

Ghidra produces C code that reads like source but is not source. Here is what Ghidra generates for port_validate:

undefined4 FUN_80000c84(int param_1)
{
    undefined4 uVar1;

    if ((param_1 < 0) || (7 < param_1)) {
        FUN_800002e4(s_ERROR_invalid_port_number_80001bb0);
        FUN_800002e4(&DAT_80001a88);
        uVar1 = 0xffffffff;
    }
    else if ((*(uint *)(param_1 * 0x20 + 0x80040200) & 1) == 0) {
        FUN_800002e4(s_ERROR_port_is_not_enabled_80001b94);
        FUN_800002e4(&DAT_80001a88);
        uVar1 = 0xffffffff;
    }
    else {
        uVar1 = 0;
    }
    return uVar1;
}

The structure is correct. The logic is correct. But look at what is missing:

  • Types are wrong: undefined4 instead of int. No port_config_t struct.
  • Names are gone: FUN_80000c84 instead of port_validate. param_1 instead of port_num.
  • Data access is raw: *(uint *)(param_1 * 0x20 + 0x80040200) instead of g_ports[port_num].flags.
  • Symbols are addresses: s_ERROR_invalid_port_number_80001bb0 and DAT_80001a88 are Ghidra-generated names that encode the binary address.
  • No headers: No #include, no type definitions, no struct layouts.

You can fix all of this manually. Rename FUN_80000c84 to port_validate, change param_1 to port_num, define the port_config_t struct. But now multiply that by every function in the firmware.

In the real project, Ghidra decompiled 4,426 functions across 1.5 MB of code. We organized them into 191 C source files matching the original module structure. Every single file compiled individually with gcc -c. It looked like we were close to a complete build.

Then we tried to link.

The Linking Wall

$ mipsel-linux-gnu-ld -T linker.ld -o firmware.elf *.o
startup.o: undefined reference to `firmware_main'
main.o: undefined reference to `s_system_starting_80001ee0'
main.o: undefined reference to `s_initializing_ports_80001cf0'
route.o: undefined reference to `DAT_80040008'
cli.o: undefined reference to `s_Available_commands_800018b4'
port.o: undefined reference to `_DAT_B8000008'
... 2,896 more ...

2,896 unresolved symbols. Every single one needs to be resolved before the linker will produce a binary.

Why? When you compile with gcc -c, each source file compiles in isolation. The compiler trusts your extern declarations: “this function exists, this variable exists, the linker will find them.” Compilation succeeds because the compiler does not check. Linking fails because the linker does.

In userspace, the linker resolves symbols against libraries (libc.so, libm.so, etc.) and other object files. In firmware, there are no libraries. Every symbol must come from your own object files, your linker script, or it does not exist.

If you are following along with the sample firmware, you will not hit this wall yourself – it is a single translation unit that links cleanly from its own source. This wall belongs to the real project: 191 separately compiled object files, each trusting the others to supply the symbols it referenced. The sample reproduces the shape of the problem, not its scale.

The Symbol Taxonomy

In the real project, the 2,896 unresolved symbols broke into six categories, each needing a different resolution strategy:

Unresolved Symbol Breakdown
============================

Category           Count   Example                          Resolution
─────────────────────────────────────────────────────────────────────────
String literals    1,172   s_system_starting_80001ee0       Extract from binary
Const data           993   DAT_C0012345                     Extract from binary
MMIO registers       444   _DAT_B8000008                    PROVIDE() in linker script
RAM globals          107   DAT_80240100                     BSS/data section placement
Code labels           91   LAB_C0012345                     Switch table extraction
Compiler runtime      15   func_0xC0170000                  Extract from binary
Other                 74   builtin_strncpy, _gp_disp        Stubs + linker magic
─────────────────────────────────────────────────────────────────────────
Total              2,896

String Literals: The Biggest Category

1,172 symbols are strings. Ghidra names them after their address: s_system_starting_80001ee0 means “the string starting at address 0x80001EE0.” The string data lives in the original binary’s .rodata section, which objcopy happily included in the flat binary.

The fix: extract the strings from the original binary and assemble them into a linkable object file:

# Extract strings from firmware.bin at known addresses
# Generate assembly that the linker can resolve

with open('firmware.bin', 'rb') as f:
    binary = f.read()

print('.section .rodata')
for name, addr in string_symbols:
    offset = addr - BASE_ADDR
    string = read_cstring(binary, offset)
    print(f'.global {name}')
    print(f'{name}: .asciz "{escape(string)}"')

Assemble this into strings.o, add it to the link – 1,172 symbols resolved. This works because the strings are data, not code. We are literally pulling bytes out of the original binary and giving them labels.

MMIO Registers: Addresses Without Content

444 symbols are hardware register addresses – UART, GPIO, timer, system control. They look like _DAT_B8000008 (UART status register at 0xB8000008). These are not in the binary at all – they are memory-mapped I/O addresses that the CPU accesses at runtime.

The linker does not need content for these. It just needs to know the address. That is what PROVIDE() does:

/* linker.ld -- MMIO register definitions */
SECTIONS
{
    /* ... code and data sections ... */
}

/* Hardware register addresses -- no content, just locations */
PROVIDE(_DAT_B8000000 = 0xB8000000);   /* UART TX */
PROVIDE(_DAT_B8000004 = 0xB8000004);   /* UART RX */
PROVIDE(_DAT_B8000008 = 0xB8000008);   /* UART Status */
PROVIDE(_DAT_B800000C = 0xB800000C);   /* UART Control */
PROVIDE(_DAT_B8010000 = 0xB8010000);   /* GPIO Data */
/* ... 439 more ... */

PROVIDE() tells the linker: “if nobody else defines this symbol, it lives at this address.” The generated code will use LUI/ADDIU to load the address and then read/write the hardware register. No content needed in the binary – MMIO registers exist in the hardware, not in flash.

In the real project, 444 MMIO registers spanned five address blocks (0xB4, 0xB6, 0xB9, 0xBD, 0xBF), covering the switch core, packet processor, PHY interface, DMA engine, and management controller. Each one got a PROVIDE() line.

The Global Pointer Problem

On MIPS, frequently accessed global variables use GP-relative addressing. Instead of a two-instruction LUI/ADDIU pair, the compiler emits a single instruction:

lw  v0, -32000(gp)    # Load global at GP - 32000

This saves one instruction per access – significant in a firmware with thousands of global accesses. But it only works if $gp (the global pointer register) is set to the right value, and all code agrees on what that value is.

The problem: the decompiled code contains hardcoded GP offsets baked in by the original compiler. If the recompiled binary places globals at different addresses – even slightly – every GP-relative access breaks.

/* Ghidra decompiled code contains things like: */
*(int *)(gp + -0x7c50) = 1;    /* original GP offset, hardcoded */

Three ways to fix it. Match GP exactly: set it to the original compiler’s value and reproduce the full original global layout, which requires perfect section placement. Disable GP-relative addressing entirely with -G0 -mno-gpopt and rewrite all accesses as full LUI/ADDIU pairs – the optimization is gone, but at least it links. Or declare all globals properly and let a fresh compiler recalculate offsets from scratch, which requires getting every global’s type and size right first.

Our sample firmware uses option two (-G0 -mno-gpopt in the Makefile) – every global uses the full LUI/ADDIU pair, so there is no GP dependency. Real vendor firmware is rarely that considerate.

The Custom Linker Script

In userspace, the linker uses a default script that puts .text at a dynamic address, .data after it, and .bss at the end. The kernel’s ELF loader handles the rest. In firmware, you need total control:

/* linker.ld -- Firmware memory layout */
MEMORY
{
    ROM (rx)  : ORIGIN = 0x80000000, LENGTH = 256K
    RAM (rw)  : ORIGIN = 0x80040000, LENGTH = 64K
}

SECTIONS
{
    /* Exception vectors MUST be at 0x80000000 */
    .vectors 0x80000000 : {
        *(.vectors)
    } > ROM

    /* Code follows immediately */
    .text : ALIGN(4) {
        *(.text)
        *(.text.*)
    } > ROM

    /* Read-only data (strings, tables) */
    .rodata : ALIGN(16) {
        *(.rodata)
        *(.rodata.*)
    } > ROM

    _etext = .;

    /* Globals in RAM (zeroed at boot by startup code) */
    .bss (NOLOAD) : ALIGN(4) {
        __bss_start = .;
        *(.bss)
        *(COMMON)
        __bss_end = .;
    } > RAM

    /* Stack at top of RAM */
    _stack_top = ORIGIN(RAM) + LENGTH(RAM);
}

It is clean because we built it from scratch. The linker.ld bundled with the sample firmware is a fuller version of this – it adds explicit .rodata.cli_* ordering (so the hidden CLI commands land after the public ones), a real .data section loaded from ROM into RAM, and a _gp assignment – all elided here for clarity. In a real reverse engineering project, you would need to derive the whole thing from the binary’s memory layout – figuring out where each section starts and ends by analyzing address patterns in the code.

The MEMORY block defines two regions: ROM (where the flat binary lives in flash) and RAM (where globals and stack live at runtime). The exception vectors must be at 0x80000000 because that is where the MIPS CPU looks for them at reset. There is no flexibility here – get it wrong by one byte and the device crashes on boot.

Compare this with the 2018 ELF post, where we relied on the kernel’s ELF loader to parse program headers and map segments. In firmware, the linker script is the loader. Everything the kernel did for ELF – segment mapping, BSS zeroing, stack setup – you do yourself in a 20-line startup assembly file and a linker script.

The Compilation Gap

Here is what full recompilation actually requires:

Full Firmware Recompilation Checklist
=====================================

1. Decompile all functions              ✓  (Ghidra automated this)
2. Fix Ghidra output artifacts          ✗  (undefined4, unaff_gp, raw casts)
3. Reconstruct struct definitions       ✗  (Ghidra doesn't recover struct layouts)
4. Reconstruct enum/constant values     ✗  (flag values inferred from usage)
5. Extract strings from binary          ~  (automated but needs validation)
6. Extract const data from binary       ~  (automated, 993 symbols)
7. Map MMIO registers to PROVIDE()      ~  (automated, 444 addresses)
8. Resolve GP-relative globals          ✗  (requires matching GP layout)
9. Stub/extract compiler runtime        ~  (15 functions from vendor compiler)
10. Handle switch tables (LAB_*)        ✗  (91 computed gotos need manual work)
11. Write startup code                  ✗  (exception vectors, BSS clear, GP setup)
12. Write linker script                 ~  (derive from binary memory map)
13. Integration test                    ✗  (need hardware or accurate emulator)

Estimated effort: 100-170 hours for 1.5 MB of firmware

Step 1 is essentially free – Ghidra decompiles thousands of functions in minutes. Steps 5-7 can be automated with scripts. Steps 2-4, 8, and 10-13 require manual analysis and domain knowledge – which is where the time actually goes.

A note on using an LLM for this: tools like Claude Opus 4.8 can compress the annotation work substantially – renaming functions from Ghidra output, reconstructing struct definitions, generating PROVIDE() lists from address patterns. In practice, the 100-170 hour estimate drops by roughly a factor of five, to something in the 20-30 hour range for comparable firmware. But it does not eliminate the need for supervision. The model will confidently reconstruct a struct layout that is subtly wrong, or generate a section offset that is plausible but off by four bytes. Every output still needs a human to verify against the binary. The hours go down; the attention required does not.

The decompilation-to-compilation gap: decompiled C is a description of the binary’s behavior, not a specification of how to build it. That is like having a photograph of a building – you can see every room, but you do not have the blueprints, the materials list, or the building code compliance.

In the real project, we stopped before linking. The goal was structural verification: every source file compiles, the function signatures are correct, the call graph is preserved – but no linked binary.

# Compile every file independently
for f in src/*.c; do
    mipsel-linux-gnu-gcc-14 -mips32r2 -EL -O1 -ffreestanding \
        -nostdlib -mno-abicalls -fno-pic -G0 -mno-gpopt \
        -c -o "${f%.c}.o" "$f"
done

# Result: 191/191 compile successfully
# But: 2,896 symbols remain unresolved at link time

This is not failure – it is a deliberate stopping point. Every file compiling confirms the decompilation is syntactically correct, function signatures match across the 191-file tree, and no conflicting type definitions exist. It does not prove the binary will boot, but for reading the code and finding bugs, it is more than enough.

For the bug fix we did in Part 3, this was exactly that. We understood the code, found the bug, and patched it in the binary. Full recompilation would have been the “right” way to fix it, but binary patching achieved the same result in an afternoon instead of several months.

When Full Recompilation Makes Sense

Binary patching is a scalpel. Recompilation is a complete rebuild. You choose based on the scope of changes:

Change scope Approach Time
Fix one branch instruction Binary patch (hex editor) Hours
Change a constant or string Binary patch + CRC update Hours
Add a new function Recompilation (probably) Weeks
Restructure a module Recompilation (definitely) Months
Port to new hardware Full rewrite from decompiled source Months+

The real project chose binary patching for the immediate fix and structural verification as a long-term investment. If the vendor never provides an official fix, having 191 compilable C files is a foundation for future work – even if linking them into a single binary is still a 100+ hour project.

It Linked. It Still Didn’t Boot.

Producing a correctly linked binary is not the finish line.

In a separate project – a 4+ MB firmware on a different embedded device – we reached the point this sample firmware’s analysis stops short of: every symbol resolved, the linker produced a binary, the image packed cleanly with valid CRC checksums. We flashed it. The device entered the primary bootloader, the secondary bootloader ran – and then silence. No UART output. No register activity. The firmware died without a word.

The first problem is the bootloader. As we covered in Part 1, the firmware image has separate partitions. The primary bootloader lives at a fixed flash address, is often write-protected, and cannot be replaced – it is the first code that runs at reset. That bootloader was compiled by the vendor with their exact toolchain, and it has specific expectations about the handoff to main firmware: entry point address, initial stack state, BSS layout, exception vector locations. A recompiled binary that differs even slightly from those expectations will fail at the handoff, silently.

The second problem is the compiler. Vendor firmware is frequently built with a proprietary SDK – a vendor-patched GCC, IAR, Green Hills MULTI, or the MIPS Technologies compiler itself. These make choices that mipsel-linux-gnu-gcc-14 does not: interrupt calling conventions, stack alignment under exception handlers, runtime initialization order, startup register state. None of these surface as linker errors. They show up as a device that boots partway and then goes quiet.

Weeks went into educated guesses – adjusting the entry point, rewriting the startup assembly, matching section alignment, trying to reconstruct what the original runtime initialization looked like. The firmware died after the secondary bootloader every single time. At some point the cost of continued guessing exceeded the value of the exercise.

Binary patching was the right answer all along. It was just less satisfying to admit.

The Full Circle

We started this series with an ELF binary on an x86-64 Linux system – sections, symbols, relocations, a well-documented format, a kernel loader that handles everything. Four posts later, we have been to the other end of the spectrum: a flat binary on a MIPS32 embedded device – no headers, no symbols, no loader, a custom container format with CRC checksums.

The journey so far:

  1. Part 1: Understood what is missing in flat firmware vs ELF
  2. Part 2: Found strings, traced functions, discovered hidden CLI commands
  3. Part 3: Decompiled the route engine, found a priority inversion bug, patched it
  4. Part 4: Tried to recompile, hit 2,896 unresolved symbols, got 191 files to compile – then learned that a linked binary still might not boot

Every technique came from a real project. Every pattern is something we found in production firmware. The domain was changed (network gateway instead of the actual device), but the methods are universal. If you can find strings in a binary, trace function calls, decompile with Ghidra, and write a linker script, you can reverse engineer any firmware on any architecture.

The firmware doesn’t want you to read it. Read it anyway.

But there is one more question we have not asked: what happens when the vendor ships an update? In Part 5: “Versions, Callgraphs, and other ways to compare what changed”, we will put two firmware versions side by side, compare how the vendor chose to fix the same bug we patched, and use call graph statistics to automatically name functions in stripped binaries.

References