tlp-tool and the Next Phase of PCIe Debugging
Note: This piece started as a pitch to LWN. The editors passed on it, so it’s going up here instead.
Peripheral Component Interconnect Express (PCIe) failures are often reported in a form that is precise, information-dense, and nearly unreadable: a Transaction Layer Packet (TLP) header dumped as four raw 32-bit words in an Advanced Error Reporting (AER) log. Understanding what those bytes mean requires knowing the TLP format — and the format just changed.
What a TLP header looks like
When a PCIe device causes an error, the hardware records the offending packet’s header in the AER Header Log register. A typical kernel log or lspci -vv output shows it as four DWORDs:
HeaderLog: 00000001 0000220f 01070000 9eece789
For PCIe 1.0 through 5.0, the layout of those bytes has been stable. DW0 — the first four bytes — encodes the packet type, traffic class, and payload length. Everything else follows from what DW0 says:
DW0: 0x00000001
Byte 0: 0x00 = 0b00000000
[7:5] FMT = 000 → 3DW header, no data
[4:0] TYPE = 00000 → Memory Read Request
Byte 1: 0x00 → TC=0, Attr[2]=0
Byte 2: 0x00 → TD=0, EP=0 (not poisoned), Attr[1:0]=00
Byte 3: 0x01 → Length = 1 DW (4 bytes)
The FMT and TYPE fields together form a matrix that determines both the packet kind and the header length (3DW or 4DW). Once you know the format, you know where every subsequent field lives:
| DW | Memory Request | Config Request | Completion |
|---|---|---|---|
| 0 | FMT+TYPE, TC, Length | FMT+TYPE, TC, Length | FMT+TYPE, TC, Length |
| 1 | Requester ID, Tag, Byte Enables | Requester ID, Tag, Byte Enables | Completer ID, Status, Byte Count |
| 2 | Address[31:2] (3DW) or Address[63:32] (4DW) | Bus/Dev/Func, Ext Reg, Register | Requester ID, Tag, Lower Address |
That table is the core of traditional TLP parsing. The same FMT+TYPE matrix, the same field positions, the same bit extraction — unchanged from 2003 to 2019. Sixteen years of stability.
Then Flit Mode happened
PCIe 6.0 introduced Flit Mode to support PAM4 signaling at 64.0 GT/s. Instead of variable-length packets delimited by STP/END framing tokens, TLPs are packed into fixed 256-byte containers called flits, each with its own forward error correction. That is a data-link layer change, but it reached up into the TLP format itself.
DW0 in Flit Mode carries a different set of fields. The FMT+TYPE matrix — the thing that has been the entry point for TLP parsing since PCIe 1.0 — is gone. In its place, there are thirteen type codes in a flat enumeration and a new set of header fields:
Classic DW0: FMT[7:5] | TYPE[4:0] | TC | Attr | Length
Flit DW0: Type[4:0] | T9 | TC | OHC-A | TH | Attr | AT | Length
TLP Prefixes — the extension mechanism in non-Flit framing — are replaced by Optional Header Components (OHC). The type encoding is not a subset or superset of the old one; it is a different numbering. The same raw DW0 bytes decode to a different packet type depending on whether the link was using Flit framing or not.
A handful of representative DW0 byte 0 values, side by side, makes the divergence concrete:
| DW0 byte 0 | Non-Flit framing | Flit Mode framing |
|---|---|---|
0x00 |
Memory Read (3DW) | NOP |
0x03 |
undefined | Memory Read (32-bit) |
0x04 |
Type 0 Configuration Read | undefined |
0x22 |
undefined | UIO Memory Read (64-bit) |
0x40 |
Memory Write (3DW) | Memory Write (32-bit) |
0x4C |
FetchAdd AtomicOp | FetchAdd AtomicOp (32-bit) |
0x4E |
CompareSwap AtomicOp | CompareSwap AtomicOp (32-bit) |
0x70 |
Message with Data → RC | Message with Data → RC |
0x8D |
Local TLP Prefix | Local TLP Prefix |
The first row is the pathological case: a byte that has meant “Memory Read, 3DW header” since 2003 now means “no operation” on a Flit Mode link. The 0x04 row is almost as severe — a Type 0 Configuration Read in classic PCIe is not a defined type at all in Flit Mode. Other rows align by design — Memory Write at 0x40, the AtomicOps at 0x4C/0x4E, the routed Message-with-Data at 0x70, and the TLP Prefix at 0x8D — because the spec writers preserved semantic continuity where they could. But neither the alignments nor the divergences are something the bytes themselves announce; the framing context is what selects the table.
The header does not tell you which interpretation is correct — the framing context does. The worked example in the next section will run 04000001 through the tool as a Type 0 Configuration Read in classic mode; on a Flit Mode link, those same bytes would not decode as a defined type at all.
No one expects you to memorize this
The field-level detail in those tables is useful for understanding the problem, but nobody should be decoding TLP headers by hand for every AER event. The PCIe maintainers agree — which is why the kernel’s PCI documentation now points users at tlp-tool, whose binary is named rtlp-tool.
On March 23, Lukas Wunner posted a patch titled “Documentation: PCI: Document decoding of TLP Header in AER messages”, arguing that the TLP Header hints at the root cause of an error but is often ignored because of its seeming opaqueness. Wunner noted that the obvious alternative — wireshark TLP dissectors — expects complete TLPs rather than just headers, and cannot consume the hex format the kernel emits directly; tlp-tool was, in his words, “the most cut and dried solution out there.” Bjorn Helgaas applied the patch to pci/for-linus for the 7.0 merge window the same day, tweaking the commit log on the way in to note that the Header Log lives in the AER Capability — which may be present on any PCIe function, not just root ports. The result is modest but significant: the kernel’s PCIe AER documentation now recommends tlp-tool as a way to decode the TLP Header into human-readable form, and ships a worked example — piping a kernel commit URL through rtlp-tool --aer — to make the recommendation directly executable. Wunner’s cover letter also floated going one step further — pointing users at the tool from a printk_once() message when the first AER error fires — but kept the patch itself to documentation only, calling that “probably sufficient” for now.
The lowest-level entry point is to hand the tool the four header words directly:
rtlp-tool -i "04000001 00200a03 05010000 00050100"
That is useful when a header is already in hand. The two modes that matter for real debugging map to where Header Logs actually surface in the kernel: dmesg AER messages, and the HeaderLog: line inside lspci -vv output.
--aer: kernel log input. A TLP Header copied straight from dmesg goes through --aer:
echo 'TLP Header: 04000001 00200a03 05010000 00050100' | rtlp-tool --aer
The output makes the bit extraction explicit:
+----------+------------------+--------------------+
| TLP Type | ConfType0ReadReq | 3DW no Data Header |
+----------+------------------+--------------------+
+------------+--------+--------+-------+
| Field Name | Offset | Length | Value |
| | (bits) | (bits) | |
+------------+--------+--------+-------+
| Fmt | 0 | 3 | 0 |
| Type | 3 | 5 | 4 |
| T9 | 8 | 1 | 0 |
| TC | 9 | 3 | 0 |
| T8 | 12 | 1 | 0 |
| Attr_b2 | 13 | 1 | 0 |
| LN | 14 | 1 | 0 |
| TH | 15 | 1 | 0 |
| TD | 16 | 1 | 0 |
| Ep | 17 | 1 | 0 |
| Attr | 18 | 2 | 0 |
| AT | 20 | 2 | 0 |
| Length | 22 | 10 | 1 |
+------------+--------+--------+-------+
+-----------------+------------------------------------------------+
| TLP: | 3DW no Data Header |
+-----------------+------------------------------------------------+
| Req ID | 0x20 |
| Tag | 0xA |
| First DW BE | 0x3 (bytes 0-1) |
| Last DW BE | 0x0 (none) |
| Target BDF | 05:00.1 |
| Bus | 0x5 |
| Device | 0x0 |
| Function | 0x1 |
| Register Offset | 0x000 |
| Register Name | Vendor ID / Device ID |
| Ext Reg Nr | 0x0 |
| Reg Nr | 0x0 |
| Operation | Read Vendor ID / Device ID register at 05:00.1 |
+-----------------+------------------------------------------------+
Three things to read out of that. First, the type: ConfType0ReadReq — a Configuration Read Request, Type 0 (terminating at the addressed device, not forwarded), 3DW header with no data payload. Second, the requester: ID 0x0020, which decodes as Bus 0, Device 4, Function 0 — a Root Complex device, almost certainly a Root Port issuing the read. Third, the Operation line spells out the rest directly: a read of the Vendor ID / Device ID register on Function 1 of the target at 05:00.1. tlp-tool did not always name the register; earlier releases stopped at the raw register number and left the lookup to the reader. A register-name table for standard PCI config space now resolves the common cases — Vendor/Device ID, Command, Status, Class Code, and so on — to a name instead of a hex offset.
That target is the interesting part. Probing Function 1 with a config read of register 0 is exactly what the kernel does during PCI enumeration to discover whether a multi-function device exposes a second function. If Function 1 does not exist, the device responds with an Unsupported Request, and the kernel logs an AER event for it. What looks like a scary uncorrectable PCIe error in dmesg turns out to be a benign enumeration miss. Without the decode, all you would see is the AER summary; with it, you can tell this case apart from a real failure in seconds.
--lspci: device-attributed input. When the AER Header Log is read directly out of device capabilities, --lspci parses the entire lspci -vv block, picks up every non-zero HeaderLog, and annotates each TLP with the BDF of the device it came from:
lspci -vv -s 05:00.1 | rtlp-tool --lspci
+----------+------------------------------------------------------------+--------------------+
| TLP Type | ConfType0ReadReq | 3DW no Data Header |
+----------+------------------------------------------------------------+--------------------+
| Source | 05:00.1 Non-Volatile memory controller: Phison Electronics | |
+----------+------------------------------------------------------------+--------------------+
+------------+--------+--------+-------+
| Field Name | Offset | Length | Value |
| | (bits) | (bits) | |
+------------+--------+--------+-------+
| Fmt | 0 | 3 | 0 |
| Type | 3 | 5 | 4 |
| T9 | 8 | 1 | 0 |
| TC | 9 | 3 | 0 |
| T8 | 12 | 1 | 0 |
| Attr_b2 | 13 | 1 | 0 |
| LN | 14 | 1 | 0 |
| TH | 15 | 1 | 0 |
| TD | 16 | 1 | 0 |
| Ep | 17 | 1 | 0 |
| Attr | 18 | 2 | 0 |
| AT | 20 | 2 | 0 |
| Length | 22 | 10 | 1 |
+------------+--------+--------+-------+
+-----------------+------------------------------------------------+
| TLP: | 3DW no Data Header |
+-----------------+------------------------------------------------+
| Req ID | 0x20 |
| Tag | 0xA |
| First DW BE | 0x3 (bytes 0-1) |
| Last DW BE | 0x0 (none) |
| Target BDF | 05:00.1 |
| Bus | 0x5 |
| Device | 0x0 |
| Function | 0x1 |
| Register Offset | 0x000 |
| Register Name | Vendor ID / Device ID |
| Ext Reg Nr | 0x0 |
| Reg Nr | 0x0 |
| Operation | Read Vendor ID / Device ID register at 05:00.1 |
+-----------------+------------------------------------------------+
The leading Source row is the practical advantage over --aer. When several PCIe devices report errors at once — a noisy enclosure, a flaky switch, an enumeration error storm after hotplug — a single lspci -vv capture aggregates everything, and --lspci decodes every Header Log and tells you which BDF produced each one. That distinction matters when you are trying to figure out whether one device is producing all the noise or several devices are independently misbehaving.
The two modes also differ in how they pick up Flit framing — and that is what the next section is about.
The Flit detection problem
On March 24, the day after the patch was applied, the linux-pci discussion surfaced a harder question. Maciej Grochowski — the author of this article and the maintainer of tlp-tool — pointed out an awkward PCIe 6.0 reality: Flit Mode is mandatory at 64.0 GT/s but supported at all link speeds, so a PCIe 6.x link operating below 64.0 GT/s may still be using Flit framing. You cannot simply check the link speed and assume the framing model — and this is not a hypothetical worry. The same message noted that the framing ambiguity is “already a concern among switch and device vendors working through the transition” to PCIe 6.x.
The raw TLP header bytes do not encode which framing produced them — the same bytes decode differently depending on whether they came from classic PCIe or from Flit Mode. The same-bytes-different-meaning ambiguity from the FMT+TYPE comparison above is no longer hypothetical; it is now a real condition in production logs.
Ilpo Järvinen then added the caveat that makes this a real debugging hazard. The Flit Mode Status bit in Link Status 2 is useful only while the link is up — which is not guaranteed in failure scenarios. The kernel tries to compensate by explicitly indicating Flit Mode in its log messages, but the spec creates an asymmetry the kernel cannot fully paper over: AER carries an explicit flag indicating which framing each TLP Log was captured under, while Downstream Port Containment’s (DPC) TLP logging path does not. Järvinen called the DPC omission “botched in the PCIe spec.” To work around it, the kernel saves off Link Status 2 contents and hopes the cached value is still valid when DPC brings the link down — “relatively likely to remain valid,” as he put it, “but fundamentally racy.”
That discussion led directly to tool changes. Grochowski replied that tlp-tool would add auto-detection of the kernel’s (Flit) suffix (introduced in kernel commit 7e077e6707b3, v6.15+) in --aer mode so that mixed Flit and non-Flit TLPs in the same log could be decoded correctly without forcing a global --flit switch. The --lspci mode would read the Flit+ indicator from LnkSta2 per device, while keeping --flit as a manual override for logs that lack markers. The v0.5.1 release notes reflect those changes.
Why this matters now
The good news first: PCIe 1.0 through 5.0 is stable. The header layout that was true in 2003 is still true on hardware shipping today, and tools that decode it work the same way they always have. If your debugging stays within that envelope, nothing has changed and nothing needs to change.
The harder news is what is now sharing the bus next to it. PCIe 6.0’s Flit Mode is the protocol-level shift this article has been about, but the broader pressure on PCIe is the proliferation of accelerators — GPUs, custom AI silicon, FPGAs, smart NICs — putting more devices, more lanes, and more error volume on every system. Users at the edge of that wave will spend more time staring at AER messages, not less; and the maintainers who triage incoming bug reports will receive more raw HeaderLog lines from people who do not yet read TLPs by eye.
tlp-tool is useful on both sides of that exchange. For a user filing a bug report, it turns an opaque four-DWORD line in dmesg into a structured report that names the requester, the target, and the nature of the access — the kind of detail that makes a bug actionable instead of a guess. For a maintainer reading that report, it removes a step: the bit extraction is already done, and the conversation can start at “this is a Type 0 Configuration Read of register 0 on Function 1” instead of “let me decode that for you.” Small per-report, large in aggregate.
The kernel maintainers pointing users at the tool is a quiet acknowledgment that the era of casually reading TLP headers by eye is ending — partly because the protocol made it harder, and partly because the population of people who now need to read them is much larger than it used to be.