feat(labels): rebuild the PDB symbol provider on purepdb - #245
Conversation
PdbSymbolProvider was gated on an optional pdbparse import. pdbparse is
GPL-licensed against a BSD-2-Clause project, unmaintained since 2020,
and was never a declared dependency, so a default install had no PDB
support at all. The import failure was reported at debug level, which
the default logging configuration hides, so a PE with a usable PDB
beside it came back with no names and nothing said why.
purepdb replaces it as a declared, permissively-licensed, pure-Python
dependency with no native code. Names now come from three merged record
sources rather than the global symbol stream alone - S_GPROC32/S_LPROC32
procedures, publics, and S_THUNK32 thunks - and each address is resolved
through the section table the symbols are expressed against, including
OMAP-translated images, instead of by recomputing it here.
Most of what remains unnamed after that is not anonymous functions but
fragments. MSVC-style codegen outlines cold paths and EH funclets into
separate chunks, and SMDA reports each chunk as a function of its own. A
function with no symbol at its entry that falls inside some procedure's
[rva, rva + code size) is now labelled from that procedure as
<parent>$+0x<delta>. The label is parent-relative rather than absolute
so that it survives a rebuild and a different load address, which is
what MCRIT's (pic_hash, function_name) dedup depends on. Import thunks
are excluded from being fragment parents, identified by the linker input
the PDB records for the address rather than by the shape of the name.
Names are reported as the PDB stores them. purepdb does not demangle, by
design, so an MSVC-decorated name now reaches the report decorated where
undname would have expanded it. Rust and current MSVC toolchains already
write undecorated names into the PDB, and there is no maintained,
permissively-licensed MSVC demangler to substitute; a decorated name is
also stable, which an approximate expansion would not be.
Silence is the other half of the bug. A .pdb sitting next to the input
while no PDB has been loaded now warns, as does a file supplied as a PDB
that is not one, and as does a PDB that fails to parse.
Measured with disassembleFile(exe, pdb_path=pdb), naming counted over
all reported functions: a Rust x86_64-pc-windows-msvc binary linked by
rust-lld goes from 1 of 477 functions named to 442 (92.7%), 46 of those
from fragment attribution; sqlite3 x64 built by link.exe from 273 of
4045 to 3698 (91.4%), 97 fragments; sqlite3 x86 from 273 of 4103 to 3730
(90.9%), 111 fragments. The function totals move because recovered
symbols also seed candidates. On the Rust binary the 35 that stay
unnamed all lie past the last procedure record, in ranges the PDB
carries no proc, public or contribution for.
_parseOep is dropped. It only ever saw the first 16 bytes of its target
while PE entry-point extraction needs at least 0x40, so it could not
produce a symbol under any input; PeSymbolProvider supplies
original_entry_point.
tests/rust_pe_msvc_i686{,_pdb}_xored is a new fixture pair for this, a
no_std i686 build linked by rust-lld, small enough to carry and paired
so the PDB can be checked against the image it describes.
Closes danielplohmann#230
Every failure mode purepdb has on a real file is an empty result rather than an exception, which reintroduced the silent no-op through a door the pdbparse version never had. A PDB with no section-header stream parses cleanly and resolves every rva to None, so the loop skips all of them and the provider reports zero symbols; because the parse succeeded it also records the path, which suppresses the sidecar warning. A managed (.NET) PDB behaves the same way - its records are keyed by metadata token and have no address to resolve. Both now warn, quoting the reasons purepdb's own diagnose() gives, so the message says which of the two it was rather than only that nothing came back. A PDB carrying publics but no procedure records is the third case, and it degrades rather than fails: names still resolve at entry points, but with no code sizes there is nothing to attribute a fragment to. That is what /DEBUG:FASTLINK and pre-2010 toolchains produce, and it is worth saying out loud because the difference it makes to naming is large. A PDB with both symbols and procedure records stays silent; verified on the x64 Rust and both sqlite fixtures, whose naming is unchanged.
0.3.0 is the first release carrying the API this provider is written against: Function.module and .aliases, PDB.functions(code_publics=...), and diagnose().warnings. 0.2.0 installs cleanly and shares the version number with a much smaller API, so a floor is the only thing that separates them. Verified against the release from PyPI rather than a source checkout. Naming is unchanged on every fixture: 442 of 477 functions on the x64 Rust binary, 3698 of 4045 and 3730 of 4103 on sqlite x64 and x86, 6 of 6 on the i686 Rust build, with 46, 97, 111 and 0 of those from fragment attribution. Runs remain reproducible and no label carries an address.
A PDB stores Rust names mangled, and this provider reported them verbatim, so a function whose only name lives in the PDB was labelled _RNvCs8JfRDQSkzJ8_15rust_pe_symbols4main. The demangler this codebase already applies to ELF, Mach-O and PE symbol tables reaches none of those names: it is driven from a parsed binary, and a PDB never becomes one -- it arrives as a separate file path. Route each recovered name through the same evidence gate the other providers use, so a name is parsed before it is rewritten. A "_R" prefix alone also matches _RTC_Initialize, _ReadFile@20 and _RoInitialize@4, all real names carrying the function flag in a sqlite3 PDB, and corrupting a good name is worse than leaving a mangled one alone. A name that fails to parse, demangles to an empty path, or raises keeps the spelling the PDB gave it rather than costing the remaining symbols. Fragment labels follow the entry point they are attributed to, so a fragment inside a Rust procedure reads rust_pe_symbols::main$+0x60 instead of carrying the mangled parent. MSVC decoration stays untouched: ?mangled@@yaxxz still comes back as the PDB spells it, since there is no maintained pure-Python undname to substitute. Validated against real PDBs. Every name this rewrites is byte-identical to rustc-demangle 0.1.28, and none of the 7221 names in the sqlite3 x86/x64 PDBs are altered. On a rust-lld/MSVC x64 image only 20 of 399 functions arrive mangled, because rustc writes readable names into its proc records and the mangled spelling survives as a public that merges into an alias; a PDB carrying only publics sees all of them.
|
Pushed a follow-up commit that demangles Rust names on this new provider. Why it belongs here rather than in its own PRThis PR's provider reports every name as the PDB spells it, which for a Rust image means What it doesEach recovered name goes through the same evidence gate the other providers use — exact prefix, full parse, and a hash-suffix requirement for the legacy form — so a name is parsed before it is rewritten. A bare Fragment labels follow the entry point they are attributed to, so a fragment inside a Rust procedure reads MSVC decoration is deliberately untouched — MeasurementsAgainst the fixtures in the purepdb repo — a rust-lld/MSVC x64 image and sqlite3 x86/x64:
Worth recording how few names this actually affects on that image, because it is not obvious: Local gates: full suite green, |
|
Follow-up on the one gap this PR states it leaves open. The body says MSVC decoration is deliberately untouched because "there is no maintained, permissively-licensed MSVC demangler to substitute", and that the half needs its own answer. It has one now: #255 writes that demangler — it reads all 609 names of the reference corpus exactly as Nothing here needs to change. The reasoning in the body was right for this PR at the time, and the decorated-name behaviour it describes is what a reader should still expect from this branch alone. Recording the pointer so the gap does not read as open. |
1 participant
Closes #230.
Dependency
Requires
purepdb>=0.3.0, which is the first release carrying the API this is written against:Function.moduleand.aliases,PDB.functions(code_publics=...), anddiagnose().warnings. The floor matters more than usual here —0.2.0installs cleanly and shares its version number with a much smaller API, so nothing but the pin separates them, and no runtime guard could.Everything below was re-measured against the release installed from PyPI, not a source checkout.
What was wrong
PdbSymbolProviderwas gated on an optionalpdbparseimport.pdbparseis GPL-licensed against a BSD-2-Clause project, unmaintained since 2020, and was never a declared dependency — so a default install had no PDB support at all. The import failure was reported atdebug, which the default logging configuration hides, so a PE with a perfectly good PDB beside it came back with no names and nothing said why.What this changes
purepdbreplaces it as a declared, permissively-licensed, pure-Python dependency with no native code.Names now come from three merged record sources rather than the global symbol stream alone:
S_GPROC32/S_LPROC32procedures, publics, andS_THUNK32thunks. Each address is resolved through the section table the symbols are actually expressed against — including OMAP-translated images — instead of being recomputed here fromimagebase + section.virtual_address + offset.Fragment attribution is the part that moves the number. Most of what remains unnamed after the record merge is not anonymous functions but fragments: MSVC-style codegen outlines cold paths and EH funclets into separate chunks, and SMDA reports each chunk as a function of its own. A function with no symbol at its entry that falls inside some procedure's
[rva, rva + code_size)is now labelled from that procedure as<parent>$+0x<delta>. The label is parent-relative rather than absolute so that it survives a rebuild and a different load address — which is what MCRIT's(pic_hash, function_name)dedup depends on. Import thunks are excluded from being fragment parents, identified by the linker input the PDB records for the address (Import:foo.dllin the Section Contribution table) rather than by the shape of the name.MSVC-decorated names are reported as the PDB stores them. purepdb does not demangle, by design.
pdbparsesuppliedundname, so this is a deliberate behaviour change, not an oversight: an MSVC-decorated name now reaches the report decorated, and there is no maintained, permissively-licensed MSVC demangler to substitute. A decorated name is also stable, which an approximate expansion would not be. Thunk records partly cover the gap on x86, where the thunk carries the undecorated import name against the public's decorated one; purepdb keeps both and the extra spellings land inFunction.aliases.Rust names are demangled, by the demangler this codebase already applies to ELF, Mach-O and PE symbol tables. It could not reach PDB names on its own, because it is driven from a parsed binary and a PDB arrives as a separate file path. Current toolchains write readable names into their proc records, so most names are already legible without this — but not all of them: on the rust-lld/MSVC x64 image measured below, 20 of 399 functions have no proc record and arrive with the mangled public in the name slot, and a PDB carrying only publics would see all of them. Names are parsed before they are rewritten, so
_RTC_Initialize,_ReadFile@20and_RoInitialize@4— real names from a sqlite3 PDB, all matching a bare_Rprefix — are left alone.The silent no-op is fixed. A
.pdbsitting next to the input while no PDB has been loaded now warns; so does a file supplied as a PDB that is not one; so does a PDB that fails to parse.Failing to parse is not the only way to come back empty, though — every failure mode purepdb has on a real file is an empty result rather than an exception, so the silence has a second door. A PDB with no section-header stream parses cleanly and resolves every RVA to
None; a managed (.NET) PDB carries records keyed by metadata token with no address to resolve. Both yield zero symbols from a successful parse. Both now warn, quoting the reasonsdiagnose()gives, so the message says which case it was. The third shape degrades rather than fails: publics but no procedure records —/DEBUG:FASTLINKand pre-2010 toolchains — still names entry points, but with no code sizes there is nothing to attribute a fragment to, and that is worth saying out loud given how much it costs. A PDB with both symbols and procedure records stays silent; verified on all the fixtures below, whose naming is unchanged.Measured
Disassembler.disassembleFile(exe, pdb_path=pdb), naming counted over every reported function. The issue's binary is not in hand, so these are different binaries and different numbers.x86_64-pc-windows-msvc, rust-lldlink.exelink.exeno_stdFunction totals move because recovered symbols also seed candidates.
On the Rust x64 binary the 35 that stay unnamed all lie past the last procedure record, in ranges the PDB carries no proc, no public and no section contribution for — precompiled
core::fmtbodies from the std rlibs. That is the ceiling for that file, not a gap in the attribution.Stability, checked on all four: analyzing the same input twice produces an identical
{offset: name}map; no label contains a virtual address; and a fragment label is unchanged when the same PDB is applied at base0x400000versus0x10000000.Also in this change
_parseOepis dropped. It only ever saw the first 16 bytes of its target while PE entry-point extraction needs at least0x40(e_lfanewlives at0x3C), so it could not produce a symbol under any input;PeSymbolProvideris what actually suppliesoriginal_entry_point.tests/rust_pe_msvc_i686{,_pdb}_xoredis a new fixture pair, xored like the other binary fixtures. It is purepdb's own i686 groundtruth build —#![no_std] #![no_main], linked by rust-lld with/nodefaultlib, so it links against no CRT and no import library and involves no Microsoft-licensed material — regenerable from thebuild.shandmain.rsbeside it there. 84 KB for the pair, and paired so the PDB can be checked against the image it describes.Validation
Against
purepdb==0.3.0from PyPI:ruff check .andruff format --check .clean;make typecheckexits 0 with no new diagnostics;pytest tests/1207 passed, 1 skipped, 1480 subtests;diff-cover --fail-under=100reports 60/60 changed lines covered. A PE with no PDB beside it is unaffected — covered by a test that removes the PDB and asserts the previous naming.Known follow-up
The same "confirmed the format, then failed silently" shape exists outside this provider: the CIL, Rust, Mach-O, Delphi and Go symbol providers and all three file loaders report a whole-parse failure at
debug, and two of those sites log nothing at all. Those are left out here to keep this change to one subject; they need failure-path fixtures the suite does not currently have.