feat(labels): demangle MSVC decorated symbol names - #255
Conversation
Demangled names were passed through a port of Ghidra's CondensedString before being stored, so <Error as core::fmt::Display>::fmt reached the report as <Error_as_core::fmt::Display>::fmt and A, B became A,B. Ghidra needs that because a Ghidra symbol name cannot contain spaces; SMDA has no such constraint, and already stores names with spaces from another source -- 39 of the 399 names recovered from a PDB on a rust-lld x64 image contain them, because MSVC proc records spell them that way. The condensation was applying one spelling rule to names from a symbol table and another to names from a PDB, in the same report. Removing it exposed a printer bug it had been masking: a function pointer's ABI ran into its fn, giving unsafe extern "C"fn(*mut u8). The closing quote was emitted without the trailing space, and comparisons against the reference implementation could not see it because the condensation deleted that space from both sides. One existing test asserted the wrong spelling and is corrected. Measured against rustc-demangle 0.1.28 on 147 real symbols read out of a rust-lld x64 image: 93 of 147 matched byte for byte before, 146 after dropping the condensation, and 147 after the ABI fix. Cross-checked against the 18 mangled symbols in the reference implementation's own test corpus: 8 match, 9 raise and so keep the name the binary gave them, and 1 differs -- a punycode identifier this port renders as a placeholder, which is pre-existing and left alone here.
src/smda/common/labelprovider/rust_demangler/ is vendored third-party code and carries no attribution. The commit that introduced it says so directly -- it vendors the rust_demangler package by Team bi0s, MIT licensed, with robustness behaviour reimplemented from Ghidra, which is Apache-2.0. Neither is recorded: LICENSE covers only this project's own BSD 2-Clause, there is no NOTICE, and the package has no header. The MIT licence asks that its copyright and permission notice travel with copies and substantial portions, and this code is redistributed in the wheel published to PyPI. Add a NOTICE carrying the MIT copyright line and permission text and a pointer to Apache-2.0 for the Ghidra-derived parts, a docstring on the vendored package pointing at it, and a line in the README credits. The in-code comments this attribution was drawn from named "Ghidra's rust-demangle.c", and no such file exists: Ghidra's Rust demanglers are Java, at Ghidra/Features/Rust/src/main/java/ghidra/app/plugin/core/ analysis/rust/demangler/RustDemanglerV0.java and RustDemanglerLegacy.java. Name those instead, in the NOTICE and in the two comments that were the source of the error, and record that Ghidra's own V0 demangler is a port of the rustc-demangle crate, which is the real upstream of the behaviour. The NOTICE lists every vendored component rather than only the one that prompted it, so Tarjan.py and DominatorTree.py -- both already credited in the README paragraph the new line joins -- appear alongside it. No pyproject change is needed: setuptools' default license-file glob already matches NOTICE*, and a build confirms it reaches both the wheel (dist-info/licenses/NOTICE) and the sdist.
Two reasons a PE reached the report with mangled names. RustSymbolProvider recovered nothing at all from a PE. Its COFF loop skipped a symbol when Symbol.section was None, and lief never populates that attribute for PE -- PeSymbolProvider already says so in a comment and resolves through section_idx instead. On the bundled mingw-linked Rust fixture, 0 of 4818 symbols carry a section while 4366 carry a usable section_idx, so the guard rejected every one, the provider contributed nothing, and PeSymbolProvider's raw spellings won by default: 2098 names reached the report spelled _RNv... Resolving through section_idx recovers those 2098 and none of them stay mangled. PeSymbolProvider never demangled anything. ElfSymbolProvider and MachoSymbolProvider both route recovered names through a demangler, while this one stored them verbatim, so a mingw-built C++ PE reported _ZN12FileExplorerC2Ev rather than a signature. It now applies the same Itanium helper ElfSymbolProvider uses, which leaves every non-Itanium name untouched. Making that COFF loop live also made the gate in front of it matter, and it was the wrong one. _is_rust_symbol tested the prefixes alone, but legacy Rust mangling shares _ZN with the C++ Itanium ABI, and this provider is consulted before the format providers -- so on a Rust binary that also carries C++ symbols it would have claimed names like _ZN4test4funcEv and replaced the full Itanium signature "test::func()" with the "test::func" the Rust legacy demangler degrades it to. It now uses the shared evidence gate, which parses a name before claiming it. Recovery on the fixture is unchanged at 2098 names. No bundled PE carried Itanium C++ symbols to show the second fix on, so add one: a small C++ translation unit compiled for x86_64-w64-mingw32 by g++ 16.2.0. Before the change it yields three mangled names and nothing readable; after it, three readable signatures and nothing mangled. The mocks in testRustSymbolProvider modelled a PE symbol through Symbol.section, which no real PE symbol has, so they exercised a path that could not work on a real binary; they now carry section_idx, and the provider reads it directly rather than through a getattr default that would turn a future lief rename back into "skip every symbol". Several tests used _ZN3foo3barE as their Rust fixture, which has no 17h<hash> suffix and so is Itanium C++ rather than legacy Rust; they now use a name that really is Rust-legacy, and the PE provider test expects foo::bar from the C++ demangler.
68f5cff to
6ea1187
Compare
MSVC decoration was the one mangling scheme with no demangler here, so a binary built with the Microsoft toolchain reported ?__crt_rotate_pointer_value@@YAIIH@Z where a GCC-built one reported a signature. Add one, and let the PE provider pick a demangler by the decoration a name carries. The contract is that a name comes back either as the reference implementation spells it or exactly as it went in, never as a third spelling. That matters more than coverage does: a decorated name is still a usable identity, matchable against a symbol server or another report, while a confidently wrong expansion matches neither the decorated form nor the real one. Every construct the parser does not fully model therefore declines. A type is built as a tree and spelled around a declarator rather than rendered eagerly, because C nests the two: the name belongs inside its own type. That is what makes int (__stdcall *j)(signed char) and int (* __cdecl f(void))[2] come out right, and it is why a function pointer used as a data declaration, a return type, or the pointee of another pointer are all ordinary cases here instead of refusals. Measured against llvm-undname on the corpus LLVM tests its own demangler with -- names from llvm/test/Demangle/ms-*.test, a compiler stress suite rather than a sample of ordinary binaries -- 363 of 609 are spelled identically, 246 come back untouched, and none get a third spelling. On the MSVC names carried by real PDBs: 10 exact of 14, none wrong. The corpus ships as test data so the demangler is measured against a reference rather than against its own output. Understood: qualified and nested names, name and argument back-references, the basic and extended type set, pointers and references with their own and their pointee's qualifiers, __restrict, arrays, function pointers, declarator composition over all of those, variadic lists, tagged types, templates, data symbols with their storage class, constructors, destructors, operators and the vftable family. Declined: back-references resolved against a table holding a template render, whose numbering does not follow the obvious rule; qualified back-references, which the reference implementation does not form; local-scope and lambda names; RTTI descriptors. The grammar comes from Microsoft's documented format, with llvm-undname settling the cases the documentation leaves ambiguous -- pointer qualifier composition, where PBQAD and PAQAD both spell char *const *; the storage classes 3 and 4, which both render with no prefix; and the spacing around a declarator, where a pointer abuts a name but is separated from a nested function declarator. Both resources a hostile name can spend are bounded. Depth is capped below the point where CPython's own recursion limit could be what stops a parse, so the answer cannot depend on how deep the caller already is. Each rendered type is capped too: argument back-references can reuse an earlier rendering repeatedly, which grew a 132-byte name into half a gigabyte of output before the bound. A symbol table holds whatever bytes were written into it, so every prefix of every corpus name is fed through in the tests and none may raise, malformed shapes each have a case asserting they come back untouched, and a hypothesis target checks that an expansion is printable and bounded rather than merely a string.
6ea1187 to
415deb2
Compare
Differential fuzzing against llvm-undname 22.1.7 - 15044 names derived by mutating and splicing the reference corpus - found 256 shapes this demangler expanded confidently while the reference refused them. A decorated name is a usable identity, so answering where the grammar does not is the one failure this demangler is built to avoid. Five tightenings take that to 4. The largest class was a missing terminator. A parameter list is closed by the throw specification, so a fixed list ends "@z" and a variadic one "ZZ", but the terminator was consumed only when the parser was not already at end of input. Any name truncated just past its parameters therefore got a plausible answer: "?a2@@Yahx" read as "int __cdecl a2(void)" and "?f@@YAXHZ" as a variadic call whose trailing byte was never there. The extended integer table carried _D through _I. No Microsoft-compatible mangler emits those - __int8, __int16 and __int32 are spelled with the plain char, short and int codes - so a name containing one is not MSVC-decorated. _L and _M stay: clang's Microsoft mangler does emit them for __int128, which is also why they are spelled from the mangler's table rather than the reference's, the reference having no reader for them. The remaining three each restore a rule the grammar states and the parser did not check: a template name is an identifier, not a digit or an operator code; a special name takes a signature or a storage class by which code it is, never both, so a vftable with a parameter list and an operator with a vftable's storage class are equally malformed; and a reference cannot carry a qualifier from the type enclosing it, which had been rendering as "char const &const volatile *". Every change refuses more and expands nothing new. The reference corpus is unchanged at 363 of 609 exact with none given a different spelling, and the MSVC names carried by real PDBs are unchanged at 11 of 14.
|
Pushed 099ac77, which hardens the demangler against names the grammar does not actually allow, and corrects one published number. The corpus proves the demangler reads well-formed names correctly, but says nothing about what it does with the rest — and a symbol table holds whatever bytes were written into it. So I used the corpus as fuzzer seed instead: 15044 candidates built by truncating, splicing and mutating the 609 reference names, each fed to both implementations, asserting that wherever this demangler answers it agrees with 256 shapes came back expanded confidently while the reference refused them. The largest class was a missing terminator: a parameter list is closed by its throw specification, but that byte was only required when the parser was not already at end of input, so That is 256 → 4, and every change refuses more rather than expanding anything new: the corpus is unchanged at 363 of 609 exact with 0 given a different spelling. The correction: the real-PDB figure is 11 of 14, not the 10 stated earlier. I re-measured the committed tree before and after this change and both give 11, so it was a measurement error on my side rather than a regression — the remainder is 3 local-scope names, not 4. The body now reflects that. Two divergences from the reference are deliberate and now documented in the body, because copying it would make the output worse in both: |
A return type is the one position that carries a qualifier of its own, and that qualifier is not optional decoration: every function returning a class by value is spelled with it, "?A" being the unqualified case rather than an absent one. Nothing here parsed the prefix, so those names were refused - "?combine@geometry@@ya?AUMatrix@@aebu2@NI@Z" among them. A return type is parsed at three places: a plain function, a function pointer, and a "$$A6" function type. All three take the prefix, so the three now share one reader rather than each calling type() directly. This came out of building a fixture with the toolchain the demangler is meant to read. No PE available here carries MSVC-decorated names - sqlite3's 277 exports are all C - so the MSVC arm of the PE provider had only synthetic symbols behind it, and its first real export was one it could not read. tests/msvc_cxx_pe_xored is that binary: a small translation unit built for x86_64-pc-windows-msvc by clang-cl 22.1.7, exporting free functions, a namespace, a class returned by value, and an extern "C" name. All four of its decorated exports now come back as signatures. Worth twenty names on the reference corpus, 363 to 383 of 609, none given a different spelling. Twenty-one corpus names used the form the whole time; they sat in the declined bucket, where a single aggregate count kept them from reading as a gap.
|
Pushed 407537c, which adds the last missing piece of the type grammar — found by finally building a binary with the toolchain this is meant to read. No PE available to me carries MSVC-decorated names (sqlite3's 277 exports are all C), so the MSVC arm of the PE provider had only synthetic symbols behind it while the Itanium arm had a real fixture. Its very first real export was one the demangler refused — Corpus goes 363 → 383 of 609 exact, still 0 given a different spelling, and all four decorated exports of the new fixture come back as signatures. Worth stating plainly, since it is the part I got wrong: 21 corpus names had used this form the whole time. They sat in the 246-name declined bucket, where a single aggregate count kept a common C++ construct from reading as a gap — a 0-wrong corpus and 100% branch coverage were both green throughout. The differential fuzzing in the previous commit could not have found it either, because it only asks whether a wrong answer is given, never whether a right one is missing. |
1 participant
MSVC-decorated names were the one mangling scheme SMDA carried no demangler for, so a C++ binary built with the Microsoft toolchain reported
?__crt_rotate_pointer_value@@YAIIH@Zwhere a GCC-built one reported a signature. This adds one and wires it into the PE provider.Part of a linear stack: #249 → #250 → #254 → #255. They are ordered that way because #249 and #254 both touch
RustSymbolProviderand would otherwise conflict in a way that silently breaksimport smdaif resolved the obvious way, and because #250 creates theNOTICEthis PR extends. Merging in that order needs no conflict resolution.The contract
A name comes back either as the reference implementation spells it, or exactly as it went in. Never as a third spelling.
That matters more here than coverage does. A decorated name is still a usable identity — it can be matched against a symbol server, an import table, or another report. A confidently wrong expansion is worse than no expansion, because it matches neither the decorated form nor the real one. So every construct the parser does not fully model declines rather than guessing, and the tests assert this over the whole corpus rather than sampling it.
Measurements
llvm-undnameis the reference. The corpus is the one LLVM tests its own demangler against — 609 unique names drawn fromllvm/test/Demangle/ms-*.test(the upstream files repeat 15), which is a compiler stress suite rather than a sample of ordinary binaries: it is full of lambdas, anonymous namespaces, member pointers and deeply nested templates.llvm-undnamespells itOn the MSVC names carried by real PDBs (the sqlite3 x86/x64 and rust-lld builds): 14 names, 11 exact, 0 wrong. The remaining 3 are local-scope names.
And on a PE built by the toolchain this is meant to read, all four decorated exports come back as signatures.
tests/msvc_cxx_pe_xoredis that binary, added here because none was available: no PE on hand carries MSVC-decorated names — sqlite3's 277 exports are all C — so this arm of the PE provider had only synthetic symbols behind it. It is a small translation unit built forx86_64-pc-windows-msvcby clang-cl 22.1.7, exporting free functions, a namespace, a class returned by value and anextern "C"name.Its first real export was one the demangler could not read, and the reason generalised. A return type is the one position carrying a qualifier of its own, and that qualifier is not decoration: every function returning a class by value is spelled with it,
?Abeing the unqualified case rather than an absent one. Parsing it — at all three places a return type is read — is worth 20 corpus names on its own. Twenty-one corpus names had used the form the whole time; they sat in the declined bucket, where a single aggregate count kept them from reading as a gap.The corpus ships as
tests/msvc_reference_corpus.txtso the demangler is measured against a reference rather than against its own output, and a regression that starts emitting third spellings fails the build rather than going unnoticed.What it understands
Qualified and nested names, name and argument back-references, the basic and extended type set, return types with their own qualifier, pointers and references with their own and their pointee's qualifiers,
__restrict, arrays, function pointers, declarator composition over all of those, variadic parameter lists, class/struct/union/enum types, templates, data symbols with their storage class, constructors, destructors, operators, and thevftable/vbtablefamily.A type is built as a tree and spelled around a declarator rather than rendered eagerly, because C nests the two — the name belongs inside its own type. That is what makes
int (__stdcall *j)(signed char, unsigned char)andint (* __cdecl f(void))[2]come out right, and it is why a function pointer used as a data declaration, a return type, or the pointee of another pointer are ordinary cases rather than refusals.What it still declines: name back-references resolved against a table containing a template render (the numbering does not follow the obvious rule and I would rather stop than emit a plausible wrong name), qualified back-references (which the reference implementation does not form either), operator templates, member function pointers, local-scope and lambda names, and RTTI descriptors.
How the grammar was established
From Microsoft's documented decorated-name format, with
llvm-undnameused to settle every case where the documentation is silent or ambiguous — pointer qualifier composition in particular, wherePBQADandPAQADboth spellchar *const *, was derived by probing the reference rather than reasoned about.Refusing what the grammar does not allow
A corpus of well-formed names only proves the demangler reads what it should. It says nothing about what it does with the rest, and a symbol table holds whatever bytes were written into it. So the corpus was also used as fuzzer seed: 15044 candidates built by truncating, splicing and mutating those 609 names, each fed to both implementations, asserting that wherever this demangler answers it agrees with the reference.
That found 256 shapes it expanded confidently while the reference refused them. Five fixes take it to 4, every one of them refusing more rather than expanding anything new:
@Z, variadic onesZZ. The terminator was consumed only when the parser was not already at end of input, so any name truncated just past its parameters got a plausible answer:?a2@@YAHXread asint __cdecl a2(void), and?f@@YAXHZas a variadic call whose last byte was never there._Dthrough_Iare not type codes. No Microsoft-compatible mangler emits them —__int8,__int16and__int32are spelled with the plainchar,shortandintcodes — so a name carrying one is not MSVC-decorated.vftablewith a parameter list and anoperator!with a vftable's storage class are equally malformed.char const &const volatile *.Two divergences from the reference are deliberate, and both are cases where copying it would make the output worse:
_Land_M(__int128) are spelled from clang's Microsoft mangler, which emits them, becausellvm-undnamehas no reader for what its own compiler produces.??_Xand??_Yare spelled from Microsoft's table as the placement delete closures. The reference recognises them structurally but emits an empty name —public: int __cdecl Base::(int)— which is not a usable identity.Bounded against hostile input
Both resources a crafted name can spend are capped. Depth stops below the point where CPython's own recursion limit could be what ends a parse — otherwise the answer would depend on how deep in SMDA the provider happened to be called, which I verified is identical across three caller stack depths. The rendered result is capped too: argument back-references can reuse an earlier rendering repeatedly, which grew a 132-byte name into 496 MB of output (2.5 GB resident, retained by the cache) before the bound; it now declines in 0.02 ms.
Robustness
Every prefix of every corpus name is fed through the demangler in the tests; none may raise. Malformed shapes — empty fragments, out-of-range back-references, unknown calling conventions, trailing bytes, missing terminators — each have a case asserting they come back untouched.
Validation
The new module is at 100% statement and branch coverage.
Follow-up
The PDB provider is untouched here. On master it still gets MSVC names expanded by
pdbparse; once #245 replaces that with purepdb, which does not demangle, this is what should fill the gap — but that belongs on top of #245 rather than in this PR.