Skip to content

fix(ghidra)!: load ESP-IDF flash code in Ghidra's default address space - #8464

Open
Entrpi wants to merge 23 commits into
morluto:mainfrom
Entrpi:fix-ghidra-esp-idf-flash-overlay
Open

Entrpi wants to merge 23 commits into
morluto:mainfrom
Entrpi:fix-ghidra-esp-idf-flash-overlay

Conversation

@Entrpi

@Entrpi Entrpi commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #8461. Part of the embedded roadmap #7955.

ESP-IDF's linker scripts for ESP32-C5, C6, C61, H2, H21 and H4 reserve the addresses of .flash.text with the NOBITS section .flash_rodata_dummy (source). Ghidra 12.1.4 then loads .flash.text into an overlay address space, which its DWARF import, call-frame processing and default-space references do not reach. Observed on ESP-IDF v6.1 ESP32-C6 (REA's CLI and analyzeHeadless) and v5.5.2 ESP32-H4 (analyzeHeadless only):

  • most DWARF-described flash functions get no imported signature (18 of 549 on C6 and 6 of 278 on H4 have one), and so no DWARF local variables either (source-derived);
  • esp_restart and panic_restart get no no-return flag;
  • each flash function gets a one-byte shadow function;
  • an overlay-reference probe found 48 (C6) and 27 (H4) computed calls from .iram0.text that resolve to shadow functions;
  • pseudocode runs past esp_restart().

This PR moves the code into Ghidra's default address space before auto-analysis, so Ghidra's own analyzers handle it. The RISC-V lanes refuse every other section overlap that involves executable code. ESP-IDF v5.5.2's C5 and C61 builds add a second reservation, .ext_ram.dummy, so they are now refused.

Problem and expected behavior

ESP-IDF's linker script reserves the addresses of .flash.text with .flash_rodata_dummy, in a second PT_LOAD at the same address (0x42000020). Ghidra's ELF loader gives the default space to the later section, the reservation.

Observed on main 6e9ee97a6 with ESP-IDF v6.1's ESP32-C6 hello_world: rea function v61-c6.elf app_main reports .flash.text:0x42008398 as undefined __rea_ilp32 app_main(). Its pseudocode continues into riscv_decode_offset_from_jal_instruction.

Expected: flash code at its ELF address with its DWARF facts, as IRAM code already has. A RISC-V image is refused when a section that overlaps executable code does not have the verified reservation shape, or when its section table cannot be read. Overlaps between data sections only, and images without a section table, stay admitted as before.

Change and scope

  • Format fact (src/domain/elfSectionOverlap.ts). For RISC-V ET_EXEC images, target resolution checks whether an executable section shares allocated addresses with another section.
    • It keeps one record per allocated section, at most 65,280 (SHN_LORESERVE), sorts the records by address and sweeps once. The sweep stops at the second overlap that involves executable code. A table with more allocated sections is unavailable: NOBITS headers take no file bytes, so the file size alone would not bound the records.
    • It records none, code-reservation, unsupported with the two sections that cause it, or unavailable.
    • The ISA and overlap readers share one loop and read the section table in one pass (readRiscvElfDeclarations, visitElfSectionHeaders).
  • Lane policy (src/ghidra/GhidraRiscvProfile.ts).
    • When one NOBITS section starts at an executable section's address and covers it, and no other section overlaps either, the lane commits both sections to the analysis profile as elf_load_preparation.
    • Every RISC-V lane refuses any other overlap that involves executable code, naming the two sections with their ranges, and a section table that cannot be read.
  • Ghidra. bridge/ghidra/ReaGhidraPrepareElfLoad.java runs before seeds and auto-analysis.
    • It checks Ghidra's blocks against the plan: address, size, initialization, execute permission and file offset. No other block may share the code's overlay space.
    • It moves program content into a new default-space block that maps the same file range. The emptied overlay block becomes the uninitialized reservation, which maps no file bytes.
    • It retargets the references into the code to the new block: its entry points and the loader's section-header reference. Labels elsewhere in the code's overlay space, such as ESP-IDF's _etext just past the code, move to the default space too. It refuses if any symbol or other reference remains in that overlay space.
    • It reports moved with the counts of the result, or canonical when Ghidra already loaded the code in the default space. The report echoes the blocks it left. The client checks their addresses, sizes, file offset and address spaces against the profile, as it does for the calling-convention policy; block names are display only. A failure during the move refuses the session. A refusal also turns off Ghidra's headless auto-analysis, so Ghidra does not analyze a program REA discards; the bridge still reports the reason.
  • Session start (src/ghidra/GhidraProviderClient.ts). Upstream fix(ghidra): reject uncommitted handshake reports before the first query #8630 now starts every query's session through the checked handshake, so a preparation or calling-convention report that the profile did not commit closes the session before any analysis call. The merge of main drops this branch's own start of every RISC-V session. The flag that selects recovery guidance also counts a committed load preparation, as it counts the calling-convention policy, so if Ghidra exits during the first query of such a session, the error tells the caller to reopen it.
  • Ghidra 12.1.4 constraints the pre-script works around:
    • Memory.moveBlock keeps a block in its address space, so the script uses ProgramDB.moveAddressRange, the call moveBlock itself makes.
    • Removing any overlay space stops the native decompiler from registering the program, so the code's overlay space becomes the reservation's. It takes the name Ghidra's memory map gives an overlay block: : and whitespace become _, and a name already in use gains .1, .2 and so on, so a reservation named like another address space, such as ram, still loads.
    • moveAddressRange moves only the content of the range: not the references into it, and not a label just past its end.
  • Precedent. ReaGhidraPrepareDol.java also rebuilds Ghidra's memory map before analysis from facts that REA's TypeScript admission validated.
  • Not included:
    • images with more than one reservation, such as the C5 and C61 builds (a possible follow-up policy);
    • making computed calls to flagged no-return functions terminal in the Listing (a possible follow-up; the pseudocode is already right);
    • overlaps between data sections only;
    • other architectures' ELF lanes;
    • the ilp32f lane (feat(ghidra): admit RISC-V ilp32f ELF with a derived calling convention #8401).

The scope questions for the maintainer are in #8461.

Contract and boundary impact

  • Semantic owner and earliest changed stage: the section facts come from REA's ELF reader at target resolution, the preparation plan and refusal text from the Ghidra RISC-V lane policy, and the memory map from the Ghidra import (pre-script). The ISA reader's own section-table loop is removed.
  • Contribution scope fit and linked maintainer decision: this improves analysis correctness inside the existing Ghidra lane. Like the DOL pre-script and the RISC-V calling-convention script, it prepares only the ephemeral analysis database. Executable bytes are never written. Questions: [Bug] Ghidra RISC-V lanes lose DWARF and no-return facts for ESP-IDF flash code #8461.
  • CLI and MCP/tool-catalog contract: no schema change.
    • For affected images, flash procedures report their ELF address (0x42008398) instead of .flash.text:0x42008398.
    • Affected sessions add two limitations: the layout, and the plan or its outcome.
    • In every RISC-V session, the admission limitation changes from "Admission reads e_type only: …" to "Admission reads e_type and section overlaps, not program headers: …".
    • New admission refusals name the two sections that cause them.
  • Provider, bridge, target-format, or platform compatibility: verified against Ghidra 12.1.4 on Linux. GhidraReleasePolicy also accepts other 12.x releases from 12.1 onward; the pre-script is not verified on them. Windows P0 accepts only PE and raw 6502 targets (GhidraProvider.ts), so it admits no RISC-V ELF. On a Ghidra that already loads the code in the default space, the pre-script reports canonical and changes nothing.
  • Evidence, artifact, provenance, or reconstruction contract: only affected images commit elf_load_preparation to their analysis profile: policy, both sections' index, name, address and size, and the code's file offset. Their digests change and their old snapshots are not reused. A test pins main's digests for two synthetic profiles without an overlap (ilp32 and double64); other images without an overlap add no parameter (source-derived). Project history keeps the stored report, with the blocks' original address spaces.
  • Target selection, authority, upstream behavior, and cleanup details: not changed.
  • Generated metadata, package, or installation impact: bridge/ghidra/ReaGhidraPrepareElfLoad.java is added to package.json files.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Tests added or updated:

    • tests/boundary/filesystem/elfSectionOverlap.test.ts: 18 cases through the real target resolver. They cover the ESP-IDF shape, no names table, data-only overlaps, eight refused shapes that each must name their cause (one is the C5 and C61 shape), a table of 10,000 executable sections at one address, tables of 65,280 and 65,281 allocated sections, an ELF64 table, a truncated table and ET_REL.
    • src/ghidra/GhidraRiscvProfile.test.ts: main's profile digests pinned for two synthetic profiles, plus the plan, profile, refusals, report comparison, handshake limitations, the check before the first lazily started call, and project history.
    • src/ghidra/GhidraSessionValues.test.ts: the report schema.
    • tests/boundary/providers/ghidra/ghidraLauncher.test.ts: pre-script arguments and order, scope and existing-project refusals.
    • npm run verify:ghidra:riscv: two ESP-layout fixtures (tests/conformance/c/riscv-esp-layout.c with tests/conformance/esp/esp32c6-shared-window.ld and esp32c6-reservation-first.ld); see below.
  • Base reproduction (observed). The fixture's checks, run through the CLI against main 6e9ee97a6, fail all 4:

    • rea_riscv_esp_main is at .flash.text:0x42000064;
    • its pseudocode ends rea_riscv_esp_restart(); rea_riscv_esp_sink = 0x5a5a; return;, which is the next function's store;
    • no preparation is committed;
    • the signature is rea_riscv_esp_sum(int param_1,int param_2).

    On this branch all 4 pass: 0x42000064, /* WARNING: Subroutine does not return */ rea_riscv_esp_restart(); }, and (int left,int right). That comparison ran on a266cadfb; the lane checks the same 4 on 78a413531, 43a8517fd and af6aab239.

  • What the RISC-V lane checks on the ESP fixture:

    • the load image maps .flash.text in the default space and the reservation as an uninitialized overlay; only the moved block maps the code's file bytes, and a file-offset: selector reaches the flash function;

    • no entry point has an overlay address, and each flash function keeps its entry mark; the absolute _text_start and _text_end keep their addresses, the section-relative _etext is at the code's end in the default space, and _flash_rodata_dummy_start is in the overlay;

    • the outcome reports 4 moved functions, the entry points the load image shows in the code's range and 1 overlay label; retained Evidence keeps it, and the reopened snapshot lists the same procedures;

    • the reservation-first build reports canonical;

    • a copy with a short reservation is refused over CLI and MCP with the reservation named, and analyzeHeadless with a code file offset one byte off makes the pre-script refuse;
      , and Ghidra then runs no auto-analysis;

    • a build whose reservation is named ram, the default space's name, reports moved, with the reservation in the overlay space ram.1;

    • a double-float copy with a truncated section table is refused when REA opens it.

  • Real ESP-IDF images (observed, Ghidra 12.1.4, Linux).

    • v6.1 ESP32-C6 through REA's CLI:

      • on main 6e9ee97a6: app_main at .flash.text:0x42008398 as undefined __rea_ilp32 app_main(), pseudocode that continues into riscv_decode_offset_from_jal_instruction, and a calling-convention outcome of 1502 functions and 541 definitions;
      • on 16ecb0fc1: 0x42008398 as void __rea_ilp32 app_main(void), pseudocode that ends fflush(stdout); /* WARNING: Subroutine does not return */ esp_restart(); }, and 954 functions and 1089 definitions;
      • inspect-native-load-image: 2,292 entry points on a266cadfb, 331 of them in the reservation overlay; 1,961 on 16ecb0fc1, none in an overlay.
    • ESP-IDF v5.5.2 hello_world builds through REA's CLI:

      • H2 on main: .flash.text:0x420077b4 as int __rea_ilp32 app_main(void), with pseudocode that continues into the next function, and 1359 functions and 577 definitions. On 16ecb0fc1: 0x420077b4 as void __rea_ilp32 app_main(void), ending at esp_restart(), and 912 functions and 1023 definitions.
      • H21 on 16ecb0fc1: 0x42002b0e as void __rea_ilp32 app_main(void), ending at esp_restart().
      • C5 and C61 on 16ecb0fc1: refused before analysis, with .ext_ram.dummy named (output below). Their PSRAM builds were refused the same way on 42243f457, which has the same TypeScript admission. C5 on main: app_main at .flash.text:0x42006430, with pseudocode that does not end at esp_restart().
      • A copy of the C5 build without the .ext_ram.dummy section header (llvm-objcopy --remove-section=.ext_ram.dummy) is admitted and prepared on 16ecb0fc1: app_main at 0x42006430 ends at esp_restart().
    • H4 and C6 through analyzeHeadless, with the pre-script of 16ecb0fc1 and feat(ghidra): admit RISC-V ilp32f ELF with a derived calling convention #8401's post-scripts at ba174afe6 (H4 needs feat(ghidra): admit RISC-V ilp32f ELF with a derived calling convention #8401's ilp32f lane). "Before" is main, without the pre-script. The pre-script reported moved with 278 functions, 187 entry points and 1 overlay label (H4), and 549, 331 and 1 (C6).

      H4 before → after C6 before → after
      Imported signatures (DWARF-described flash functions) 6 → 278 of 278 18 → 549 of 549
      No-return flags 6 → 8 6 → 8
      Pseudocode sites past a no-return call 4 → 0 4 → 0
      Shadow functions 278 → 0 549 → 0
      Functions 914 → 636 1,506 → 957
      Functions whose Listing still falls through a no-return call (the possible follow-up) 48 → 44 122 → 118
  • DWARF-specific checks, separate from the imported-signature counts: the fixture's rea_riscv_esp_sum(int left,int right), and C6 app_main's void app_main(void), both from the image's DWARF.

  • User-visible CLI/MCP output (C6, from this branch):

    • "RISC-V ELF load layout: NOBITS section .flash_rodata_dummy reserves the addresses of executable section .flash.text (0x42000020, 0xfff2 bytes). When the reservation follows the code in the section table, Ghidra 12.1.4's ELF loader puts the code in an overlay address space, which its DWARF import, call-frame entries and default-space references do not reach; see docs/ghidra-riscv.md."
    • "RISC-V load preparation outcome: before analysis REA moved .flash.text from overlay space .flash.text into default space ram, with the same file bytes and offsets, and kept .flash_rodata_dummy as uninitialized overlay space .flash_rodata_dummy. Functions in the moved block: 549; entry points: 331; labels in the overlay: 1."
    • The v5.5.2 C5 build's refusal: "Sections .flash.text (executable, 0x42000020, 0xf7c4 bytes) and .ext_ram.dummy (NOBITS, 0x42000020, 0x1ffe0 bytes) share addresses, so Ghidra 12.1.4's ELF loader would put one of them in an overlay address space. This lane prepares only one NOBITS section that starts at an executable PROGBITS section's address and covers it, with no other section overlapping either; see docs/ghidra-riscv.md."
  • Coverage that is synthetic only, or missing:

    • malformed section tables reach the lane only as synthetic facts in unit tests; REA rejects a truncated table when it opens the file;
    • a committed plan that does not match the target, and a report whose blocks or address spaces contradict the plan, are rejected in TypeScript (unit tests);
    • in Ghidra, the pre-script's refusal of a wrong file offset runs in the lane, before anything moves. Its refusal of references left in the code's old overlay space ran once in the lane, on 82d0df4e0, and led to 42243f457; no test covers it now. Its refusal of another block in that overlay space, and its handling of an exception during the move, never ran. If Ghidra rolls back a failed script's changes, that refusal's reason is lost and the session is refused without it (inferred).
  • Remaining proof gaps:

    • H4 through REA's own CLI (it needs feat(ghidra): admit RISC-V ilp32f ELF with a derived calling convention #8401);
    • the preparation in the double-float RV32 and RV64 lanes, which only unit tests cover;
    • an entry mark that Ghidra's loader gives a reservation label inside the code: the mark stays, and Ghidra could start a function there. REA neither removes nor checks it; no surveyed image has one (source-derived);
    • C5, C61, H2 and H21 beyond ESP-IDF v5.5.2 hello_world builds;
    • apps other than hello_world;
    • Ghidra releases other than 12.1.4, and hosts other than Linux.

The change adds no lifecycle stage. The pre-script runs inside the existing headless import; a refusal or a failure there refuses the session, and a refusal also skips auto-analysis.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

On the head d378c361c, which merges upstream main c3647ade1 (one conflict, with #8630's checked start, in GhidraProviderClient.ts) and adds the two review fixes:

  • npm run check:fast — pass.
  • npm run test:focused -- 18 files: the Ghidra RISC-V profile, session-value, launcher, client and provider-client tests — 810 passed, 3 skipped.
  • npm run verify:ghidra:riscv (Linux container, Ghidra 12.1.4) — pass: seven fixtures, "mocked": false, process lineage verified with no descendants, cleanup complete. The refusal run without -noanalysis logs no analysis, and the ram reservation moves to ram.1. verify:ghidra:bridge-sources passed on da567c42e, whose bridge sources are the same.

On af6aab239, which started every RISC-V session through the checked handshake:

  • npm run check:fast — pass.
  • npm run test:focused -- 72 files: every src/ghidra test, the Ghidra launcher, MCP and process boundary tests, and this PR's changed test files — 1,480 passed, 4 skipped. The two new regressions, a double-float session that reports a preparation and one that reports a calling-convention policy, each queried first with list_procedures, fail without the change.
  • npm run verify:ghidra:riscv (Linux container, Ghidra 12.1.4, Clang and LLD 19.1.7) — pass: six fixtures, "mocked": false, process lineage verified with no descendants, cleanup complete.

On 43a8517fd, which merges main d3ab0faf5 without text conflicts. main's load-image observation (#8531) and procedure search (#8520) revisions change every Ghidra profile digest, so both pinned profiles of images without an overlap moved. With this branch's own bridge and shared semantics, the merged tree keeps the earlier pins; the pins now hold main's own digests at d3ab0faf5, computed on main.

  • npm run check:fast and docs:check — pass. format:check fails only on 16 files that this PR does not change; they are byte-identical to main, whose own CI fails the same way.
  • npm run test:focused -- 17 files: the test files this PR changes and the Ghidra tests that main changed — 615 passed, 3 skipped.
  • npm run verify:ghidra:riscv (Linux container, Ghidra 12.1.4, Clang and LLD 19.1.7) — pass: six fixtures, "mocked": false, process lineage verified with no descendants, cleanup complete.
  • npm run verify:ghidra:bridge-sources (javac 21 against Ghidra 12.1.4's 195 jars) — pass.

On the previous head 78a413531:

  • npm run check:fast — pass.
  • npm run format:check — pass (6,619 files).
  • npm run docs:check — pass (9 tests).
  • npm run verify:module-boundaries — pass (3,508 source files).
  • npm run test:focused -- with 21 files: the four changed test files plus both binaryTarget suites, binaryTarget.riscv, binaryTarget.mips, binaryTarget.powerpc, both riscvIsaDeclaration suites, GhidraProvider.behavior, GhidraMipsProfile, ghidraBridgeSources, ghidraPspExtension, packageTarListing, packedBridgeVerification, liveBinaryTargetSummary, riscv-checks, readmeConsistency and GoBinaryService — 789 passed, 4 skipped.
  • npm run knip — one finding, an unused exported type DarwinHelperCache in src/process/DarwinHelperCache.ts. main 6e9ee97a6 reports the same finding, so it is not this PR's.
  • npm run verify:ghidra:riscv (Linux container, Ghidra 12.1.4, Clang and LLD 19.1.7) — pass: six fixtures, process lineage verified, cleanup complete.
  • node scripts/verify-ghidra-bridge-sources.mjs (javac 21 against Ghidra 12.1.4's 195 jars) — pass, including the new script.

Not run: npm run check:pr; verify:ghidra and the other real-Ghidra lanes (verify:ghidra fails on main today, #8172); macOS and Windows hosts.

Compatibility, safety, and release

  • Breaking changes. Previously valid calls that now fail or change meaning:
    • for affected images, a stored selector such as .flash.text:0x42008398 no longer resolves, because that overlay no longer exists;
    • their analysis profiles change, so their snapshots are not reused;
    • RISC-V images whose sections overlap executable code in any other way, such as ESP-IDF v5.5.2's C5 and C61 builds, are now refused.
  • Why compatibility cannot be kept:
    • an alias for the .flash.text overlay would keep two representations of the same code;
    • admitting the newly refused layouts would keep analyzing them with flash code in an overlay, as main does for the C5 build: app_main at .flash.text:0x42006430 (observed). By Ghidra's last-in rule, .ext_ram.dummy also displaces the C5 build's rodata (source-derived).
  • Migration:
    • replace stored .flash.text:0x… selectors with the ELF address (0x42008398), or with a function name verified in the new analysis;
    • rebuild snapshots of affected images;
    • comparisons with older analyses show every flash procedure as changed, so take a new baseline;
    • for a newly refused image, the refusal names the two sections. ESP-IDF's C5 and C61 builds have no policy yet.
    • llvm-objcopy --remove-section=.ext_ram.dummy app.elf app-no-ext-ram-dummy.elf makes a transformed copy with the prepared shape. REA admitted and prepared such a copy of the v5.5.2 C5 build; C61 copies were not tried. Evidence identifies the copy, not the original image (docs/ghidra-riscv.md).
    • a search_bytes call without bounds now covers flash code; before, it skipped the overlay (source-derived; docs/ghidra-provider.md).
  • Real Hopper/Ghidra, browser, or OS coverage: real Ghidra 12.1.4 on Linux: the RISC-V lane and the ESP-IDF images above.
  • Package or release metadata impact: one new packaged bridge source.
  • Security, privacy, process, or containment review: no new authority. The pre-script changes only the ephemeral Ghidra database, before analysis; the file bytes and their offsets do not change. It uses one internal Ghidra call, ProgramDB.moveAddressRange, verified against Ghidra 12.1.4.

Squash commit message. GitHub's default squash body would join this branch's commit messages, which describe intermediate steps. Please squash with this message instead:

fix(ghidra)!: load ESP-IDF flash code in Ghidra's default address space

ESP-IDF reserves the addresses of .flash.text with the NOBITS section
.flash_rodata_dummy, and Ghidra 12.1.4's ELF loader then puts the code
in an overlay that DWARF import, call-frame processing and default-space
references do not reach. The RISC-V lanes now commit that layout to the
analysis profile. Before analysis, a pre-script moves the code into the
default address space with its file bytes, symbols, functions, entry
points and references, and keeps the reservation as an uninitialized
overlay; REA checks the script's report against the profile. Every
other section overlap that involves executable code is refused.

BREAKING CHANGE: RISC-V images with this layout, such as ESP-IDF
ESP32-C6, H2 and H21 builds, report flash code at its ELF address
(0x42...) instead of .flash.text:0x42..., and their analysis profiles
change. RISC-V images with any other section overlap that involves
executable code are now refused with the overlapping sections named.
These include ESP-IDF v5.5.2 ESP32-C5 and C61 builds, whose second
reservation, .ext_ram.dummy, also covers .flash.text. See
docs/ghidra-riscv.md.

Effect on the merged ilp32 lane (#8229): its C6, H2 and H21 images now get DWARF facts for flash code, and their flash addresses and profiles change. Its C5 and C61 images are refused. Other ilp32 images keep their profiles.

Effect on #8401 (ilp32f): source-derived: its signature refusal check reads DWARF types, and with this change it can see flash functions on H4-style images. #8401 has to merge this branch; conflicts are expected in GhidraRiscvProfile.ts, the RISC-V docs and tests. Its H4 evidence should then be measured again.

Author readiness

Human owner: @Entrpi

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

ESP-IDF's linker script reserves the addresses of .flash.text with the
NOBITS section .flash_rodata_dummy. Ghidra 12.1.4's ELF loader gives the
default address space to the later section, so .flash.text lands in an
overlay that its DWARF import, call-frame processing and default-space
references do not reach. Flash functions lose their DWARF signatures,
locals and no-return flags, each gets a one-byte shadow function in the
reservation, and pseudocode runs past calls such as esp_restart().

The ELF reader now records allocated ranges that an executable section
shares with another section, through one section-table visitor that the
RISC-V ISA reader also uses. When a NOBITS section starts with one
executable section and covers it, the RISC-V lanes commit both sections
to the analysis profile. ReaGhidraPrepareElfLoad.java then gives the code
the default address space before auto-analysis and keeps the reservation
as an uninitialized overlay; bytes, file offsets, symbols, functions and
entry points are kept. The session reports the outcome, and any memory
map that differs from the committed plan is refused. Images without such
an overlap keep their analysis profiles byte for byte.

BREAKING CHANGE: RISC-V images with this layout, such as ESP-IDF
ESP32-C6, H2 and H21 builds, report flash code at its ELF address
(0x42...) instead of .flash.text:0x42..., and their analysis profiles
change. RISC-V images with any other section overlap that involves
executable code are now refused with the overlapping sections named.
These include ESP-IDF v5.5.2 ESP32-C5 and C61 builds, whose second
reservation, .ext_ram.dummy, also covers .flash.text. See
docs/ghidra-riscv.md.
Domain:
- Find section overlaps with one sort and sweep that stops at the second
  overlap involving executable code, and read the ISA and the overlap in
  one section-table pass.
- Record the two sections that cause a refusal as witnesses. The Ghidra
  lane owns the refusal text and names them with their ranges.

Ghidra lane:
- Check the preparation report before the first lazily started analysis
  call, as for the calling-convention report.
- The pre-analysis script removes the code's overlay entry points, checks
  that no symbol or reference remains in the old overlay range, and reports
  the counts of the result. Its report echoes the blocks it left, which the
  client compares with the committed sections. A failed move refuses and
  rethrows. Reservation labels mark no entry points.
- Session limitations no longer state a planned move after a canonical
  outcome.

Verification:
- The ESP fixture defines ESP-IDF's absolute text bounds and the
  reservation's label. The lane checks their addresses, the entry points,
  file-offset uniqueness and snapshot replay, a reservation-first build,
  a pre-analysis refusal through analyzeHeadless and a truncated
  double-float section table.
- Boundary tests cover the refusal witnesses, a 10,000-section table and
  an ELF64 table.
The move takes the references from the code's range but not the
references into it. Besides the entry points, Ghidra 12.1.4's loader
records one data reference from the .flash.text section header to the
code, which the new post-condition refused in the RISC-V lane. Retarget
plain memory references and entry points into the old range to the moved
code; any other kind stays behind and is refused.

Removing the reservation block keeps the references into its range, so
the pinned symbols' entry points need no re-add.
Ghidra 12.1.4 places a section-relative symbol at the section's end, such
as ESP-IDF's _etext, in the code's overlay space one byte past the block.
The move covers the block only, so the label stayed in the overlay and
ended up in the reservation. Everything in the code's overlay space now
follows the code: references are retargeted, labels outside the block are
created again in the default space, and the post-condition checks the
whole space. The script refuses when another block shares that space.

The ESP fixture defines _etext, and the lane checks its address.
The client compared the pre-script's echoed block names with the ELF
section names, although section names are display only: an unnamed
section's empty name never matches the name Ghidra gives its block, so
that preparation was refused. It also accepted any reported address
spaces. The check now compares addresses, sizes and the code's file
offset, and requires moved code to go from an overlay into the
program's default space while its reservation leaves it, and canonical
code to stay there. An unnamed section is labelled by its index.

The docs and the pre-script now state that a failure during the move
may be refused without its reason, and that an entry mark on a
reservation label stays at its default-space address. The docs give
the llvm-objcopy route for the refused C5 and C61 builds. The lane
checks that no other block maps any of the code's file bytes, and the
overlap tests add the C5 and C61 section shape.
The overlap check kept one record per allocated, non-empty section, and
only the file size limited their number. NOBITS headers take no file
bytes, so a crafted 1 GB ELF32 table holds about 27 million of them.
The check now keeps at most 65,280 (SHN_LORESERVE, from which ELF
counts sections through section zero) and reports the overlap as
unavailable beyond that, which the RISC-V lanes refuse as they refuse
a table that cannot be read.
@greptile-apps

greptile-apps Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The reviewed changes appear safe to merge, with both numbered previous findings addressed and no new blocking issue found.

Summary

This PR puts supported ESP-IDF flash code in Ghidra’s default address space before analysis. It commits the preparation to the analysis profile and checks the reported result before serving queries.

  • The RISC-V lane checks executable section overlaps before admitting an image.
  • Supported flash procedures use their ELF addresses in Ghidra’s default space.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Read RISC-V ELF sections] --> B{Overlap layout}
  B -->|No code overlap| C[Normal import]
  B -->|One supported reservation| D[Commit preparation to profile]
  B -->|Unsupported or unreadable| E[Refuse target]
  D --> F[Prepare memory before analysis]
  F --> G{Preparation result}
  G -->|Refused| H[Disable analysis and reject session]
  G -->|Moved or already canonical| I[Analyze program]
  I --> J[Check report against profile]
  J --> K[Serve queries]
Loading

Reviews (8) · Last reviewed commit: "fix(ghidra): give the ESP-IDF load prepa..." · Reviewed by Greptile

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-11T03:39:47.073751Z fed6719 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 78a4135311

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Entrpi added a commit to Entrpi/rea that referenced this pull request Nov 1, 2026
# Conflicts:
#	bridge/ghidra/ReaGhidraBridge.java
#	docs/ghidra-riscv.md
#	scripts/verify-real-ghidra-riscv.mjs
#	src/ghidra/GhidraLauncher.ts
#	src/ghidra/GhidraProviderClient.ts
#	src/ghidra/GhidraRiscvProfile.test.ts
#	src/ghidra/GhidraRiscvProfile.ts
#	src/ghidra/GhidraSessionValues.test.ts
#	src/ghidra/GhidraSessionValues.ts
#	tests/boundary/providers/ghidra/ghidraLauncher.test.ts
#	tests/conformance/ghidra/riscv-fixture.mjs
Entrpi added a commit to Entrpi/rea that referenced this pull request Nov 1, 2026
Change 2 (fix-ghidra-computed-noreturn-calls) revises no_return_repair,
which changes every Ghidra profile. Tests that pin main's digests now
pin this branch's: morluto#7951's raw digests and its pre-morluto#7823 snapshot
comparison, which also sets aside no_return_repair, and Q10's digests
outside the M-profile ELF lane. The overlay merge (cb7e88e) did the
same for morluto#8464's RISC-V pins. Whichever of these lands upstream after
Change 2 takes the same change.
Entrpi added a commit to Entrpi/rea that referenced this pull request Nov 1, 2026
main's load-image observation (morluto#8531) and procedure search (morluto#8520)
revisions change every Ghidra profile, so this branch's pins move again:
morluto#7951's raw digests and its pre-morluto#7823 snapshot comparison, which now
also sets aside the two revisions, Q10's digests outside the M-profile
ELF lane, and morluto#8464's RISC-V images without an overlap. With this
branch's earlier bridge and shared semantics, every earlier pin holds,
so the move is main's alone. The legacy raw comment now says that these
pins include Change 2's revision, as Q10's does.
main's load-image observation (morluto#8531) and procedure search (morluto#8520)
revisions change every Ghidra profile digest, so the profiles of images
without an overlap move with main. With this branch's own bridge and
shared semantics, the merged tree keeps the earlier pins, so the move is
main's alone. Both pins take main's own digests at d3ab0fa, which the
merged branch matches.
Entrpi added a commit to Entrpi/rea that referenced this pull request Nov 1, 2026
A RISC-V session that committed neither a calling-convention policy nor
a load preparation started its bridge lazily on the first query other
than `health`. An `elf_load_preparation` or calling-convention report
that the profile did not commit was then never checked, and the query
returned its result. Every RISC-V session now starts through the
checked handshake, so such a report closes the session before any
analysis call, whatever the first query is. A double-float session that
reports a preparation, and one that reports a calling-convention
policy, are now tested with `list_procedures`; both fail without the
fix.

Admission, profile parameters and profile digests do not change. If
Ghidra exits during the first query of a double-float session, the
error now tells the caller to reopen the session, as other checked
sessions do.
A refused memory map only wrote the refusal, so headless Ghidra still
ran seeds, auto-analysis and post-script analysis on a program that REA
then discarded. The preparation now turns off headless auto-analysis when
it refuses; later pre-scripts and the bridge still run, so the session
reports the reason.

The swap also renamed the reservation's overlay space to the section's
raw name, so a reservation named like another address space, such as
the RISC-V default space ram, or containing characters Ghidra rejects,
failed after the move. The space now takes the name Ghidra's memory map
derives for an overlay block: ':' and whitespace become '_', and a name
in use gains .1, .2 and so on.

verify:ghidra:riscv drops -noanalysis from the refusal run and checks
that Ghidra analyzed nothing, and a seventh fixture names the
reservation ram and checks that it moves into the overlay space ram.1.
main's morluto#8630 starts every Ghidra query's session and checks its
handshake first, which supersedes this branch's eager start of every
RISC-V session. Take main's start and drop the branch's riscv term. The
condition main kept selects recovery guidance only; it now also counts
a committed load preparation, a startup input like the calling-convention
policy. A double-float image can have the first without the second.
Entrpi added a commit to Entrpi/rea that referenced this pull request Nov 4, 2026
# Conflicts:
#	docs/ghidra-riscv.md
#	src/ghidra/GhidraProviderClient.ts
main's byte-search (morluto#8840), data-type (morluto#8602) and reference revisions change every
Ghidra profile. With those revisions reversed, the old pins hold.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 054016b5c9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

GhidraProviderClient.ts keeps the branch's isDeepStrictEqual import and
main's GHIDRA_LITERAL_CANDIDATES_LIMITATION import.

Main's profile revisions since 8eff651 change the RISC-V profile
digests. The no-overlap pins take the digests main computes at
5553acf for the same ilp32 and double64 images, which this branch
still reproduces.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3111486211

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/ghidra/GhidraRiscvProfile.test.ts Resolved
Comment thread src/ghidra/GhidraRiscvProfile.ts Resolved
Entrpi added a commit to Entrpi/rea that referenced this pull request Nov 8, 2026
# Conflicts:
#	src/ghidra/GhidraProviderClient.ts
#	src/ghidra/GhidraRiscvProfile.test.ts
Static checks failed on main 5553acf's own src/cli/coreNativeCommands.ts
formatting, which main has since fixed.
The overlap refusal, the load-layout limitation and the bridge comments
named docs/ghidra-riscv.md, which main moved to docs/providers/.
The refusal, report-check and handshake tests compared whole messages.
They now check the sections, coordinates, spaces and counts each message
must state, and the handshake compares the session's limitations with
ghidraRiscvLoadPreparationLimitations. Rewording the messages passes;
dropping a count fails.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5099cc9e2d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

The preparation created its temporary blocks as <name>.rea, so a disjoint
section already named that refused an otherwise admitted image. It now
takes the first unused <name>.rea[.N]. verify:ghidra:riscv gains an
esp-edge-names fixture with such a section.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38f32b55b4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@CassieFox23

Copy link
Copy Markdown
Contributor

TL;DR: I ran the 4 changed test files from this PR on three Node versions, and each run gave 200 passed and 3 skipped. CI on the head failed one test in a file this PR does not change.

Findings

I ran head 38f32b5 in a VM. The PR changes 30 files against the merge base: 4 test files and 26 others. The 4 test files are src/ghidra/GhidraRiscvProfile.test.ts, src/ghidra/GhidraSessionValues.test.ts, tests/boundary/filesystem/elfSectionOverlap.test.ts and tests/boundary/providers/ghidra/ghidraLauncher.test.ts.

Check Node 22.23.4 Node 24.22.1 Node 26.12.1
Test Files 4 passed (4) 4 passed (4) 4 passed (4)
Tests 200 passed, 3 skipped (203) 200 passed, 3 skipped (203) 200 passed, 3 skipped (203)

The head is 5 commits behind main 99d00d1.

CI on the head, which I read live with gh: mergeable is true, state is blocked, and the PR is not a draft.

  • Workflow runs: CI is completed/failure. CodeQL production is completed/skipped. PR readiness is completed/success. PR title is completed/success.
  • Check runs failed (3): CI required, Portable tests and Test shard (7/8).
  • Check runs cancelled (2): Test shard (1/8) and Test shard (3/8).
  • Check runs skipped: 20.
  • Check runs success (21), among them Source tests, Static checks, Build package and five Test shard runs.
  • Commit status: PR readiness is success.

The Test shard (7/8) and Portable tests logs show one failure, preserves the source when an inline result with unknown retention exceeds delivery. It is in tests/boundary/mcp/sourcePublicationComposed.test.ts, which this PR does not change. The CI required job fails with Error: CI lanes did not pass: test, test-shard.

Not run

  • A run of npm run verify:ghidra:riscv at 38f32b5, with the wrong-file-offset plan sent through REA's full script chain. I would check the log for ANALYZING, with a normal ESP run as positive control.
  • A merge with current main, 99d00d1.
Test setup
  • Box: Fedora 44 VM (4 vCPUs, 8 GB) on my Nobara 44 PC, Linux 7.2.11
  • Node: 22.23.4, 24.22.1, 26.12.1
  • Head / base: 38f32b5 / c78c8ae (merge base with main)
  • Install: npm ci --offline and npm run test:prepare on Node 24.22.1
Commands
npm ci --offline
npm run test:prepare
xvfb-run -a npm run test:focused -- src/ghidra/GhidraRiscvProfile.test.ts src/ghidra/GhidraSessionValues.test.ts tests/boundary/filesystem/elfSectionOverlap.test.ts tests/boundary/providers/ghidra/ghidraLauncher.test.ts

🤖 Written and tested with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

The code's temporary block could take the removed reservation's name,
such as a reservation named <code>.rea, and the reservation's rename
back then collided. The picker now skips both final names.
verify:ghidra:riscv's esp-edge-names fixture names its reservation
.flash.text.rea beside a disjoint .flash.text.rea.1.
Entrpi added a commit to Entrpi/rea that referenced this pull request Nov 9, 2026
# Conflicts:
#	src/ghidra/GhidraRiscvProfile.test.ts
…o#8340

main's morluto#8340 bumps call_relationship_coverage from v5 to v7, which changes
every Ghidra profile digest. With the revision reverted in place, the old
pins hold (79 of 79).
Entrpi added a commit to Entrpi/rea that referenced this pull request Nov 9, 2026
# Conflicts:
#	src/ghidra/GhidraRiscvProfile.test.ts
…dgets

main's morluto#9706 adds native_value_flow_projection to the shared analysis
semantics, which changes the profiles of images without a section overlap.
With that one revision removed, the previous pins hold (79 of 79).
Entrpi added a commit to Entrpi/rea that referenced this pull request Nov 11, 2026
# Conflicts:
#	src/ghidra/GhidraRiscvProfile.test.ts

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Ghidra RISC-V lanes lose DWARF and no-return facts for ESP-IDF flash code

2 participants