Matching MSC 5.1 output instruction-for-instruction on a 16-bit DOS reconstruction

[xor2003]

Sharing some hard-won findings from reconstructing a ~900KB overlaid 16-bit DOS game executable built with Microsoft C 5.1/6.0-era tools — back into readable C that recompiles to instruction-equivalent code. Currently ~600 of 753 routines match gold instruction-for-instruction, whole modules byte-identical modulo relocations.

The loop

Ghidra headless decompile → clean up into idiomatic C → compile with the period MSC (CL.EXE under a DOS VM) → instruction-level diff against retail function extents → iterate until match. Binary equivalence is the proof the C is right — a routine isn't done until readable source produces it.

The hard part: MSC 5.1 codegen is compiler-state-driven, not source-local

The single biggest surprise: whether cmp A, B or x + y materializes operand A or B into ax is decided by module-global symbol-table interning state, not just the source text. The same source line emits both forms in the same module at different parse states. Levers that flip it, in rough order of usefulness:

• Function-local extern decls. extern int foo; inside a function's declaration block perturbs the interner without emitting any data — and can flip a compare's operand pick, change a register-resident loop variable, or shorten a block enough to cross a short-branch boundary. We used them to turn a push [bp-2] into gold's push ax (saving 2 bytes and pulling a jz back inside ±127 range).
• Declaration position. Where a local sits inside the int a,b,c; decl list drives its stack slot and whether MSC's allocator promotes it to a register. Moving int cnt from first-decl to mid-decl flipped a loop test from mov ax,[cnt] to gold's mov ax,[NPLANES]. Moving a register int later in the decl block killed its di promotion entirely (gold kept it in memory — decl order beats the register keyword).
• Symbol flooding / period-16 seesaw. File-scope filler decls shift the intern table for everything downstream — observed modulo-16 cycles in which operand wins. Per-function windows align at different counts (one routine wants k≡0 mod 16, another k≈6), so we decouple functions using decls placed between them plus function-local externs inside each.
• Signedness and operator class. ==/!= and relational compares pick differently; unsigned-vs-signed operands flip which lands in ax.
• Name hashing. Local slot buckets track ~sum(uppercased identifier bytes) % 16; decompiler-guessed names (cnt, endval, di2) intern differently than the originals did, so the same source compiles differently until decl context is restored.
• Commutative ops still pick sides. For x + y a param operand beats a global into ax almost unconditionally — but gold sometimes shows the global first, meaning the original expression shape differed.
• register keyword. MSC 5.1 honors it as a hint and auto-promotes; declaration order beats the keyword.

Tooling traps worth knowing

• /Os vs /Ot is per-module and visible: shared vs inlined epilogues, and /Ot emits nop pads before odd branch targets.
• Stale extent maps produce phantom diffs — a verifier decoding via an outdated overlay map reported instructions (add al,[0x619a]) that don't exist anywhere in the file. Always spot-check against the raw image before trusting a mismatch.
• MSC 5.1 warns function too large for global optimizations past a symbol threshold — your padding decls can push a routine over it and change codegen everywhere.
• A failed link can still leave a newer executable on disk; verify timestamps before trusting any diff.

Happy to answer questions or compare notes with anyone else doing byte-exact MSC/16-bit reconstruction — the "compiler-state archaeology" layer turned out to be most of the work.

[xor2003]

Following up on the toolkit question — the tools are public:

• xor2003/mzretools — MZ-executable inspection/diff tools. The workhorse is mzdiff: instruction-level routine comparison that normalizes segment bases, call targets, relocations and dseg offsets, so a rebuilt routine is compared against the retail extent structurally rather than byte-by-byte in the wrong coordinate space. Also mzmap (routine extent maps from linker MAP output), mzdup (cross-binary routine matching — how we find that two differently-named routines in different segments are the same original function), mzhdr.
• xor2003/kvikdos — a very fast headless DOS emulator (Linux KVM). We run the period toolchain under it: CL.EXE (MSC 6.00), LINK.EXE 5.31, MASM.EXE 5.10. It mounts host directories as DOS drives, so a full compile+link cycle of a 7000-line module takes seconds inside the normal Linux build loop — no DOSBox interaction needed.

How it's used

# compile one module with the same flags the original used
CL.EXE /FPa /G2 /Gs /AM /Osegcl /c /FcMODULE.LST MODULE.C     # under kvikdos
# relink the image (LINK 5.31, original object order + overlays)
LINK /MAP /SEG:256 @objects.lrf
# single-routine verdict vs the retail extent
mzdiff --loose --nocall --map gold_fn.map --tmap my.map GOLD.EXE:_fn MY.EXE:_fn

Typical iteration: mzdiff shows the first mismatching instruction (push [bp-2] vs gold's push ax), you figure out which source-level construct produces gold's form, edit the C, recompile, re-diff. The /Fc listing (interleaved source+asm) is invaluable for seeing exactly what MSC did with each line.

The non-obvious part documented in the post above: at this point the remaining diffs are almost never "wrong decompilation" — they're compiler-state differences (symbol interning order, declaration placement, register-residency) that change instruction selection on otherwise-correct C. That's the layer we haven't seen written up anywhere else, and it's what makes byte-exact MSC reconstruction slow.