zdebel

zdebel / DOS_RECONSTRUCTION_PLAYBOOK.md

Last active 2 weeks ago

Like 0

Revision cccf7a5b1e7d18231c969a6babd992614aea053f

DOS_RECONSTRUCTION_PLAYBOOK.md Raw

Universal DOS Game Reconstruction Playbook

A Game-Agnostic Engineering Guide to Bit-Exact (1:1) Clean-Room Reverse Engineering, Differential CPU Harnessing, and Modern Recompilation


1. Core Philosophy: Bit-Exact Clean-Room Re-engineering

Porting a legacy DOS title to modern architectures (Python, C++, Rust, WebAssembly) typically falls into one of two paradigms:

  1. Approximate Recreation ("By Eye"): Recreating physics, AI, and rules by playing the game in DOSBox or reading high-level disassembly.
    • The Problem: Subpixel physics, collision corner cases, fixed-point integer rounding, and subtle AI timers inevitably diverge. The resulting game feels different and fails historical preservation standards.
  2. Differential Re-engineering (Dual-Execution Parity): Executing the original 16-bit x86 binary alongside the newly reconstructed modern engine, feeding identical inputs, and asserting that the state of both systems is bit-for-bit identical on every single simulation tick.
flowchart TD
    ROM["Original DOS Executable (.EXE / .COM)"] --> DISASM["Static Triage (Ghidra / IDA / objdump)"]
    ROM --> UNICORN["Differential CPU Sandbox (Unicorn x86 Engine)"]
    DISASM --> CODE["Reconstructed Engine (Python / C++ / Rust)"]
    UNICORN --> HARNESS["Differential Assertion Harness"]
    CODE --> HARNESS
    HARNESS -- "Memory, Registers & State Match?" --> PASS["Bit-Exact Parity Confirmed"]
    HARNESS -- "Mismatch at Tick N" --> DEBUG["Automated Register / Memory Delta Report"]
    DEBUG --> CODE

Important

The Golden Rule of 1:1 Parity: Do not guess formulas. If an algorithm's behavior is ambiguous, isolate its entry point in an x86 CPU emulator, feed it controlled register/memory states, and verify that the re-implemented function reproduces the exact register and memory mutations across all value domains.


2. Universal Tooling Stack

A complete clean-room reconstruction requires four decoupled layers:

2.1. Static Analysis & Binary Triage

  • Disassemblers: objdump / ndisasm for rapid machine code inspection (-m i8086 -M intel).
  • Interactive Decompilers: Ghidra or IDA Pro for mapping call graphs, data segment references (DGROUP), indirect jump tables (switch/case), and runtime compiler signatures (e.g., Borland C++, Microsoft C, Watcom C).
  • Binary Container Unpackers: Custom scripts for parsing DOS executable headers (MZ / LE / Linear Executable / COM), segment relocation tables, and legacy compression schemes (LZSS, RLE, Huffman, PKWARE DCL).

2.2. Execution Sandbox & CPU Emulation

  • CPU Emulation Core: Unicorn Engine (lightweight x86 16-bit real mode and 32-bit protected mode emulator based on QEMU).
  • Interception Framework: CPU hooks (UC_HOOK_CODE, UC_HOOK_INTR) for capturing hardware port I/O (IN/OUT), BIOS/DOS software interrupts (INT 10h, INT 21h, INT 09h), and external driver far-calls.

2.3. Audio Synthesis & Hardware Emulation

  • FM Synthesis Core: Nuked-OPL3 (or Nuked-OPL2) compiled as a shared library (.so / .dll). Translates direct register writes to ports 0x388/0x389 (AdLib / Sound Blaster) into a 16-bit stereo PCM stream (44,100 Hz).
  • PCM Sample Engine: Direct Sound Blaster DSP register emulation (0x220..0x22C) for digitized 8-bit mono audio samples.
  • Low-Latency Streaming: Modern SDL2 audio queue (SDL_QueueAudio) or native OS audio backends.

2.4. Presentation & Windowing

  • Modern surface rendering (SDL2 / Pygame / WebGL) with integer scaling, 4:3 aspect-ratio correction, and virtual framebuffers decoupling internal simulation resolution (e.g., 320x200 / 320x240) from host display resolutions.

3. The Differential Execution Harness Architecture

The differential harness executes the original DOS machine code inside an emulated memory space. Below is the game-agnostic architecture for a 16-bit DOS real-mode sandbox.

3.1. Real-Mode Memory Layout

In x86 16-bit real mode, physical memory addresses are calculated as: $$\text{Physical Address} = (\text{Segment} \times 16) + \text{Offset}$$

A standard 2 MB virtual address space accommodates the entire legacy environment:

0x000000 ┌──────────────────────────────────────────────┐
         │ Interrupt Vector Table (IVT, 256 vectors * 4B)│
0x000400 ├──────────────────────────────────────────────┤
         │ BIOS Data Area (BDA, 256 bytes)              │
0x000500 ├──────────────────────────────────────────────┤
         │ DOS Conventional RAM Free Pool               │
0x01DB00 ├──────────────────────────────────────────────┤
         │ Application Program Image (CS, Code Segments)│
0x05C090 ├──────────────────────────────────────────────┤
         │ Application Data Segment (DS / DGROUP)       │
0x090000 ├──────────────────────────────────────────────┤
         │ Stack Segment (SS:SP = 0x9000:0xFFFE)        │
0x0A0000 ├──────────────────────────────────────────────┤
         │ Video RAM (VRAM - Mode 13h / Mode X / Planar)│
0x0FFFFF └──────────────────────────────────────────────┘

3.2. Reusable Unicorn Real-Mode Harness (Python Template)

import struct
from unicorn import Uc, UC_ARCH_X86, UC_MODE_16, UC_HOOK_CODE, UC_HOOK_INTR
from unicorn.x86_const import (
    UC_X86_REG_CS, UC_X86_REG_IP, UC_X86_REG_DS, UC_X86_REG_ES,
    UC_X86_REG_SS, UC_X86_REG_SP, UC_X86_REG_BP,
    UC_X86_REG_AX, UC_X86_REG_BX, UC_X86_REG_CX, UC_X86_REG_DX,
    UC_X86_REG_SI, UC_X86_REG_DI
)

class RealModeHarness:
    """Game-agnostic execution sandbox for 16-bit DOS binaries."""

    def __init__(self, memory_mb: int = 2):
        self.mem_size = memory_mb * 1024 * 1024
        self.uc = Uc(UC_ARCH_X86, UC_MODE_16)
        self.uc.mem_map(0, self.mem_size)

        # Setup standard stack at top of conventional memory
        self.ss = 0x9000
        self.sp = 0xFFFE
        self.uc.reg_write(UC_X86_REG_SS, self.ss)
        self.uc.reg_write(UC_X86_REG_SP, self.sp)

        # Intercept hooks
        self.io_reads = {}
        self.io_writes = {}
        self.external_calls = {}
        self.uc.hook_add(UC_HOOK_CODE, self._on_code)

    def load_image(self, segment: int, data: bytes):
        """Loads executable code or data at the specified segment."""
        phys_addr = segment * 16
        self.uc.mem_write(phys_addr, data)

    def stub_far_call(self, target_seg: int, target_off: int, handler_fn=None):
        """Intercepts far calls to external libraries/drivers and simulates RETF."""
        phys_addr = target_seg * 16 + target_off
        self.external_calls[phys_addr] = handler_fn

    def _on_code(self, uc, address, size, user_data):
        # 1. Handle stubbed library calls (far calls)
        if address in self.external_calls:
            handler = self.external_calls[address]
            if handler:
                handler(self)
            # Read far return address from stack and simulate RETF
            sp = uc.reg_read(UC_X86_REG_SP)
            ss = uc.reg_read(UC_X86_REG_SS)
            ret_ip, ret_cs = struct.unpack_from("<HH", uc.mem_read(ss * 16 + sp, 4))
            uc.reg_write(UC_X86_REG_SP, sp + 4)
            uc.reg_write(UC_X86_REG_CS, ret_cs)
            uc.reg_write(UC_X86_REG_IP, ret_ip)
            return

    def call_procedure(self, seg: int, off: int, args: list = None, near: bool = True):
        """Executes a target procedure in the binary and returns upon completion."""
        # Setup sentinel return address on stack
        sp = self.uc.reg_read(UC_X86_REG_SP)
        sentinel_ip = 0xFFF0
        sentinel_cs = 0x9000
        sentinel_phys = sentinel_cs * 16 + sentinel_ip

        # Place a halt instruction (HLT = 0xF4) at the sentinel address
        self.uc.mem_write(sentinel_phys, b"\xF4")

        # Push arguments (cdecl / pascal calling convention)
        if args:
            for arg in reversed(args):
                sp -= 2
                self.uc.mem_write(self.ss * 16 + sp, struct.pack("<H", arg & 0xFFFF))

        # Push return address
        if not near:
            sp -= 2
            self.uc.mem_write(self.ss * 16 + sp, struct.pack("<H", sentinel_cs))
        sp -= 2
        self.uc.mem_write(self.ss * 16 + sp, struct.pack("<H", sentinel_ip))
        self.uc.reg_write(UC_X86_REG_SP, sp)

        # Set execution pointers
        self.uc.reg_write(UC_X86_REG_CS, seg)
        self.uc.reg_write(UC_X86_REG_IP, off)

        # Run until sentinel HLT
        start_phys = seg * 16 + off
        self.uc.emu_start(start_phys, sentinel_phys)
        return self.uc.reg_read(UC_X86_REG_AX)

4. Universal 5-Phase Reconstruction Pipeline

graph LR
    P1["Phase 1: Binary & Asset Triage"] --> P2["Phase 2: Physics Core Isolation"]
    P2 --> P3["Phase 3: Entity Machines & AI"]
    P3 --> P4["Phase 4: Audio Synthesis & Timers"]
    P4 --> P5["Phase 5: Lifecycle & Frontend"]

Phase 1: Binary & Asset Triage

  1. Identify the Compiler & Runtime (CRT):
    • Inspect the entry point (CS:IP from MZ header). Identify Borland C0.ASM, Microsoft CRT, or custom assembly loaders. Borland C++ binaries typically locate DGROUP at a fixed segment following code segments.
  2. Resource Extraction:
    • Dump static palette tables (VGA 6-bit DAC values 0..63). Convert via (val * 255) // 63 or val << 2.
    • Unpack graphics buffers. Distinguish between packed pixel formats (Mode 13h linear 320x200) and planar bitplane formats (Mode X unchained 4-plane memory at 0xA000).
    • Extract level descriptor arrays and tile lookup indices.

Phase 2: Physics Core Isolation

  1. Map the Fixed-Point Physics Loop:
    • DOS physics calculations almost universally use fixed-point arithmetic instead of floating-point units (which were rare on 286/386 systems). Typical formats: $1/16$ px (4 fractional bits), $1/32$ px (5 fractional bits), or $16.16$ fixed-point.
  2. Isolate the Primary Entity Step:
    • Trace the physics routine (inputs -> velocity integration -> tile collision mask -> snap/bounce).
    • Wire the procedure into the differential harness.
    • Run Monte Carlo fuzzing: generate 50,000 randomized input sequences (joypad + terrain configurations) and assert that Python positions match x86 positions on every tick.

Phase 3: Entity Dispatch & Actor State Machines

  1. Decode the Entity Table Structure:
    • DOS games typically manage entities via an array of uniform structures in DGROUP (e.g., 32 to 64 bytes per actor).
    • Locate the handler jump table (call word ptr [bx + state_offset]).
  2. Translate Jump Tables into Reconstructed State Machines:
    • Convert indirect assembly function pointer tables into explicit state enumerations:
      class EntityState(Enum):
          INIT = 0x08
          PATROL = 0x0A
          ALERT = 0x0E
          ATTACK = 0x12
          STUNNED = 0x16
          DEAD = 0x1E
      
  3. Differential Actor Verification:
    • Step each actor type independently across its entire state lifecycle, verifying sprite index, velocity delta, and hit-box boundaries.

Phase 4: Audio Driver & Hardware Timer Parity

  1. Deconstruct the Low-Level Sound Engine:
    • DOS titles typically use an AdLib/Sound Blaster driver executing via timer interrupt (INT 08h).
    • Identify whether the driver reprograms the Intel 8253/8254 PIT (Programmable Interval Timer) channel 0.
  2. The PIT Timing Trap:
    • The standard BIOS timer ticks at 18.2 Hz (divisor 0).
    • Game music drivers often reprogram PIT Channel 0 to divisors like 6211 ($\approx 192.1\text{ Hz}$) or 1193 ($\approx 1000\text{ Hz}$).
    • The Solution: Do not bind music playback to video frame rate. Use a dedicated sub-accumulator ticking the audio engine at the exact hardware PIT frequency.
class PitAudioAccumulator:
    """Ensures audio driver ticks at authentic PIT timer frequency."""
    def __init__(self, pit_frequency_hz: float = 192.107):
        self.pit_dt = 1.0 / pit_frequency_hz
        self.accumulator = 0.0

    def advance(self, real_dt: float, sound_driver_tick_fn):
        self.accumulator += real_dt
        while self.accumulator >= self.pit_dt:
            sound_driver_tick_fn()
            self.accumulator -= self.pit_dt

Phase 5: Lifecycle & Frontend Integration

  1. State Flow Mapping:
    • Trace the global finite state machine: $$\text{Intro Cutscene} \longrightarrow \text{Title Menu} \longrightarrow \text{Level Loading} \longrightarrow \text{Game Loop} \longrightarrow \text{Level Transition} \longrightarrow \text{Game Over / Victory}$$
  2. Binary Data Persistence:
    • Map save-game structures and high-score files.
    • If the DOS game XOR-encrypts or checksums save files (e.g., XOR 0xFF on high-score bytes), re-implement the exact byte manipulation so existing legacy save files load seamlessly.

5. Architectural Pitfalls & Root-Cause Analysis

These 5 failure modes represent the most frequent sources of discrepancies in DOS-to-modern ports:

Warning

Pitfall 1: Procedure Coverage vs. Integration Lifecycle Blind Spots

A test harness verifying that 100% of individual assembly procedures produce identical math when called in isolation does not prove that the game loop connects them correctly.

  • Failure Example: An entity collision function works identically, but the main game loop fails to invoke the sound trigger event or resets a multiplier on the wrong screen transition.
  • Mitigation: Always complement procedural unit tests with long-running, multi-level Scenario Tests (End-to-End) that simulate real player death, level completions, and menu exits.

Caution

Pitfall 2: Silent Audio Loss via Emulator Stubs

When constructing a CPU differential harness, external driver far-calls (such as play_sfx or start_music) are routinely stubbed out with RETF to prevent the CPU emulator from getting bogged down in low-level sound card hardware registers.

  • Failure Example: The CPU emulator succeeds because the stub returns cleanly, but the developer forgets to wire that specific trigger into the modern audio queue. The game runs with missing sound effects.
  • Mitigation: Build a static audit listing every single call site to the audio entry points in the original binary disassembly. Assert that each call site corresponds to an event dispatched in the modern engine.

Important

Pitfall 3: Level Transition Audio State Leaks

In DOS games, the main loop often re-arms background music at the beginning of a level reload.

  • Failure Example: A modern port starts background music in start_game(), but when the player loses a life and restarts the level, the level-card display silences the audio, and no routine restarts the track.
  • Mitigation: Track the entry-point address of restart loops in disassembly. If the DOS assembly jumps to a location that calls start_music(TRACK_ID) before re-entering the main loop, ensure the modern loop mirrors that exact call.

Note

Pitfall 4: 16-Bit Integer Wrapping and Subpixel Boundary Inaccuracies

In 16-bit x86 code, registers wrap at 65,536 (0xFFFF), and signed comparisons (JL, JLE) behave differently from unsigned comparisons (JB, JBE).

  • Failure Example: Translating CMP AX, BX / JLE target to high-level code as if ax < bx: instead of if ax <= bx: causes boundary conditions (e.g., surface water line, platform edge) to trigger one pixel early or late.
  • Mitigation: Always enforce explicit 16-bit masking (val & 0xFFFF) on accumulator registers and translate conditional jumps strictly according to x86 flag definitions (ZF, SF, OF, CF).

Tip

Pitfall 5: Display Refresh Rate Bottlenecks (70 Hz vs. 35 Hz)

Standard VGA Mode 13h and Mode X refresh at 70.086 Hz ($25.175\text{ MHz} / (800\text{ clocks} \times 449\text{ lines})$). The DOS main loop polls VBLANK (IN AL, 0x3DA / TEST AL, 8).

  • The Issue: On original 286/386 hardware, complex planar rendering often exceeded 14.2 milliseconds per frame. As a result, the game regularly missed the vertical retrace and dropped to 35 FPS (every second VBLANK).
  • On a modern machine running the code at an unconstrained 70.086 Hz, the game will feel significantly faster than players remember from childhood hardware.
  • Mitigation: Keep internal game simulation strictly deterministic at the native rate (70.086 Hz) using a fixed-timestep clock, but provide a configurable timing mode or half-rate throttle for players seeking period-accurate 386 performance.

6. Deterministic Fixed-Timestep Clock Template

Modern host operating systems run with variable refresh rates, multitasking jitter, and sleep latency. The game loop must decouple wall-clock time from internal simulation ticks:

import time

class FixedTimestepClock:
    """Deterministic game loop clock modeled on DOS hardware VBLANK timing."""

    def __init__(self, target_hz: float = 70.0866, max_ticks_per_frame: int = 4):
        self.tick_ns = round(1e9 / target_hz)
        self.max_ticks = max_ticks_per_frame
        self.accumulator_ns = 0
        self.total_ticks = 0

    def advance(self, dt_seconds: float) -> int:
        """Accumulates elapsed wall-clock time and returns logic ticks to execute."""
        dt_ns = max(0, round(dt_seconds * 1e9))
        self.accumulator_ns += dt_ns

        # Calculate pending whole ticks
        ticks = self.accumulator_ns // self.tick_ns

        if ticks > self.max_ticks:
            # Drop backlog to prevent spiral of death during backgrounding / debug pause
            ticks = self.max_ticks
            self.accumulator_ns = 0
        else:
            self.accumulator_ns -= ticks * self.tick_ns

        self.total_ticks += ticks
        return ticks

    def reset(self):
        """Call upon unpausing or level reload to prevent time bursts."""
        self.accumulator_ns = 0

7. Comprehensive End-to-End Verification Checklist

Before certifying a DOS reconstruction as 1:1 Bit-Exact Complete, verify all 8 test suites:

1. Long-Run Physics Determinism

  • Run the differential harness over $\ge 50,000$ continuous ticks with randomized and scripted inputs.
  • Assert $0$ mismatches across position ($X, Y$), velocity ($VX, VY$), and collision states.

2. Rare State Machine Edge-Cases

  • Verify actor knockback, hitstun, and recovery states.
  • Verify entity destruction, rubble/shrapnel countdown timers, and despawn cycles.
  • Verify environmental submergence, buoyancy physics, and out-of-bounds checks.

3. Complete Audio Call-Site Parity

  • Conduct a binary-wide search for all call play_sfx instruction sites.
  • Construct automated integration tests ensuring every single audio call site dispatches its associated SFX event during gameplay.

4. Audio Driver Parity & FM Synthesis

  • Capture raw OPL register streams (0x388/0x389) from the x86 binary and assert identical byte output from the modern driver.
  • Verify PIT timer accumulation ($192\text{ Hz}$ or equivalent) to prevent tempo dragging.

5. Session Lifecycle & Input Abort

  • Test player death down to 0 lives: verify game over transition, delay timers, and failure audio.
  • Test mid-game input abort (e.g., ESC key): verify clean score carry-over and life zeroing without process termination.
  • Test pause toggle: verify audio continuity during pause and zero physics jumps on unpause.

6. Persistence & Serialization

  • Save and load high-score tables and game save files.
  • Assert identical byte-for-byte outputs against legacy DOS disk formats (including XOR masks and padding).

7. Game Completion (Outro & Victory)

  • Simulate completion of the final level.
  • Verify that victory cutscenes execute their complete animation script and play the authentic ending music track.

8. Cooperative & Difficulty Modes

  • Verify multi-player modes: test split input handling, shared life pools, and player-to-player collision boxes.
  • Verify difficulty modifiers: test crash thresholds, fuel/energy drain rate multipliers, and score bonuses.

8. Summary

A true 1:1 reconstruction does not rely on approximations. By establishing an x86 CPU differential harness, cataloging every procedure, intercepting hardware registers, and enforcing strict dual-execution assertions, legacy DOS titles can be preserved in modern software stacks with absolute mathematical fidelity.