105 lines
11 KiB
Markdown
105 lines
11 KiB
Markdown
# Regression matrix — exe-reverse
|
|
|
|
The measured record behind `tests/`: one row per dimension, with the target, the scripts it
|
|
exercised, the result this host produced, and how strong that result is.
|
|
|
|
## When to load this
|
|
|
|
Read it when you need to know what was actually run rather than what the suite covers, or when a
|
|
coverage claim needs a target and a number behind it. `tests/README.md` describes the suite;
|
|
this file records its output.
|
|
|
|
## How a row is filled in
|
|
|
|
Every row below was produced by running the command in it on this host, except where the row says
|
|
otherwise in the result column. Rows that were not run say so and are not filled with an estimate.
|
|
Strength labels follow the repository convention: `observed` means the command and its output
|
|
exist, `inferred` means the result is reasoned from an observation, `unverified` means it is
|
|
reported but was not reproduced here.
|
|
|
|
Targets are public and legally usable: the generated fixture, and platform binaries that ship with
|
|
the operating system. No third-party product is a target, and no sample is stored in the
|
|
repository — the images are built at test time into a temporary directory.
|
|
|
|
Environment: Windows 11 build 26100, Python 3.14.0, pytest 8.4.1.
|
|
|
|
## The matrix
|
|
|
|
| Dimension | Target | Scripts exercised | Measured result | Strength |
|
|
| --- | --- | --- | --- | --- |
|
|
| PE32+ header surface | generated fixture, `default` recipe | `peutil.py` | 4096 bytes, 5 sections, magic 0x20B, entry RVA 0x1000, image base 0x140000000, declared checksum 0x10bad, zero warnings | observed |
|
|
| PE32 header surface | generated fixture, `pe32` recipe | `peutil.py` | magic 0x10B, machine 0x014C, image base 0x400000, same five sections | observed |
|
|
| Optional-header layout difference | both fixtures | `peutil.py` | number-of-directories lives at optional+92 (PE32) or optional+108 (PE32+), directories at +96 or +112; reading the other layout yields 40 and 0 respectively, neither being the real count 16 | observed |
|
|
| RVA to file offset, both directions | generated fixture | `peutil.py` | 0x1010 to 0x410 and back; header RVAs identity below SizeOfHeaders 0x400; an RVA in no section maps to nothing | observed |
|
|
| Virtual-only tail | generated fixture | `peutil.py` | `.rdata` has 1408 virtual and 512 raw bytes, so 896 bytes have no file image; RVA 0x2200 resolves to nothing while 0x21FF resolves to 0x9FF | observed |
|
|
| PE checksum | four generated recipes | `peutil.py` | declared equals computed equals a second, independent implementation of the published algorithm | observed |
|
|
| PE checksum on real images | platform binaries under `%SystemRoot%\System32` | `peutil.py` | `notepad.exe` (360448 bytes) declared 0x5e673 equals computed; `kernel32.dll` and `ntdll.dll` parse with matching checksums | observed |
|
|
| Imports | generated fixture | `peutil.py` | one DLL with one named import (hint 7) and one import by ordinal 0x2A, descriptor terminator respected, no delay imports | observed |
|
|
| Exports, resources, TLS, debug, relocations | generated fixture | `peutil.py` | one named export; the resource tree is exactly VERSION / 1 / 1033 with a 4-byte leaf; TLS callback RVA 0x1000; RSDS record with the fixture's PDB path and age 1; one relocation block with two entries | observed |
|
|
| Pattern locating, three verdicts | generated fixture | `aob_scan.py` | the planted pair reports 2 hits as `ambiguous` and exits 1; the PDB string reports 1 hit as `unique` and exits 0; an absent pattern reports 0 as `no-match` and exits 1 | observed |
|
|
| Verdict boundaries | generated fixture | `aob_scan.py` | 0 to `no-match`, 1 to `unique`, 2 through 7 to `ambiguous`, 8 and above to `pervasive` | observed |
|
|
| Equal-length patch and verification | generated fixture | `pepatch.py`, `peutil.py` | one patch at 0x410 pinned with `--expect`; the written file re-reads with no verification problems and a recomputed checksum of 0x10c24; exactly 3 bytes differ from the input, the patched byte and 2 checksum bytes | observed |
|
|
| Ambiguity refusal | generated fixture | `pepatch.py` | exit 1, `matched 2 sites`, no output file created, input hash unchanged | observed |
|
|
| Expect-mismatch refusal | generated fixture | `pepatch.py` | exit 1 with `the site moved or the wrong binary is open` | observed |
|
|
| Length discipline | generated fixture | `pepatch.py` | no `--expect` and no `--original-length` reports `[length unproven: pass expect or original-length]`; a 1-into-3 byte patch is refused without `--allow-resize`; with it, the run is marked structural and the file length is unchanged because the write is in place | observed |
|
|
| Dry run | generated fixture | `pepatch.py` | exit 0, no file created, input hash unchanged | observed |
|
|
| Control run, no-op patch | generated fixture | `pepatch.py` | output SHA-256 equals input SHA-256: the file is byte-identical, checksum included | observed |
|
|
| Cross-build comparison | two independently built fixtures | `aob_scan.py` | same recipe 1 versus 1, exit 0; `bare` recipe 1 versus 0, exit 1; patched build 2 versus 1, exit 1; a pattern with 2 hits on both sides still exits 1 because the count agreeing is not the same as the pattern being usable | observed |
|
|
| Identity scan, rules | synthetic corpus in a temporary root | `scan_leaks.py` | all 18 rules fire on their shaped samples; the corpus reports 13 strong findings and 4 weak, exit 1 | observed |
|
|
| Identity scan, exemptions | same corpus | `scan_leaks.py` | tool, protector, format, platform, placeholder, documentation-address and well-known-SID samples produce no finding; 19 hits suppressed | observed |
|
|
| Identity scan, clean surface | temporary root with no identity in it | `scan_leaks.py` | `RESULT=clean`, exit 0 | observed |
|
|
| Identity scan, this suite's own surface | `tests/` | `scan_leaks.py` | `RESULT=clean` with 0 strong, 0 weak and 19 suppressed hits | observed |
|
|
| Identity scan, the skill's own default surface | `skills/` (the root the script resolves with no arguments) | `scan_leaks.py` | 1 strong finding — a standard platform name read as a product identity bound to a version — with 18 suppressed, exit 1 | observed |
|
|
| Host capability probe | this machine | `hwprobe.py` | 8 routes, 3 of them `available`, exit 0; 2 rows answer with a sentence instead of one of the three documented states | observed |
|
|
| Suite runtime | this machine | `pytest` | 269 tests in 23.2 s: unit 1.5 s, cli 19.7 s, integration 4.6 s | observed |
|
|
| Reader throughput | 360448-byte platform image | `peutil.py` | `--summary` took 159 / 162 / 157 ms over three runs | observed |
|
|
| Feature-probe cost | this machine | `hwprobe.py` | `--json` took 1834 / 1821 ms over two runs | observed |
|
|
| Fixture reproducibility | generated fixture | `make_fixture_pe.py` | the same recipe produces the same SHA-256 and the same bytes on repeated runs; the `no-virtual-tail` recipe exits 1 and writes nothing | observed |
|
|
| Truncated-image handling | fixture cut off mid-header | `peutil.py`, `pepatch.py` | both print a traceback and exit non-zero instead of using their documented exit codes; a fully non-PE input does use them | observed |
|
|
| Missing spec file | `pepatch.py` with an absent `--spec` | `pepatch.py` | traceback rather than the documented usage error | observed |
|
|
| Bare JSON pattern array | `aob_scan.py` with a top-level array | `aob_scan.py` | `AttributeError`; only the object form is accepted | observed |
|
|
| Documented empty-surface exit code | `scan_leaks.py` over an empty directory | `scan_leaks.py` | exit 0 with `RESULT=clean`; the documented exit 3 has no code path | observed |
|
|
| Patched image launch | a copy of a platform binary patched at the entry point | `pepatch.py` | not run by this suite. Recorded from the project's measurement pass: one byte changed 48 to CC, exactly two bytes differed from the original (the patched byte and the checksum field), and the process terminated with exit code 0x80000003 about 4.0 s after launch while the unpatched copy showed no surviving process | unverified here |
|
|
| Device, debugger, network and driver routes | none | none | not run. This suite has no device, no debugger, no network and no kernel component; those routes are conditional host facts and are reported by `hwprobe.py` instead | unknown |
|
|
|
|
## Measurements obtained in this run
|
|
|
|
Reader throughput on a 360448-byte platform image, `peutil.py --summary`, three consecutive runs:
|
|
159 ms, 162 ms, 157 ms. Host capability probe, `hwprobe.py --json`, two runs: 1834 ms and 1821 ms.
|
|
Both sit within a few percent of the values recorded during the project's earlier measurement pass
|
|
(158/166/167 ms and 1858 ms), which is the expected spread for process startup on a busy desktop.
|
|
|
|
Suite runtime, `python -m pytest -q` from the repository root: 269 passed in 23.2 s. The cli layer
|
|
dominates because it starts a fresh interpreter per assertion, twice over for the feature probe.
|
|
|
|
Exact byte-level effect of a two-byte patch at file offset 0x410: three bytes differ from the
|
|
input, at 0x410, 0xD8 and 0xD9. The checksum field is four bytes wide, so a patch that changes two
|
|
bytes of image data can leave part of the old checksum value already correct.
|
|
|
|
## Rows not run
|
|
|
|
No row in the matrix covers a debugger attach, a hardware breakpoint, a trace capture, a driver
|
|
load, a network endpoint, a device, or a live target process. That is the suite's stated boundary,
|
|
not an oversight: none of those can be asserted deterministically on a bare checkout, and a row
|
|
that claims them would be describing a different machine on a different day.
|
|
|
|
Two further dimensions were considered and left out. Mass-scanning a large directory of real
|
|
images was left out because the time it takes depends on the machine's disk rather than on the
|
|
scripts. Comparing two builds of the same third-party product was left out because no such pair is
|
|
publicly usable here; the two-build comparison in the matrix uses two images this repository
|
|
builds itself.
|
|
|
|
## Notes for a reader of the results
|
|
|
|
The one strong finding on the skill's own default surface is also the only failing repository gate
|
|
at the time of writing: the tracked-surface scan reads the standard platform name `TPM 2.0` in a
|
|
shipped reference as a product bound to a version. Nothing in `tests/` contributes to it — this
|
|
suite's own surface reports 0 strong and 0 weak.
|
|
|
|
The scripts under `skills/exe-reverse/scripts/` were being edited while this matrix was being
|
|
produced, and two of its rows moved with them: the public-address rule gained an exclusion that
|
|
stops it reading a four-part version number as an address, and a standard-name exemption that had
|
|
suppressed every URL finding was corrected. Both the suite and this file record the state after
|
|
those corrections. Re-run the matrix after any further change to those scripts rather than
|
|
trusting these numbers.
|