Fix the budget gate and add CI, lint and coverage guards
- check_budget.py escalated a hard index-row violation to 'warn' on both branches, so INDEX_ROW_HARD could never fail anything. It fails now, and tests/unit/test_budget_measurement.py pins both thresholds. - The corpus note counted scripts/__pycache__/*.pyc, so the same commit reported 27 scripts after the suite had run and 16 on a clean checkout. - TOKENS_PER_CHAR is re-derived from a measurement (26,795 chars -> 6,065 tokens = 4.418); the comment's old figure and its arithmetic were both wrong for this file. - check_repo.py's yaml sentinel carries the type ignore, so mypy is clean with PyYAML installed as well as without it. - The CPUID test now skips with the probe's own note on a host without CPUID instead of failing, and pytest.ini points at tests/README.md, which exists, instead of an EXTENSION file that never did. Tooling: .flake8 (a measured 120-column limit), .gitattributes, .gitignore and .github/workflows/ci.yml (the five gates, the suite on 3.11-3.14 and a lint job), which is what CONTRIBUTING.md already claimed CI runs. The pull-request template it refers to now exists. Content: pitfalls P26-P34 -- the analysis-level failures the catalogue was missing (sampling read as observation, a self-built parser trusted, a verification that could not have failed, a shell-mangled command line) -- registered in the symptom index and its routing mirror. FINDINGS F11-F15 record the defects above. Tests: the coverage guard derives its subject from scripts/ rather than a hand-written list, pe_triage.py and pe_diff.py gain their first tests, and plan.py / preflight.py are declared uncovered with reasons.
This commit is contained in:
root
@@ -0,0 +1,79 @@
|
||||
<!--
|
||||
Fill this in. Three sections are read by a reviewer and one is not yours to fill.
|
||||
|
||||
An unchecked box is not evidence of absence: a pull-request template with a blank effect-surface
|
||||
checklist has not been shown to be documentation-only, it has been shown to be unfinished
|
||||
(CONTRIBUTING.md, section Review). The readiness declaration is left for a human owner; an agent
|
||||
leaves it blank rather than writing a claim it cannot make.
|
||||
-->
|
||||
|
||||
## What this changes
|
||||
|
||||
One paragraph. What changes, and what a reader can do afterwards that they could not before. If the
|
||||
change is documentation only, say so here and again in the effect surface.
|
||||
|
||||
## Evidence
|
||||
|
||||
Every command below was run in this checkout. Paste the output that decides the question; a summary
|
||||
of confidence is not a result, and `should pass` is not a run.
|
||||
|
||||
```
|
||||
python check_repo.py
|
||||
python check_refs.py
|
||||
python check_budget.py
|
||||
python check_routing.py
|
||||
python check_commands.py
|
||||
python -m pytest -q
|
||||
mypy *.py skills/exe-reverse/scripts/*.py
|
||||
flake8 *.py skills/exe-reverse/scripts/*.py
|
||||
```
|
||||
|
||||
Output that matters:
|
||||
|
||||
```
|
||||
<paste the decisive lines, and say what they decide>
|
||||
```
|
||||
|
||||
Claims added or changed:
|
||||
|
||||
| Claim | Grade | What it rests on (command + result) |
|
||||
|---|---|---|
|
||||
| | observed / inferred / unverified | |
|
||||
|
||||
- [ ] No claim is graded above the weakest step in its chain.
|
||||
- [ ] Every `observed` has the command and the output beside it.
|
||||
- [ ] Every negative result names the range, the pattern and what was searched. A search that
|
||||
returned nothing is evidence about the search.
|
||||
- [ ] Every limit cites the measurement or the constraint that set it, rather than a preference.
|
||||
|
||||
## Effect surface
|
||||
|
||||
Check each line that applies. `Documentation only` is a claim, so make it explicitly rather than
|
||||
leaving the list blank.
|
||||
|
||||
- [ ] `SKILL.md` — the budget gate output is attached above; it is loaded in full on activation.
|
||||
- [ ] `references/` — a new or renamed file is registered in `references/routing.md`, reachable from
|
||||
`SKILL.md` or the README, and carries a navigable head if it passes 100 lines.
|
||||
- [ ] `references/pitfalls.md` — a new entry takes the next number and keeps the same shape.
|
||||
- [ ] `scripts/` — a new script is in the routing script table, answers `--help` on a bare
|
||||
interpreter, and its argument table matches every command the docs tell a reader to run.
|
||||
- [ ] `tests/` — a new script is added to `conftest.SCRIPT_FILES` or declared uncovered with a
|
||||
reason in `tests/cli/test_cli_coverage.py`; a fixed defect has a test that fails without it.
|
||||
- [ ] `evidence/` and `docs/tool-verification/` — a capability claim moved in both, and the
|
||||
measurement record quotes the command it rests on.
|
||||
- [ ] `README.md` and `README.zh-CN.md` — the two stay in step.
|
||||
- [ ] `docs/tool-verification/FINDINGS.md` — a defect found by the repository's own tools is
|
||||
recorded with its symptom, root cause, how it surfaced, and the check that now catches it.
|
||||
- [ ] Documentation only — no behaviour changed, and the gates above still pass.
|
||||
|
||||
## Readiness declaration (human owner only)
|
||||
|
||||
An agent does not fill this section in. Leave it blank; the reviewer completes it.
|
||||
|
||||
- [ ] The change belongs in this repository rather than in a fork or a downstream project.
|
||||
- [ ] The boundary of what was not exercised is stated, not implied: every route that needs a
|
||||
debugger, a driver, a kernel component, a hypervisor, test signing or a second machine is
|
||||
named as not run here.
|
||||
- [ ] The serialized evidence is committed, and nothing in it carries target identity or a
|
||||
machine-specific path.
|
||||
- [ ] The commit is ready to be fast-forwarded onto `main`.
|
||||
@@ -0,0 +1,125 @@
|
||||
# CI for the exe-reverse skill repository.
|
||||
#
|
||||
# WHAT THIS RUNS
|
||||
# * the repository's own structural gates: check_repo.py, check_refs.py, check_budget.py,
|
||||
# check_routing.py, and check_commands.py
|
||||
# * the pytest suite: tests/unit, tests/cli, tests/integration
|
||||
# * lint: mypy and flake8 over the gate scripts and the shipped skill scripts
|
||||
#
|
||||
# WHAT THIS DOES NOT RUN, ON PURPOSE
|
||||
# * anything needing a debugger, a driver, a kernel component, a live target process, a hypervisor,
|
||||
# test signing or a network peer. Those routes are recorded experiments in tests/benchmark.md,
|
||||
# and what the suite deliberately does not cover is stated in tests/README.md.
|
||||
# * anything about agent behaviour. Whether a model routes, classifies or refuses correctly is not
|
||||
# observable from a unit test; skills/exe-reverse/evals/evals.json is the skeleton for that, and
|
||||
# it has never been executed.
|
||||
# * the hardware-observation routes. They need particular silicon and privilege; hwprobe.py
|
||||
# reports what a runner lacks and the suite skips with that reason instead of failing.
|
||||
#
|
||||
# The runner is Linux while the skill targets Windows. That is deliberate: the suite observes the
|
||||
# scripts' behaviour -- parsing, patching, gate logic, exit codes -- and every check that needs a
|
||||
# Windows fact asks the probe or skips. A green run here is a statement about the *code*, never
|
||||
# about a route that needs a Windows host.
|
||||
#
|
||||
# Coverage is therefore: static structure, documentation anchors, budget, documented commands,
|
||||
# routing coherence, pure parsers, CLI contracts, fixture construction, typing/style lint and
|
||||
# no-debugger flows. Nothing more is claimed.
|
||||
name: ci
|
||||
|
||||
on:
|
||||
push:
|
||||
pull_request:
|
||||
#
|
||||
# NOTE ON TRIGGERS: no `paths:` filter, deliberately. A filter that watches only `*.py` and the lint
|
||||
# configuration would skip the structural gates on exactly the changes they exist to catch -- a
|
||||
# reference index or a documented command edited without touching a script. check_refs.py and
|
||||
# check_routing.py are documentation gates, so they have to run on documentation commits.
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
test:
|
||||
name: python ${{ matrix.python-version }}
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
python-version: ["3.11", "3.12", "3.13", "3.14"]
|
||||
|
||||
env:
|
||||
PYTHONIOENCODING: utf-8
|
||||
PYTHONDONTWRITEBYTECODE: "1"
|
||||
# The gates spawn sibling scripts and decode their output. On a Windows runner the default
|
||||
# codec is the console's (GBK here), and one script's --help text is not GBK-decodable, which
|
||||
# is reported as a crash in that script rather than as a harness decoding problem. Pinning
|
||||
# UTF-8 removes the whole class of failure on every platform.
|
||||
PYTHONUTF8: "1"
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
|
||||
- uses: actions/setup-python@v7
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
|
||||
- name: Install test dependencies
|
||||
# `tiktoken` is optional for the suite but not for the budget gate's *measurement*: without
|
||||
# it, check_budget.py falls back to characters/constant, and the two paths must agree.
|
||||
# Installing it here means CI exercises the real tokenizer path, and the unit tests
|
||||
# comparing both paths run instead of skipping.
|
||||
# Not upgrading pip itself: it is not a goal here, and a pip self-upgrade failing would be
|
||||
# reported as this repository's regression.
|
||||
run: python -m pip install --upgrade pytest tiktoken
|
||||
|
||||
- name: Structural gate - repository layout
|
||||
run: python check_repo.py
|
||||
|
||||
- name: Structural gate - documentation anchors
|
||||
run: python check_refs.py
|
||||
|
||||
- name: Structural gate - context budget
|
||||
run: python check_budget.py
|
||||
|
||||
- name: Structural gate - documented commands
|
||||
run: python check_commands.py
|
||||
|
||||
- name: Structural gate - routing inventory matches the entry point
|
||||
run: python check_routing.py
|
||||
|
||||
- name: Regression suite
|
||||
run: python -m pytest -q
|
||||
|
||||
- name: Summarise skipped coverage
|
||||
if: always()
|
||||
run: python -m pytest -q -ra --no-header -p no:cacheprovider 2>&1 | tail -n 40
|
||||
|
||||
lint:
|
||||
# Static checks over the same tracked surface the local commands cover, so what passes locally
|
||||
# cannot drift from what CI enforces. Both tools read their configuration (mypy.ini, .flake8)
|
||||
# from the repository root automatically.
|
||||
#
|
||||
# The environment is deliberately bare -- only mypy and flake8, none of PyYAML, capstone,
|
||||
# tiktoken or pyelftools. That is the poorest environment the configuration must keep working
|
||||
# in, which is what mypy.ini's ignore_missing_imports overrides exist for: CI exercises the
|
||||
# fallbacks instead of a machine-local install hiding a regression.
|
||||
#
|
||||
# One interpreter, not the matrix: the checks are static, and the matrix's job is runtime
|
||||
# behaviour, which lint does not observe.
|
||||
name: lint (mypy + flake8)
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
|
||||
- uses: actions/setup-python@v7
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install lint dependencies
|
||||
run: python -m pip install --upgrade mypy flake8
|
||||
|
||||
- name: mypy
|
||||
run: mypy *.py skills/exe-reverse/scripts/*.py
|
||||
|
||||
- name: flake8
|
||||
run: flake8 *.py skills/exe-reverse/scripts/*.py
|
||||
Reference in New Issue
Block a user