feat(collect): 按本次构建生成 collect 的工具链 + 回环回归测试

tools/tdc_collectgen.py: 静态导出槽位序 -> 按 #36 规则算「项->槽」-> 重排 -> 该构建密钥加密。
同一构建实测: 37 项 -> 38 槽, 明文 686B / collect 920 字符, 解密回环逐字节一致。

规则修正(重要):
  * kind=1 的串只按**括号外**逗号切槽 —— '[]'/'[360,360]'/'["zh-CN","zh"]' 内逗号不构成新槽
  * kind=2 的项**仍产出 1 槽**(r3 清空转存 r4, r4 在收尾 @4905 拼回)

新增 tcaptcha/tests/test_collectgen.py (已并入 selftest) 与 fixture:
  outer_commas / 槽数推算 == 真机槽数 / 重排是置换 / 加解密回环 / 不匹配必须报错。
账本新增已知 #37(工具链跑通) 与 #38(纯离线生成尚未打通, 诚实标注)。
This commit is contained in:
yml2213
2026-10-12 00:45:23 +08:00
parent 6460dd78d6
commit 6aaf6bc290
5 changed files with 328 additions and 1 deletions
+3 -1
View File
@@ -66,7 +66,9 @@
| 33 | **VM 程序结构层已吃透:顶层就是 62 个模块工厂的注册表** | 顶层 `@0` 做 `registry[id] = function(module, exports, require){...}`(`DUP; PUSH_IMM [id]; REVERSE_TOP2; JUMP -> CREATE_CLOSURE; SET_PROP`)。`tools/tdc_program.py --modules` 直接导出。修了两个真 bug:① 反汇编器漏了 `RET_TRUE`/`THROW` 是终结符(控制流穿过函数结尾 → 子程序归属全错);② 提升器终结符判定写死 opcode 数值 | 确证 |
| 34 | **`cd` 槽位序的来源已定位并可静态导出** | 装配链:清单模块 `arr[arr.length] = require(N)`(**追加,所以 require 顺序就是数组顺序**,已核)→ 分发模块按 `.on`/`.reset`/`.get` 分三数组 → 装配模块 `r3='{"cd":['` + `for(...) r3 += arr[i].get()`。线上构建实测 **37 个 require + 1 个字面量槽 = 38**,与真机 `cd` 槽数完全吻合;两份构建的 require 串不同(与 #31 一致)。工具:`tools/tdc_layout.py` | 确证(顺序)/ 语义标签部分 |
| 35 | **运行时反证通过:静态导出的 require 序 == 运行时真实 require 序(逐项一致)** | 在真机 iframe(`tdc.js` 实际执行的上下文,`window.TDC.getData` 就在这里)里包一层 `Function.prototype.apply`,模块工厂调用形状 = **4 参 `[object,object,object,function]`**(require 是最后一个),按工厂实例分组记账。触发一次刷新后取运行时串,用 `tools/cdp_script_source.js`(Debugger.scriptParsed + getScriptSource)取到**运行时真正执行的那份 tdc.js**(exactly-once,不能用重新 fetch 代替——#29 每次返回不同构建),两边串:<br>静态 = 运行时 = `[8,9,10,13,14,15,16,17,18,19,20,21,23,24,25,30,31,32,34,36,37,1,38,39,40,41,42,43,44,47,48,49,50,51,56,57,58]`(37 项,逐位相同)<br>→ 由此可判定:之前"部分槽语义对不上"是**标注启发式**的问题,**顺序本身正确**。工具:`scripts/experiment_require_order.py` | 确证 |
| 36 | **「清单项 → cd 槽」的精确规则(同一构建端到端校验确立)** | 装配循环(线上构建 `@4362`,370 条)分支树已读通:<br>`if(!item) continue;` `v,w,kind = item.get();`<br>· **`kind===2`(`@4820`)→ 不进 `cd`**:它把 `r3` 清空(`r3=""`)转存到 `r4`(`@4860`),`r4` 在收尾 `@4905` 被二次加工后另作他用(如随机串 `vXc29RxhvT34Q…`)<br>· `v===null/undefined`(`@4799`)→ `out += "null"`<br>· `typeof v==="number"`(`@4738`)→ `out += (v||0)`<br>· **`kind===1`(`@4724`)→ `out += v` 原样拼**:返回值是 JSON 碎片串,串里的逗号会**切出额外槽**(实测 `"641547489,1791732001"` → 2 槽)<br>· 其余(`@4703`)→ `out += '"' + v + '"'`<br>· 收尾 `@4753`:`typeof w === "number"` 才再 +1 槽(实测 kind=2 的第二元素是 null,不触发;此路径本次未命中)<br>→ **37 项 → 38 槽**,仅第 33 项(模块 1)占 2 槽。工具:`tools/tdc_slotmap.py`(`--items` 运行时取样 + `--plain` 明文对拍) | 确证 |
| 36 | **「清单项 → cd 槽」的精确规则(同一构建端到端校验确立)** | 装配循环(线上构建 `@4362`,370 条)分支树已读通:<br>`if(!item) continue;` `v,w,kind = item.get();`<br>· **`kind===2`(`@4820`)**:把 `r3` 清空转存 `r4`(`@4860`),`r4` 在收尾 `@4905` 二次加工后**拼回**,因此**仍产出 1 槽**(实测第 0 项,随机串 `vXc29RxhvT34Q…`)<br>· `v===null/undefined`(`@4799`)→ `out += "null"`<br>· `typeof v==="number"`(`@4738`)→ `out += (v||0)`<br>· **`kind===1`(`@4724`)→ `out += v` 原样拼**:返回值是 JSON 碎片串;**只按括号外的逗号**切槽 —— `'[]'`/`'[360,360]'`/`'["zh-CN","zh"]'` 内的逗号不构成新槽,而 `"641547489,1791732001"` → 2 槽<br>· 其余(`@4703`)→ `out += '"' + v + '"'`<br>· 收尾 `@4753`:`typeof w === "number"` 才再 +1 槽(实测 kind=2 的第二元素是 null,不触发;此路径本次未命中)<br>→ **37 项 → 38 槽**,仅第 33 项(模块 1)占 2 槽。工具:`tools/tdc_slotmap.py`(`--items` 运行时取样 + `--plain` 明文对拍) | 确证 |
| 37 | **按构建生成 collect 的工具链已跑通(含回环自测)** | `tools/tdc_collectgen.py`:静态导出槽位序 → 按 #36 规则算「项→槽」→ 重排 → 该构建密钥加密。<br>实测(同一构建):37 项 → **38 槽**,明文 686B / collect 920 字符;**解密回环逐字节一致**,cd 与真机样本逐槽相同。<br>规则修正(重要):`kind=1` 的串**只按括号外逗号**切槽 —— `'[]'`/`'[360,360]'`/`'["zh-CN","zh"]'` 内的逗号不构成新槽;`kind=2` 的项**仍产出 1 槽**(转存 r4 后在收尾 @4905 拼回)。<br>回归:`tcaptcha/tests/test_collectgen.py`(已并入 selftest) | 确证 |
| 38 | **纯离线生成还没打通(诚实标注)** | 当前 `--from-sample` 依赖**一份同机的真机明文**做重排;`--values`(语义名→值)需要调用方提供取值字典。**仍缺**:纯算地为每个语义名算出值(多数是静态环境量可配置,但 canvas/WebGL 指纹只能取自真实环境)。这一步完成前,不能声称"纯离线过风控"。 | 状态说明 |
---
+1
View File
@@ -16,6 +16,7 @@ SUITES = [
"tcaptcha.tests.test_glyph_solver",
"tcaptcha.tests.test_icon_click_solver",
"tcaptcha.tests.test_slider_solver",
"tcaptcha.tests.test_collectgen",
"tcaptcha.tests.test_xtea_key",
"tcaptcha.tests.test_tdc_key",
]
+1
View File
@@ -0,0 +1 @@
{"seq": [8, 9, 10, 15, 18, 19, 20, 21, 27, 28, 31, 33, 34, 36, 37, 38, 39, 40, 41, 26, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 55, 1, 56, 57, 58], "kinds": [2, 1, null, null, null, null, null, null, 1, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 1, null, null, null, null, null, 1, null, 1, null], "v0s": ["vXc29RxhvT34Qjg4BO81Rcg6kcS36LwS", "1", "1440", "0", "0", "0", "2560", "98k", "[]", "+08", "https://turing.captcha.gtimg.com/1/template/drag_ele.html?rand=1519713624347", "1440-2560-2530-24-*-*-|-*", "0", "MacIntel", "2", "https://1ms.run/console/personal/benefits/check-in", "1791736469", "", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36", "1984571394", "ANGLE (Apple, ANGLE Metal Renderer: Apple M4, Unspecified Version)", "iframe", "", "0", "1023", "Google Inc. (Apple)", "", "[360,360]", "1791736491", "24", "UTF-8", "10", "GgoAAAANSUhEUgAAASwAAACWCAYAAATf3v8bh+ryWcfSBSLkmN3N1PQinPL7Bi3TG0qE2CogAAAABJRU5ErkJggg==", "641547489,1791732001", "2", "[\"zh-CN\",\"zh\"]", "0"], "sample_cd": [1791736322, 2560, [[1, 1, 12]], 0, "1440-2560-2530-24-*-*-|-*", 1440, "iframe", 0, "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36", "ANGLE (Apple, ANGLE Metal Renderer: Apple M4, Unspecified Version)", "https://turing.captcha.gtimg.com/1/template/drag_ele.html?rand=1519713624347", "", 0, "+08", 1023, "", [360, 360], "UTF-8", "", 0, 2, 1791736260, 2086701610, 3, 1, "https://1ms.run/console/personal/benefits/check-in", "98k", [], "MacIntel", "Google Inc. (Apple)", "GgoAAAANSUhEUgAAASwAAACWCAYAAATf3v8bh+ryWcfSBSLkmN3N1PQinPL7Bi3TG0qE2CogAAAABJRU5ErkJggg==", 0, 10, 641547489, 1791732001, 24, ["zh-CN", "zh"], 0]}
+105
View File
@@ -0,0 +1,105 @@
# -*- coding: utf-8 -*-
"""
按构建生成 collect 的回归测试(承 PROGRESS.md #34/#35/#36;AGENTS.md §5.9)。
固化的性质:
1. `outer_commas`: 只数**括号外**的逗号 —— 方括号内的逗号不构成顶层分隔
('[]' / '[360,360]' / '["zh-CN","zh"]' 拼回去仍是一个值);
2. `SlotSpec` 推算出的槽数 == 真机明文 cd 的实际槽数;
3. 重排是**置换**:值集合不变(排序后逐项相同);
4. 加密/解密**回环一致**(明文逐字节还原);
5. 槽数不匹配的样本**必须报错**,不能静默错位。
样本: tcaptcha/tests/fixtures/tdc_slotmap_case.json
= {seq, kinds, v0s, sample_cd},取自 2026-10-11 真机会话(37 项 -> 38 槽)。
运行:
.venv/bin/python -m tcaptcha.tests.test_collectgen
"""
from __future__ import annotations
import base64
import json
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
if str(ROOT / "tools") not in sys.path:
sys.path.insert(0, str(ROOT / "tools"))
from tcaptcha.crypto import xtea_decrypt_bytes # noqa: E402
import tdc_collectgen as cg # noqa: E402
FIX = ROOT / "tcaptcha" / "tests" / "fixtures" / "tdc_slotmap_case.json"
# 固定密钥,仅用于回环自洽(不代表任何真实构建)
KEY = [1497711441, 1213874789, 1799903316, 1835232858]
_fails: list = []
def check(name: str, cond: bool, detail: str = "") -> None:
print(f" [{'PASS' if cond else 'FAIL'}] {name}" + (f" {detail}" if detail else ""))
if not cond:
_fails.append(name)
def main() -> int:
if not FIX.exists():
print(f"缺少 fixture: {FIX}")
return 1
case = json.loads(FIX.read_text(encoding="utf-8"))
seq, kinds, v0s, sample_cd = case["seq"], case["kinds"], case["v0s"], case["sample_cd"]
print(f"fixture: 项 {len(seq)} / kind {len(kinds)} / v0 {len(v0s)} / 真机 cd {len(sample_cd)}")
print("\n[1] outer_commas: 只数括号外的逗号")
check("'[]' -> 0", cg.outer_commas("[]") == 0)
check("'[360,360]' -> 0", cg.outer_commas("[360,360]") == 0)
check("'[\"zh-CN\",\"zh\"]' -> 0", cg.outer_commas('["zh-CN","zh"]') == 0)
check("'641547489,1791732001' -> 1", cg.outer_commas("641547489,1791732001") == 1)
check("'a,b,c' -> 2", cg.outer_commas("a,b,c") == 2)
print("\n[2] 槽数推算 == 真机明文槽数")
spec = cg.SlotSpec(seq, kinds, v0s)
print(f" {spec.describe()}")
check("槽数一致", spec.n_slots == len(sample_cd),
f"{spec.n_slots} vs {len(sample_cd)}")
check("fixture 覆盖 kind=1", any(k == 1 for k in kinds))
check("fixture 覆盖 kind=2", any(k == 2 for k in kinds))
print("\n[3] 重排是置换(值集合不变)")
out = cg.reorder(sample_cd, spec)
key = lambda v: json.dumps(v, sort_keys=True, ensure_ascii=False)
check("长度不变", len(out) == len(sample_cd), f"{len(out)} vs {len(sample_cd)}")
check("值集合相同(排序后逐项相等)",
sorted(map(key, out)) == sorted(map(key, sample_cd)))
print("\n[4] 加密/解密回环")
plain = cg.build_plain(out)
b64 = cg.encrypt_plain(plain, KEY)
back = xtea_decrypt_bytes(base64.b64decode(b64), KEY).decode("latin1").rstrip("\x00").strip()
check("明文逐字节还原", back == plain)
back_obj = json.loads(back)
check("cd 还原", back_obj["cd"] == out)
check("sd 还原", back_obj["sd"] == {"od": "C", "ft": "qf_7Pf__H"})
print("\n[5] 槽数不匹配必须报错")
raised = False
try:
cg.reorder(sample_cd[:-1], spec)
except ValueError:
raised = True
check("少一槽 -> ValueError", raised)
print(f"\n{'=' * 62}")
if _fails:
print(f"❌ collectgen 测试失败: {_fails}")
return 1
print("✅ collectgen 测试全部通过")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+218
View File
@@ -0,0 +1,218 @@
# -*- coding: utf-8 -*-
"""
tdc_collectgen.py —— **按本次构建**生成 collect 明文(并加密)。
承 har/1ms_dianxuan/PROGRESS.md #29~#36:
* #29/#31 槽位序**按构建随机化**,不能用缓存/旧模板;
* #34/#35 槽位序 = 清单模块的 require 顺序,**可纯静态导出**;
* #36 「清单项 → cd 槽」不是一一对应:`kind===2` 的项不进 cd,
`kind===1` 的返回值是 JSON 碎片串、串内逗号会切出额外槽。
本模块做两件事:
1. `reorder(template_cd, mapping, values)` —— 把"语义值"按**本次构建的槽位序**排成 cd;
2. `build(...)` —— 组装成 `{"cd":[...],"sd":{...}}` 明文,并用该构建的 XTEA 密钥加密。
⚠️ 关于「语义值」从哪来(诚实说明,别把它当成已经解决的事):
目前有两种取值来源,都可用,但**完备性不同**:
* `--values <json>`:由调用方提供「语义名 -> 值」字典(最稳,推荐);
* `--from-sample <cd json>`:用**另一份已解密的真机明文**做"同源重排"。
这要求两份样本跑在**同一台机器/同一环境**(取值集合相同,只有顺序不同,见 #31),
适用于"先拿一份真机明文当素材、再按本次构建重排"的验证场景。
**仍需补齐**:纯算地把每个语义名对应的值算出来(多数是静态环境量,可配置;
canvas/WebGL 指纹这类必须取自真实环境)。这一步未完成前,纯离线生成还不能保证过风控。
用法:
# 用一份同机真机明文做重排, 按本次构建输出 collect
.venv/bin/python tools/tdc_collectgen.py <tdc.js> \\
--from-sample /tmp/real_cd.json --out /tmp/collect.b64
"""
from __future__ import annotations
import argparse
import base64
import json
import sys
from pathlib import Path
from typing import Any, Dict, List, Optional, Sequence, Tuple
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
if str(ROOT / "tools") not in sys.path:
sys.path.insert(0, str(ROOT / "tools"))
from tcaptcha.crypto import xtea_encrypt_bytes # noqa: E402
from tcaptcha.tdc_key import extract_xtea_key_pure # noqa: E402
import tdc_layout # noqa: E402
import tdc_program as tp # noqa: E402
# --------------------------------------------------------------------------- #
# 槽位映射 #
# --------------------------------------------------------------------------- #
def outer_commas(s: str) -> int:
"""数**不在**任何 `[]` / `{}` 内的逗号个数(方括号内的逗号不构成顶层分隔)。"""
depth = 0
n = 0
for ch in s:
if ch in "[{":
depth += 1
elif ch in "]}":
depth = max(0, depth - 1)
elif ch == "," and depth == 0:
n += 1
return n
class SlotSpec:
"""一次构建下的槽位规格(规则见 PROGRESS.md #36)。"""
def __init__(self, seq: Sequence[int],
kinds: Optional[Sequence[Any]] = None,
v0s: Optional[Sequence[str]] = None) -> None:
self.seq = list(seq)
self.kinds = list(kinds) if kinds is not None else [None] * len(seq)
# v0s[i] = 该项 get()[0] 的字符串化结果;kind=1 时按它的逗号数切出额外槽
self.v0s = list(v0s) if v0s is not None else [""] * len(seq)
self.assign: List[Tuple[int, int, int, int]] = [] # (item, module, first_slot, n_slots)
self._build()
def _build(self) -> None:
slot = 0
for i, m in enumerate(self.seq):
k = self.kinds[i]
n = 1
if k == 1 and isinstance(self.v0s[i], str):
# 原样拼 -> 该串会被 JSON 解析器按逗号切成多个值。
# 但**数组/对象形态**的串('[]' / '[360,360]' / '["zh-CN","zh"]')里的逗号
# 处在方括号内, 拼回去仍是一个值, 不产生新槽 —— 只数"括号外"的逗号。
n += outer_commas(self.v0s[i])
if k == 2:
# kind=2 把 r3 清空转存 r4(不接在 cd 后面),但 r4 会在收尾 @4905
# 被二次加工后重新拼进 r3 —— 实测它**仍然产出一个槽**。
n = 1
self.assign.append((i, m, slot, n))
slot += n
@property
def n_slots(self) -> int:
return sum(a[3] for a in self.assign)
def describe(self) -> str:
multi = [(i, m, n) for (i, m, _s, n) in self.assign if n > 1]
return (f"项 {len(self.seq)} -> 槽 {self.n_slots} "
f"(kind=2 跳过 {sum(1 for k in self.kinds if k == 2)} 项"
+ (f"; 多槽项 {multi}" if multi else "") + ")")
def load_shape(items_path: str, seq: Sequence[int]) -> Tuple[List[Any], List[str]]:
"""从运行时取样(items json)里读出每项的 kind 与 get()[0] 的字符串化值。"""
d = json.load(open(items_path, encoding="utf-8"))
res = d.get("result", d)
rows = res.get("out") or res.get("items") or []
kinds: List[Any] = []
v0s: List[str] = []
for r in rows:
if not isinstance(r, dict):
kinds.append(None); v0s.append(""); continue
kinds.append(r.get("kind"))
v = r.get("v0")
v0s.append(str(v) if v is not None else "")
if len(kinds) < len(seq):
pad = len(seq) - len(kinds)
kinds += [None] * pad
v0s += [""] * pad
return kinds[: len(seq)], v0s[: len(seq)]
def load_sample_cd(path: str) -> List[Any]:
raw = json.load(open(path, encoding="utf-8"))
return raw["cd"] if isinstance(raw, dict) else raw
def reorder(sample: Sequence[Any], spec: SlotSpec) -> List[Any]:
"""把"上一份 cd"的值按**本次构建**的槽位序重排。
原理(#31):同一台机器上的多份样本,**取值集合相同**、只有顺序不同;
所以只要样本与本次构建的"槽位数一致",就能按"项 -> 槽"的对应关系做置换。
"""
if len(sample) != spec.n_slots:
raise ValueError(
f"样本槽数 {len(sample)} != 本次构建槽数 {spec.n_slots};"
f"两者必须来自同一环境(见 PROGRESS.md #31)")
out: List[Any] = []
for (_i, _m, slot, n) in spec.assign:
out.extend(sample[slot:slot + n])
return out
# --------------------------------------------------------------------------- #
# 明文组装 + 加密 #
# --------------------------------------------------------------------------- #
def build_plain(cd: Sequence[Any], sd: Optional[dict] = None) -> str:
sd = sd or {"od": "C", "ft": "qf_7Pf__H"}
return json.dumps({"cd": list(cd), "sd": sd}, separators=(",", ":"))
def encrypt_plain(plain: str, key: Sequence[int]) -> str:
data = plain.encode("latin1")
data += b"\x00" * ((8 - len(data) % 8) % 8) # 与真实样本一致: \x00 补齐
return base64.b64encode(xtea_encrypt_bytes(data, list(key))).decode("ascii")
def main() -> int:
ap = argparse.ArgumentParser(description="按本次构建生成 collect")
ap.add_argument("path", help="本次会话的 tdc.js")
ap.add_argument("--from-sample", default=None, help="一份已解密的 cd 数组(json)")
ap.add_argument("--values", default=None, help="语义名->值的 json 字典(实验性)")
ap.add_argument("--out", default=None, help="输出 collect(base64) 的文件")
ap.add_argument("--plain-out", default=None, help="同时输出明文")
ap.add_argument("--kinds", default=None,
help="运行时取样的 items json(提供每项的 kind,见 #36)")
args = ap.parse_args()
prog = tp.load(args.path)
_mid, _entry, seq = tdc_layout.find_manifest_module(prog)
key = extract_xtea_key_pure(open(args.path, encoding="utf-8", errors="replace").read())["key"]
kinds, v0s = load_shape(args.kinds, seq) if args.kinds else (None, None)
spec = SlotSpec(seq, kinds, v0s)
print(f"文件 : {args.path}")
print(f"清单模块 : registry[{_mid}] 项数 {len(seq)}")
print(f"槽位规格 : {spec.describe()}")
print(f"XTEA 密钥 : {key}")
if args.from_sample:
sample = load_sample_cd(args.from_sample)
try:
cd = reorder(sample, spec)
except ValueError as exc:
print(f"❌ {exc}")
return 1
print(f"样本槽数 : {len(sample)} -> 重排后 {len(cd)}")
elif args.values:
vals = json.load(open(args.values, encoding="utf-8"))
cd = []
for (_i, m, _slot, n) in spec.assign:
v = vals[str(m)]
cd.extend(v[:n] if isinstance(v, list) else [v])
print(f"按语义字典 : {len(cd)} 槽")
else:
print("需要 --from-sample 或 --values")
return 1
plain = build_plain(cd)
b64 = encrypt_plain(plain, key)
print(f"明文长度 : {len(plain)}")
print(f"collect : {len(b64)} 字符 (tlg={len(b64)})")
if args.plain_out:
Path(args.plain_out).write_text(plain, encoding="latin1")
print(f"明文 -> {args.plain_out}")
if args.out:
Path(args.out).write_text(b64, encoding="ascii")
print(f"collect -> {args.out}")
return 0
if __name__ == "__main__":
raise SystemExit(main())