diff --git a/docs/设计-账号全生命周期.md b/docs/设计-账号全生命周期.md index d19f401..05e02f8 100644 --- a/docs/设计-账号全生命周期.md +++ b/docs/设计-账号全生命周期.md @@ -266,11 +266,11 @@ CREATE INDEX ix_account_events_ledger | ops | `assigned` | 分配 / 取消分配 | `audit_service.record_account_action`、批量走 `emit_bulk` | — | | ops | `tag_changed` | 标签变更(detail 带 from/to) | 同上 | — | | ops | `remark_changed` | 备注变更(detail 带 from/to) | `accounts.set_account_remark`(单个) | — | -| business | `recharge` | 充值到账 | 收集器(S5) | + ¥ | -| business | `exchange` | 兑换(含消耗积分) | 收集器(S5) | − 积分 | -| business | `points_credit` | 积分到账 | 收集器(S5) | + 积分 | +| business | `recharge` | 充值到账 | `business_events.record_recharge`(二维码/供应商轮询/**供应商回调** 三条路径,同一 `task_ref` 去重) | + ¥ | +| business | `exchange` | 兑换(含消耗积分) | `record_exchange`:`exchange_goods` 的真兑换分支、`exchange_esports_goods`、`exchange_xpd_goods`、`pay_locked_order` | − 积分/点券 | +| business | `points_credit` | 积分到账(detail 带 `confirmed`) | `record_points_credit`:`donate_elite_gift`、`donate_esports_*` | + 积分 | | business | `elite_opened` / `esports_opened` | 开通精英宝典 / 电竞手册 | `douyu_runner_manual` | — | -| business | `bind_ok` / `rebind` | 角色绑定 / 换绑 | 收集器(S5) | — | +| business | `bind_ok`(detail 带 `rebind`) | 角色绑定 / 换绑 | `record_bind`:`confirm_bind` 的真回查分支、`confirm_esports_bind` | — | > **只有一个不落表**:`imported` —— 导入时间直接读 `accounts.created_at`。 > @@ -329,12 +329,33 @@ CREATE INDEX ix_account_events_ledger ### 3.8 `douyu_tasks` 降级:执行日志,不再是账号历史来源 -- 职责收窄为**执行层**:二维码、轮询、锁单、支付、结果回传、WebSocket 推送; -- 业务事实在任务成功时**同时**写入 `account_events`(而不是事后从 JSON 里挖); -- 补保留策略:`query_*` / `refresh_*` 等**查询类任务保留 7 天**,业务类保留 90 天 —— 直接解决 687MB; -- `account_events` 独立长期保留(体量小:实测每天数百条,一年 10 万条以内)。 +> **S5 已落地(2026-10-04)**。职责收窄为**执行层**:二维码、轮询、锁单、支付、 +> 结果回传、WebSocket 推送。业务事实由业务代码在发生那一刻**直接发事件** +> (`services/business_events.py` 的命名构造函数),不再事后从 JSON 里挖。 ---- +**收集器已删除**(`jobs/account_events_collect.py` 连同 `_task_to_event`、 +`COLLECTED_TASK_TYPES`、水位线一起删掉,迁移 `20261004_0043` 清掉作业行)。 +它必然漏数据:靠 allowlist 认业务类型,新增类型没人想起维护就**静默丢** +(电竞兑换、和平小店兑换、锁单支付、电竞礼物积分因此从来没进过时间线)。 + +**保留策略**(`jobs/douyu_tasks_retention.py`,默认**关闭**,由运维打开): + +| 档 | 判定 | 保留 | +| --- | --- | --- | +| 噪声查询 | 只读看一眼就走,且前端**不**拿它派生账号状态 | **7 天** | +| 其余一切(业务事实 + 状态派生依赖的查询 + **以后新增的类型**) | — | **90 天** | + +> 长保留档用 `task_type NOT IN (噪声名单)` 表达,于是**新增任务类型默认长保留**, +> 不会因为「忘了加进某个列表」被误删 —— 正是收集器当年翻车的反面。 +> +> ⚠️ **「状态派生依赖类的查询」不能按 7 天删**:工作台是从任务列表**反推**账号 +> 当前状态的(`DouyuTasksPage` 的 `latest*ByAccount`,且要求 `task.result` 非空)。 +> 这是历史包袱(与 `bind_status` 同源:账号的「现在是什么」存在执行日志里), +> §3.6 的状态列拆解才是根治;在拆解完成前,这类查询保守留长。 +> +> `account_events` 独立长期保留(体量小:每天数百条,一年 10 万条以内)。 +> 注:`DELETE` 只让 InnoDB 的空间**可复用**,文件不会自动缩 —— 要缩需在维护窗口 +> `OPTIMIZE TABLE`。 ## 4. 数据源矩阵:记录从哪来 @@ -342,11 +363,11 @@ CREATE INDEX ix_account_events_ledger | --- | --- | --- | | 导入 | 导入动作本身 | — | | 登录 / CK / 失效 / 重登 | 登录批次、`ck_health`、`ck_relogin` 直接发事件 | — | -| 开通(精英宝典 / 电竞手册) | 开通任务成功时发事件(`run` 内直发,不经 JSON 回捞) | 斗鱼侧状态查询可校正当前值 | -| 充值鱼翅 | 充值任务成功时发事件 | 供应商回调 | -| 积分到账 / 兑换 | 任务成功时发事件 | **斗鱼侧权威接口**:`exchange_records`(兑换)、`xpd_purchase_records`(小店)、`query_points`(积分) | -| 角色绑定 / 换绑 | 绑定任务成功时发事件 | — | -| 人工操作(分配 / 标签 / 停用) | 审计日志 + 直接发事件(需打通 `audit_logs.account_id`) | — | +| 开通(精英宝典 / 电竞手册) | 开通任务成功时发事件(`run` 内直发) | 斗鱼侧状态查询可校正当前值 | +| 充值鱼翅 | **任务成功或供应商回调**时发事件(三条路径同一 `task_ref` 去重) | — | +| 积分到账 / 兑换 | 任务成功时发事件(**电竞与小店兑换、锁单支付也在内**) | **斗鱼侧权威接口**:`query_exchange_records`、`query_xpd_purchase_records`、`query_points` | +| 角色绑定 / 换绑 | 确认接口成功 + 回查生效时发事件 | — | +| 人工操作(分配 / 标签 / 备注 / 停用) | 审计日志 + 直接发事件(`audit_logs.account_id` 已打通) | — | > **历史怎么办**:**不从 `douyu_tasks` 回填**(噪声大、结构靠猜)。 > 需要历史时走**斗鱼侧权威接口按需同步**(如 `query_exchange_records`), @@ -441,7 +462,7 @@ WHERE a.cookie <> src.cookie; | **S2** | 读写一次切换:12 个文件、24 读点 / 10 写点全部改 `LoginTask.*` → `Account.*`;批次行复用逻辑重写;在途子查询 + 索引 | 登录记录与 CK 彻底解耦 | S1 | | **S3** ✅ | 状态收口(**已完成 2026-10-04**):`account_lifecycle` 提供 `derive_status()` / `sync_status()` / `set_manual_state()` / `on_login_success()`;9 处判据写入点全部接线;6 状态状态机上线;自动化选行排除 `disabled`/`archived`;新增 `account:manage` 权限与停用/归档/恢复接口;前端列表加状态列与筛选、详情页加状态卡与操作入口 | `status` 可信 | S2 | | **S4** ✅ | 事件发射点补齐(**已完成 2026-10-04**):`assigned` / `tag_changed` 接上(此前定义了却从没发过)、新增 `ck_cleared`、`ck_invalid` 改为只在转变时发;`audit_logs.account_id` 在分配/标签/检测/清 CK 处打通;新增 `emit_bulk`(批量操作逐账号写事件)与 `record_account_action`(审计+事件一次写完);时间线支持 `category` 筛选;前端补 `status_changed` / `ck_cleared` 样式与分类筛选;补上改备注接口 `PUT /api/accounts/{id}/remark`(`remark_changed` 从此有发射点)| 时间线完整 | S2 | -| **S5** | 业务事实直发事件:开通 / 充值 / 积分 / 兑换 / 绑定在任务成功时写事件;`douyu_tasks` 加保留策略 | 台账可用,687MB 可控 | S4 | +| **S5** ✅ | 业务事实直发事件(**已完成 2026-10-04**):新增 `services/business_events.py` 命名构造函数(`record_recharge` / `record_exchange` / `record_points_credit` / `record_bind`);9 处成功点直发(含**此前从未收录**的电竞兑换、和平小店兑换、锁单支付、电竞礼物积分);修掉「到账判据 `!=`」与「回调晚于超时」两个洞;幂等键 `douyu_task:{id}` 让直发与旧收集器/回调互相去重;**删除收集器**(含映射、水位线、测试,迁移 `0043` 清作业行);新增 `douyu_tasks` 保留策略作业(噪声 7 天 / 其余 90 天,默认关闭) | 台账可用,687MB 可控 | S4 | | **S6** | 账号页重构:列表加状态与关键列;详情页做成「生命周期主页」(状态卡 + 台账 + 时间线 + 操作);`bind_status` 三列拆解落地 | 账号管理真正独立 | S3、S5 | | **S7** | 账号检测并入账号域(`account-check` 从「粘贴文本」改为「按账号选」,结果落事件) | 检测不再是孤岛 | S6 | | **S8** | 清理:drop `login_tasks` 旧列与 `accounts` 三个 `*_bind_status` 列(**不可逆,前置:每日备份 + 反向回填脚本**) | 包袱丢干净 | S6 | diff --git a/docs/设计-账号详情页与事件历史.md b/docs/设计-账号详情页与事件历史.md index 671e62a..f51d823 100644 --- a/docs/设计-账号详情页与事件历史.md +++ b/docs/设计-账号详情页与事件历史.md @@ -106,10 +106,29 @@ class AccountEvent(Base): | `esports_opened` | **电竞手册开通**(新增记录) | `douyu_runner_manual` | | `assigned` / `tag_changed` / `remark_changed` | 人工操作 | `audit_logs`(需补 account_id,见 §7) | -## 4. 业务事件怎么收录:增量收集器(不回填) +## 4. 业务事件怎么收录:~~增量收集器~~ → 已改为**直发**(不回填) -既有历史**不导入**。业务事件(充值/兑换/积分到账/绑定)用**一个增量收集器**从 `douyu_tasks` -往后提取 —— 好处是**完全不侵入 runner 代码**(不必改 8 个业务文件),且新老任务类型统一处理。 +> ## ⚠️ 本节已作废(2026-10-04,S5) +> +> 收集器方案**已放弃并删除代码**。原因(详见 +> [账号全生命周期设计](设计-账号全生命周期.md) §3.8): +> +> 1. **靠 allowlist 认业务类型必然漏**:新增类型没人想起维护名单就静默丢数据 —— +> 电竞兑换、和平小店兑换、锁单支付、电竞礼物积分因此**从来没进过时间线**; +> 2. **「任务 success」不等于「业务真的发生了」**:预约到货、已预兑、「检测到本来就 +> 已绑定」、积分未确认到账,都被标成 success,收集器照单全收写成事实; +> 3. **字段路径靠抽样猜**:二维码充值的 `result` 里根本没有金额,收集器因此恒写 +> `amount=None`,`recharge_total` 系统性少计; +> 4. 作业**默认关闭**,生产上它一直没跑。 +> +> 现在业务代码在动作成功那一刻用 `services/business_events.py` 的命名构造函数直发 +> (幂等键 `douyu_task:{id}`,与旧收集器一致,迁移窗口内可互相去重)。 +> +> 下文 §4.1 / §4.2 / §5.2 保留作为**历史记录**:其中的 `result` 字段实测结论仍然 +> 有价值(接入斗鱼侧接口时用得上),但**不要再按它实现收集器**。 + +既有历史**不导入**。业务事件(充值/兑换/积分到账/绑定)~~用**一个增量收集器**从 `douyu_tasks` +往后提取 —— 好处是**完全不侵入 runner 代码**(不必改 8 个业务文件),且新老任务类型统一处理~~。 ### 4.1 收集器设计 @@ -264,7 +283,7 @@ class AccountEvent(Base): | E1 | `feat(db): account_events 表 + audit_logs.account_id` | 无 | | E2 | `feat(events): 事件发射 helper + CK 事件(失效/登录/重登)` | **CK 解耦已完成**(否则写入点还会变) | | E3 | `feat(events): 开通事件(精英宝典 / 电竞手册)` | 无 | -| E4 | `feat(events): douyu_tasks 增量收集器 + 水位线` | 无 | +| ~~E4~~ | ~~`feat(events): douyu_tasks 增量收集器 + 水位线`~~ **已废弃**(S5 改为直发并删除) | 无 | | E5 | `feat(api): 账号事件与概览接口` | E1 | | E6 | `feat(ui): 账号详情页(状态卡 + 统计 + 时间线)` | E5 | | E7 | `docs: 落点与验收` | —— | @@ -282,7 +301,7 @@ class AccountEvent(Base): 4. **发射事件是旁路**:必须保证 `emit()` 抛错不影响登录/兑换主流程(用 try 包住 + 记日志); 5. **积分口径要先定**(决策 C):`accounts.points` 与 `donate_elite_gift` 的 `points_query.points` 有不一致的情况(如兑换后 `points_query.points=0`),时间线显示哪个要明确; -6. 绑定类任务的 `result` 结构**尚未抽样**(§4.2),收集器实现前先 dump 一条确认。 +6. ~~绑定类任务的 `result` 结构尚未抽样(§4.2),收集器实现前先 dump 一条确认。~~ **已随收集器一并作废**:直发不再需要解析 `result`。 ## 10. 待确认的决策点 diff --git a/tests/test_account_events.py b/tests/test_account_events.py index 3fae8a2..794f661 100644 --- a/tests/test_account_events.py +++ b/tests/test_account_events.py @@ -17,9 +17,8 @@ from sqlalchemy.orm import sessionmaker import web.backend.database as database_module from tests.support.db import get_test_engine, make_test_session -from web.backend.models import Account, AccountEvent, AutomationJob, DouyuTask, User +from web.backend.models import Account, AccountEvent, User from web.backend.services import account_events_service as svc -from web.backend.services.automation.jobs import account_events_collect as collect_job @contextmanager @@ -49,10 +48,6 @@ class _Base: database_module.SessionLocal.configure(bind=self.engine) self.session = make_test_session(self.engine) self.session.query(AccountEvent).delete() - self.session.query(DouyuTask).delete() - self.session.query(AutomationJob).filter( - AutomationJob.key == collect_job.JOB_KEY - ).delete() self.session.commit() user = User(username="evt-admin", password_hash="h", role="super_admin") @@ -339,263 +334,6 @@ class TestEmit(_Base): db.close() -class TestTaskMapping: - def test_exchange_maps_amount_negative(self): - payload = svc._task_to_event( - "exchange_goods", - { - "commodity_name": "套装-狂战萌兔", - "commodity_id": "20260120QYOOB_bag5", - "score": 480, - "points_query": {"points": 0}, - "order_id": "147874", - "game_name": "窝补誓渣男", - "game_channel": "微信 / 安卓(android)", - }, - ) - assert payload["event_type"] == svc.EVENT_EXCHANGE - assert payload["amount"] == -480, "消耗要记成负数,否则累计统计会反向" - assert payload["balance_after"] == 0 - assert payload["detail"]["game_name"] == "窝补誓渣男" - - def test_recharge_maps_amount(self): - payload = svc._task_to_event( - "create_gold_qr", - { - "buy_num": 98, - "order_id": "DYGF5C24C5E4EA84T238035", - "supplier_order": { - "order_id": "200000229425", - "finish_time": "2026-10-03 11:46:07", - }, - "recharge_channel": "supplier_api", - "douyu_uid": "678503799", - }, - ) - assert payload["event_type"] == svc.EVENT_RECHARGE - assert payload["amount"] == 98 - assert payload["detail"]["supplier_order_id"] == "200000229425" - - def test_points_credit_keeps_expire_time(self): - payload = svc._task_to_event( - "donate_elite_gift", - { - "gift_points_expected_gain": 980, - "points": 1280, - "gift_points_baseline": 300, - "gift_points_target": 1280, - "points_query": {"raw": {"data": {"willExpireTime": 1798819199}}}, - }, - ) - assert payload["event_type"] == svc.EVENT_POINTS_CREDIT - assert payload["amount"] == 980 - assert payload["balance_after"] == 1280 - assert payload["detail"]["points_expire_at"] == 1798819199 - - def test_unparsable_result_returns_none(self): - assert svc._task_to_event("exchange_goods", "{not json") is None - assert svc._task_to_event("exchange_goods", None) is None - assert svc._task_to_event("unknown_type", {"a": 1}) is None - - -class TestCollector(_Base): - def _make_job(self, watermark=None): - job = AutomationJob( - key=collect_job.JOB_KEY, - title="账号事件收集", - enabled=True, - interval_seconds=600, - batch_size=100, - concurrency=1, - config={"watermark": watermark} if watermark is not None else None, - updated_at=datetime.now(UTC), - ) - self.session.add(job) - self.session.commit() - return job - - def _add_task(self, task_type, result, status="success", account_id=None): - task = DouyuTask( - batch_id="evt-batch", - account_id=account_id or self.account_id, - task_type=task_type, - status=status, - result=result, - created_by=self.user_id, - ) - self.session.add(task) - self.session.commit() - return task.id - - def test_first_run_sets_watermark_without_backfill(self): - """首次运行只定水位线,不回溯历史(2026-10-03 已定不回填)。""" - self._add_task( - "exchange_goods", - {"commodity_name": "旧兑换", "score": 100, "points_query": {"points": 0}}, - ) - job = self._make_job() - db = self._fresh() - try: - result = collect_job.run_account_events_collect(db, job) - finally: - db.close() - assert result["detail"]["first_run"] is True - assert result["scanned"] == 0 - - db = self._fresh() - try: - assert db.query(AccountEvent).count() == 0, "首次运行不应回填历史事件" - finally: - db.close() - - def test_collects_only_new_rows_and_advances_watermark(self): - first = self._add_task( - "exchange_goods", - {"commodity_name": "老兑换", "score": 100, "points_query": {"points": 5}}, - ) - job = self._make_job(watermark=first) - new_id = self._add_task( - "create_gold_qr", - {"buy_num": 68, "order_id": "DYGF-x", "supplier_order": {}}, - ) - - db = self._fresh() - try: - result = collect_job.run_account_events_collect(db, job) - finally: - db.close() - - assert result["detail"]["written"] == 1 - assert result["detail"]["watermark"] == new_id - - db = self._fresh() - try: - rows = db.query(AccountEvent).all() - assert len(rows) == 1 - assert rows[0].event_type == svc.EVENT_RECHARGE - assert rows[0].source_ref == f"douyu_task:{new_id}" - finally: - db.close() - - def test_rerun_does_not_duplicate(self): - """重复跑不能造重复事件(source_ref 兜底)。""" - job = self._make_job(watermark=0) - self._add_task( - "create_gold_qr", - {"buy_num": 98, "order_id": "o1", "supplier_order": {}}, - ) - db = self._fresh() - try: - collect_job.run_account_events_collect(db, job) - finally: - db.close() - - # 把水位线倒回去,模拟「重复处理同一批」 - job2 = self._fresh() - try: - row = ( - job2.query(AutomationJob) - .filter(AutomationJob.key == collect_job.JOB_KEY) - .first() - ) - row.config = {"watermark": 0} - job2.commit() - finally: - job2.close() - - db = self._fresh() - try: - job3 = ( - db.query(AutomationJob) - .filter(AutomationJob.key == collect_job.JOB_KEY) - .first() - ) - collect_job.run_account_events_collect(db, job3) - finally: - db.close() - - db = self._fresh() - try: - assert db.query(AccountEvent).count() == 1 - finally: - db.close() - - def test_failed_task_produces_no_event(self): - """失败的兑换不该记成「消耗了积分」。""" - job = self._make_job(watermark=0) - self._add_task( - "exchange_goods", - {"commodity_name": "失败兑换", "score": 100}, - status="failed", - ) - db = self._fresh() - try: - result = collect_job.run_account_events_collect(db, job) - finally: - db.close() - assert result["detail"]["written"] == 0 - assert result["skipped"] == 1 - - def test_unparsable_row_does_not_stall_watermark(self): - """坏数据要跳过并推进水位线,不能让事件流永久停摆。""" - job = self._make_job(watermark=0) - bad_id = self._add_task("exchange_goods", "{not json") - good_id = self._add_task( - "create_gold_qr", {"buy_num": 1, "order_id": "o2", "supplier_order": {}} - ) - db = self._fresh() - try: - result = collect_job.run_account_events_collect(db, job) - finally: - db.close() - assert result["detail"]["written"] == 1 - assert result["detail"]["watermark"] == good_id - assert good_id > bad_id - - def test_collected_occurred_at_matches_source_task_time(self): - """⚠️ 端到端回归:收集器写的事件时间必须等于源任务的业务时间。 - - 这条用例存在的理由:收集器用**裸 SQL** 读 `douyu_tasks`,拿到的是 naive - datetime(绕过 `UTCDateTime`)。旧实现把它当本地时间换算,导致事件时间 - 整整早 8 小时(本机 UTC+8,库里实测过)——而「什么时候兑换/充值了」 - 正是这张表存在的意义。 - - 旧用例只断言 written / watermark 的数量,从不看时间,所以这个 bug - 一路绿灯溜了过去。 - """ - from web.backend.models import DouyuTask - - task_id = self._add_task( - "create_gold_qr", - {"buy_num": 98, "order_id": "o-time", "supplier_order": {}}, - ) - # 给源任务一个明确的业务完成时间(UTC 语义) - source = self.session.query(DouyuTask).filter(DouyuTask.id == task_id).one() - source.finished_at = datetime(2026, 10, 3, 11, 46, 7) # noqa: DTZ001 - self.session.commit() - - job = self._make_job(watermark=task_id - 1) - # 显式切到 UTC+8:这个 bug 在 UTC 机器上不可见,不切时区会假绿。 - with _force_non_utc_timezone(): - db = self._fresh() - try: - result = collect_job.run_account_events_collect(db, job) - finally: - db.close() - assert result["detail"]["written"] == 1 - - db = self._fresh() - try: - event = ( - db.query(AccountEvent) - .filter(AccountEvent.source_ref == f"douyu_task:{task_id}") - .one() - ) - assert event.occurred_at == datetime(2026, 10, 3, 11, 46, 7, tzinfo=UTC) - finally: - db.close() - - class TestAccountApi(_Base): """账号详情接口:概览统计 + 时间线分页 + 删除账号时清理事件。""" diff --git a/tests/test_business_events.py b/tests/test_business_events.py new file mode 100644 index 0000000..9f35c69 --- /dev/null +++ b/tests/test_business_events.py @@ -0,0 +1,345 @@ +"""S5:业务事实直发事件。 + +取代了原来的「收集器事后扫 `douyu_tasks.result` 反推事件」。本文件守住三件事: + +1. **命名构造函数的字段口径**(金额符号、detail、subject); +2. **幂等**:`source_ref=task_ref(task_id)` —— 让直发与仍在跑的收集器能互相去重, + 也让「runner 轮询」与「供应商回调」不会对同一笔充值重复记账; +3. **发射顺序**:事件必须在 `_mark_task` **之前**发。`_mark_task` 内部 commit, + 而 `emit` 只 RELEASE SAVEPOINT、不提交外层事务,runner 的 `finally` 只 close + (未提交即回滚)—— 发在后面会被静默丢弃,且 `emit` 仍返回 True。 +""" + +from datetime import UTC, datetime + +from tests.support.db import get_test_engine, make_test_session +from web.backend.models import Account, AccountEvent, DouyuTask, User +from web.backend.services import account_events_service as svc +from web.backend.services import business_events as biz + +NOW = datetime(2026, 10, 4, 12, 0, 0, tzinfo=UTC) + + +class _Base: + def setup_method(self): + self.engine = get_test_engine() + self.session = make_test_session(self.engine) + self.session.query(AccountEvent).delete() + self.user = User(username="biz-admin", password_hash="h", role="super_admin") + self.session.add(self.user) + self.session.flush() + self.account = Account( + username="biz-acc", password="p", email="b@e.com", email_password="ep" + ) + self.session.add(self.account) + self.session.commit() + + def teardown_method(self): + self.session.close() + + def events(self, event_type: str) -> list[AccountEvent]: + return ( + self.session.query(AccountEvent) + .filter( + AccountEvent.account_id == self.account.id, + AccountEvent.event_type == event_type, + ) + .order_by(AccountEvent.id.asc()) + .all() + ) + + +class TestConstructors(_Base): + def test_recharge_records_amount_and_channel(self): + assert ( + biz.record_recharge( + self.session, + self.account, + task_id=101, + amount=100, + channel="wechat_qr", + douyu_uid="u1", + ) + is True + ) + self.session.commit() + events = self.events(svc.EVENT_RECHARGE) + assert len(events) == 1 + event = events[0] + assert event.amount == 100 + assert event.category == svc.CATEGORY_BUSINESS + assert event.source_ref == "douyu_task:101" + assert event.detail["channel"] == "wechat_qr" + assert event.detail["confirmed"] is True + + def test_recharge_without_amount_still_records(self): + """金额拿不到时宁可记一条金额为空的事件,也不要凭空编一个数字。""" + biz.record_recharge(self.session, self.account, task_id=102, amount=None) + self.session.commit() + event = self.events(svc.EVENT_RECHARGE)[0] + assert event.amount is None + + def test_recharge_marks_supplier_confirmation_honestly(self): + """供应商终态成功 ≠ 斗鱼余额已变,必须如实标注。""" + biz.record_recharge( + self.session, + self.account, + task_id=103, + amount=50, + channel="supplier_api", + confirmed=False, + ) + self.session.commit() + event = self.events(svc.EVENT_RECHARGE)[0] + assert event.detail["confirmed"] is False + assert "供应商回执" in event.title + + def test_exchange_amount_is_negative_and_subject_is_commodity(self): + biz.record_exchange( + self.session, + self.account, + task_id=201, + name="兔兔白日梦", + score=1200, + commodity_id="sku-9", + points_after=800, + scope="elite", + ) + self.session.commit() + event = self.events(svc.EVENT_EXCHANGE)[0] + assert event.amount == -1200 + assert event.balance_after == 800 + assert event.subject_type == "commodity" + assert event.subject_key == "sku-9" + assert event.detail["scope"] == "elite" + + def test_exchange_without_score_keeps_amount_null(self): + biz.record_exchange( + self.session, self.account, task_id=202, name="x", score=None + ) + self.session.commit() + assert self.events(svc.EVENT_EXCHANGE)[0].amount is None + + def test_points_credit_marks_unconfirmed(self): + """未确认到账也记,但必须标出来 —— 不能把「预期」写成「已到账」。""" + biz.record_points_credit( + self.session, + self.account, + task_id=301, + gain=980, + points_after=100, + confirmed=False, + ) + self.session.commit() + event = self.events(svc.EVENT_POINTS_CREDIT)[0] + assert event.amount == 980 + assert event.detail["confirmed"] is False + assert "未确认" in event.title + + def test_points_credit_confirmed(self): + biz.record_points_credit( + self.session, self.account, task_id=302, gain=10, confirmed=True + ) + self.session.commit() + event = self.events(svc.EVENT_POINTS_CREDIT)[0] + assert event.detail["confirmed"] is True + assert "未确认" not in event.title + + def test_bind_records_scope_and_rebind(self): + biz.record_bind( + self.session, + self.account, + task_id=401, + scope="esports", + role_name="小明", + rebind=True, + ) + self.session.commit() + event = self.events(svc.EVENT_BIND_OK)[0] + assert event.subject_type == "esports_role" + assert event.subject_key == "小明" + assert event.detail["rebind"] is True + assert "换绑" in event.title + + def test_bind_weak_is_flagged(self): + biz.record_bind( + self.session, + self.account, + task_id=402, + scope="elite", + role_name="小华", + weak=True, + ) + self.session.commit() + event = self.events(svc.EVENT_BIND_OK)[0] + assert event.detail["confirmed"] is False + assert "未回查确认" in event.title + + +class TestIdempotency(_Base): + """同一个 task 只能对应一条事实 —— 这是直发与收集器/回调共存的基石。""" + + def test_same_task_id_does_not_duplicate(self): + biz.record_recharge(self.session, self.account, task_id=500, amount=10) + self.session.commit() + # 模拟「另一个路径也观测到同一笔」(收集器重跑 / 供应商回调) + biz.record_recharge(self.session, self.account, task_id=500, amount=10) + self.session.commit() + assert len(self.events(svc.EVENT_RECHARGE)) == 1 + + def test_different_task_id_is_a_new_fact(self): + biz.record_recharge(self.session, self.account, task_id=501, amount=10) + self.session.commit() + biz.record_recharge(self.session, self.account, task_id=502, amount=10) + self.session.commit() + assert len(self.events(svc.EVENT_RECHARGE)) == 2 + + def test_task_ref_matches_old_collector_key(self): + """必须与旧收集器的 key 完全一致,否则迁移窗口内会重复记账。""" + assert biz.task_ref(123) == "douyu_task:123" + + +class TestEmitSurvivesWorkerSessionClose(_Base): + """发射顺序的**行为验证**:事件要能挺过 runner 的 `finally: worker_db.close()`。 + + runner 的每个任务在 `_execute_one` 的 finally 里只 close 不 commit。 + `_mark_task` 内部有 commit,所以「紧贴 _mark_task 之前 emit」会被那次 commit + 一起提交;若把 emit 放在 _mark_task 之后,它就落在一个没人提交的事务里, + close 时被回滚 —— 而且 `emit` 照样返回 True,没有任何日志。 + """ + + def test_event_committed_together_with_mark_task(self): + # 模拟 runner:emit 后由 _mark_task 的 commit 收尾 + biz.record_recharge(self.session, self.account, task_id=600, amount=7) + self.session.commit() # 等价于 _mark_task 内部的 commit + self.session.close() # 等价于 runner 的 finally + + fresh = make_test_session(self.engine) + try: + rows = ( + fresh.query(AccountEvent) + .filter(AccountEvent.source_ref == "douyu_task:600") + .all() + ) + assert len(rows) == 1, "事件没有挺过 session close —— 发射顺序错了" + finally: + fresh.close() + + def test_emit_without_commit_is_lost_on_close(self): + """反证:不 commit 就 close,事件确实会丢(这就是顺序规则存在的理由)。""" + biz.record_recharge(self.session, self.account, task_id=601, amount=7) + self.session.close() # 故意不 commit + + fresh = make_test_session(self.engine) + try: + rows = ( + fresh.query(AccountEvent) + .filter(AccountEvent.source_ref == "douyu_task:601") + .all() + ) + assert rows == [], "若不 commit 也能落库,说明事务边界与预期不符" + finally: + fresh.close() + + +class TestCollectorIsGone: + """收集器必须被彻底删掉,不留半条命。""" + + def test_module_removed(self): + import importlib + + try: + importlib.import_module( + "web.backend.services.automation.jobs.account_events_collect" + ) + except ModuleNotFoundError: + return + raise AssertionError("收集器模块还在") + + def test_no_collector_symbols_left(self): + assert not hasattr(svc, "COLLECTED_TASK_TYPES") + assert not hasattr(svc, "_task_to_event") + + def test_job_key_not_registered(self): + """⚠️ `registered_jobs()` 返回的是 **dict**(key → JobSpec)。 + + 写成 `{spec.key for spec in registered_jobs()}` 会遍历出字符串键、 + `spec.key` 直接报错;而如果它恰好是空字典,推导式又会得到空集让断言**假绿**。 + 所以这里显式 register 后再按键判断。 + """ + from web.backend.services.automation import ( + register_default_jobs, + registered_jobs, + ) + + register_default_jobs() + keys = set(registered_jobs()) + assert "account_events_collect" not in keys + # 反向确认这个断言不是假绿:新作业确实在里面 + assert "douyu_tasks_retention" in keys + + +class TestEmitOrderingInRunners: + """源码扫描:runner 里的 `record_*` 必须在 `_mark_task(..., "success")` **之前**。 + + 这条规则很隐蔽(违反了不会报错、只是事件静默消失),所以用用例钉住。 + 做法:对每个 `record_*` 调用,找它之后出现的第一个 `_mark_task(`,该调用里 + 必须带 `"success"`;如果事件被写在 mark 之后,那么它后面第一个 `_mark_task` + 要么是另一个分支的、要么根本不存在 —— 两种情况都会让用例变红。 + """ + + RUNNER_FILES = ( + "web/backend/services/douyu_runner_goods.py", + "web/backend/services/douyu_runner_gold.py", + "web/backend/services/douyu_runner_bind.py", + "web/backend/services/douyu_runner_donate.py", + "web/backend/services/douyu_runner_xpd.py", + ) + + def test_record_calls_precede_success_mark(self): + import re + from pathlib import Path + + root = Path(__file__).resolve().parents[1] + record_re = re.compile(r"\brecord_(recharge|exchange|points_credit|bind)\(") + offenders: list[str] = [] + checked = 0 + + for rel in self.RUNNER_FILES: + source = (root / rel).read_text(encoding="utf-8") + for match in record_re.finditer(source): + checked += 1 + tail = source[match.end() :] + mark = tail.find("_mark_task(") + line_no = source[: match.start()].count("\n") + 1 + if mark == -1: + offenders.append(f"{rel}:{line_no} 之后没有 _mark_task") + continue + # 取这次 _mark_task 调用的前若干字符,判断是不是 success 分支 + snippet = tail[mark : mark + 400] + if '"success"' not in snippet: + offenders.append( + f"{rel}:{line_no} 之后的第一个 _mark_task 不是 success" + ) + + assert checked >= 10, f"只扫到 {checked} 个 record_ 调用,扫描规则可能失效了" + assert not offenders, ( + "record_* 必须在 _mark_task(success) **之前**调用" + "(否则事件会被 runner 的 close 静默回滚):\n" + "\n".join(offenders) + ) + + +class TestTaskReadyForApi(_Base): + def test_douyu_task_still_works(self): + """兜底:删掉收集器不应影响 douyu_tasks 本身的读写。""" + task = DouyuTask( + batch_id="b1", + account_id=self.account.id, + task_type="exchange_goods", + status="success", + result={"commodity_name": "x"}, + created_by=self.user.id, + ) + self.session.add(task) + self.session.commit() + assert task.id is not None diff --git a/tests/test_douyu_tasks_retention.py b/tests/test_douyu_tasks_retention.py new file mode 100644 index 0000000..9293707 --- /dev/null +++ b/tests/test_douyu_tasks_retention.py @@ -0,0 +1,248 @@ +"""S5b:`douyu_tasks` 保留策略。 + +守住的核心不是「删得掉」,而是**删得对**: + +- 噪声查询按短窗口删; +- **业务事实与前端状态派生依赖的查询按长窗口删**(删早了工作台账号行会变「未查」); +- **未知类型默认长保留**(新增业务类型不该因为「忘了加进某个列表」而被误删); +- **在途任务绝不删**(批次线程正拿着那一行)。 +""" + +from datetime import UTC, datetime, timedelta + +from sqlalchemy import func + +from tests.support.db import get_test_engine, make_test_session +from web.backend.models import Account, AutomationJob, DouyuTask, User +from web.backend.services.automation.jobs import douyu_tasks_retention as retention + +NOW = datetime.now(UTC) + + +class _Base: + def setup_method(self): + self.engine = get_test_engine() + self.session = make_test_session(self.engine) + self.session.query(DouyuTask).delete() + self.session.query(AutomationJob).filter( + AutomationJob.key == retention.JOB_KEY + ).delete() + self.session.commit() + + self.user = User(username="ret-admin", password_hash="h", role="super_admin") + self.session.add(self.user) + self.session.flush() + self.account = Account( + username="ret-acc", password="p", email="r@e.com", email_password="ep" + ) + self.session.add(self.account) + self.session.commit() + # 记下 id:`run()` 会 expunge_all(),之后不能再碰 self.account 的属性 + # (commit 会 expire 属性,detach 后再访问就是 DetachedInstanceError)。 + self.account_id = self.account.id + self.user_id = self.user.id + + def teardown_method(self): + self.session.close() + + def make_task( + self, + task_type: str, + *, + age_days: float, + status: str = "success", + result: dict | None = None, + ) -> DouyuTask: + task = DouyuTask( + batch_id="ret-batch", + account_id=self.account_id, + task_type=task_type, + status=status, + message="m", + result=result if result is not None else {"payload": {}}, + created_by=self.user_id, + created_at=NOW - timedelta(days=age_days), + ) + self.session.add(task) + self.session.commit() + return task + + def job(self, **config) -> AutomationJob: + """取(或建)作业行。 + + 幂等:`automation_jobs` 的主键就是 `key`,一个用例里跑两次不能插两行。 + """ + job = self.session.get(AutomationJob, retention.JOB_KEY) + if job is None: + job = AutomationJob( + key=retention.JOB_KEY, + title="任务日志清理", + enabled=True, + interval_seconds=24 * 3600, + batch_size=100, + concurrency=1, + config=config or None, + updated_at=NOW, + ) + self.session.add(job) + job.config = config or None + job.batch_size = 100 + self.session.commit() + return job + + def run(self, **config) -> dict: + out = retention.run_douyu_tasks_retention(self.session, self.job(**config)) + # 作业用**裸 SQL** 删行,ORM 的 identity map 不知道行没了; + # 不 expunge 的话后面访问这些对象会抛 ObjectDeletedError(而不是干净地查不到)。 + self.session.expunge_all() + return out + + def remaining(self) -> set[tuple[str, int]]: + return {(t.task_type, t.id) for t in self.session.query(DouyuTask).all()} + + def count(self) -> int: + return int(self.session.query(func.count(DouyuTask.id)).scalar() or 0) + + +class TestRetention(_Base): + def test_noise_query_deleted_after_short_window(self): + self.make_task("query_limited_goods", age_days=10) + self.make_task("query_change_bind_time", age_days=10) + out = self.run(noise_days=7, business_days=90) + assert out["deleted"] == 2 + assert out["deleted_noise"] == 2 + assert self.remaining() == set() + + def test_noise_query_kept_within_short_window(self): + self.make_task("query_limited_goods", age_days=3) + out = self.run(noise_days=7, business_days=90) + assert out["deleted"] == 0 + assert self.count() == 1 + + def test_business_fact_kept_past_short_window(self): + """业务事实不该被 7 天窗口带走 —— 这是分档的全部意义。""" + for task_type in ( + "exchange_goods", + "create_gold_qr", + "confirm_bind", + "exchange_xpd_goods", + ): + self.make_task(task_type, age_days=30) + out = self.run(noise_days=7, business_days=90) + assert out["deleted"] == 0 + assert self.count() == 4 + + def test_business_fact_deleted_after_long_window(self): + self.make_task("exchange_goods", age_days=100) + out = self.run(noise_days=7, business_days=90) + assert out["deleted"] == 1 + assert out["deleted_rest"] == 1 + + def test_state_derivation_queries_kept_longer(self): + """工作台从这些查询的 result 反推账号当前状态,删早了显示会变「未查」。""" + for task_type in ( + "query_xpd_bind_info", + "query_xpd_role", + "query_esports_game_name", + "query_game_name", + "prepare_esports_bind", + "get_esports_bind_qr", + ): + self.make_task(task_type, age_days=30) + out = self.run(noise_days=7, business_days=90) + assert out["deleted"] == 0, f"状态派生依赖的任务被误删:{self.remaining()}" + + def test_unknown_task_type_defaults_to_long_retention(self): + """新增的任务类型默认长保留 —— 绝不能因为「没列进名单」而被删掉。""" + self.make_task("some_brand_new_task_type", age_days=30) + out = self.run(noise_days=7, business_days=90) + assert out["deleted"] == 0 + assert self.count() == 1 + + # 但只要过了长窗口就该被清掉(否则未知类型会永久堆积) + self.make_task("another_new_type", age_days=200) + out = self.run(noise_days=7, business_days=90) + assert out["deleted"] == 1 + + def test_in_flight_tasks_never_deleted(self): + """批次线程可能正拿着这些行,无论多老都不能删。""" + for status in ("planned", "pending", "running"): + self.make_task("query_limited_goods", age_days=999, status=status) + self.make_task("query_limited_goods", age_days=999, status="success") + out = self.run(noise_days=7, business_days=90) + assert out["deleted"] == 1 + statuses = {t.status for t in self.session.query(DouyuTask).all()} + assert statuses == {"planned", "pending", "running"} + + def test_config_windows_are_honoured(self): + self.make_task("query_limited_goods", age_days=2) + out = self.run(noise_days=1, business_days=90) + assert out["deleted"] == 1 + assert out["noise_days"] == 1 + + def test_invalid_config_falls_back_to_defaults(self): + out = self.run(noise_days=0, business_days="abc") + assert out["noise_days"] == retention.DEFAULT_NOISE_DAYS + assert out["business_days"] == retention.DEFAULT_BUSINESS_DAYS + + def test_result_payload_is_dropped_with_the_row(self): + """删整行(而不是只清 result)—— 避免「有行但读不出状态」的隐蔽坏状态。""" + task = self.make_task( + "query_limited_goods", + age_days=30, + result={"limited_goods": [{"a": 1}] * 100}, + ) + task_id = task.id # 先取:run() 会 expunge_all,之后对象就 detached 了 + self.run(noise_days=7, business_days=90) + assert self.session.get(DouyuTask, task_id) is None + + def test_batch_budget_is_reported_not_overclaimed(self): + """预算用尽只报「预算用光」,不谎称「仍有积压」。 + + 预算在上一档用光时,下一档根本没轮到执行 —— 这时无法区分「没得删」和 + 「没轮到」,所以只给提示。 + """ + for _ in range(5): + self.make_task("query_limited_goods", age_days=30) + out = self.run(noise_days=7, business_days=90, max_batches=1) + # batch_size=100(job 默认),5 条一轮删完(不足一批就 break),预算正好用光 + assert out["batches"] == 1 + assert out["deleted"] == 5 + assert out["batch_budget_exhausted"] is True + + def test_budget_not_exhausted_when_work_runs_out(self): + self.make_task("query_limited_goods", age_days=30) + out = self.run(noise_days=7, business_days=90, max_batches=50) + assert out["batch_budget_exhausted"] is False + + +class TestClassification(_Base): + def test_noise_list_has_no_duplicates(self): + assert len(retention.NOISE_TASK_TYPES) == len(set(retention.NOISE_TASK_TYPES)) + + def test_noise_list_has_no_business_types(self): + business = { + "exchange_goods", + "exchange_esports_goods", + "exchange_xpd_goods", + "pay_locked_order", + "lock_goods", + "create_gold_qr", + "donate_elite_gift", + "donate_esports_chicken_gift", + "donate_esports_firework_gift", + "confirm_bind", + "confirm_esports_bind", + "confirm_xpd_bind", + "create_elite_qr", + "create_esports_qr", + } + overlap = business & set(retention.NOISE_TASK_TYPES) + assert not overlap, f"业务类型被当成噪声了:{overlap}" + + def test_noise_types_exist_in_supported_types(self): + """噪声名单里的类型必须都是真实存在的任务类型(防拼写错误导致永不生效)。""" + from web.backend.services.douyu_service import SUPPORTED_DOUYU_TASK_TYPES + + unknown = set(retention.NOISE_TASK_TYPES) - set(SUPPORTED_DOUYU_TASK_TYPES) + assert not unknown, f"噪声名单里有不存在的任务类型:{unknown}" diff --git a/tests/test_migrations.py b/tests/test_migrations.py index 7cde956..dee9120 100644 --- a/tests/test_migrations.py +++ b/tests/test_migrations.py @@ -26,7 +26,7 @@ TESTS_DIR = Path(__file__).resolve().parent PROJECT_ROOT = TESTS_DIR.parent VERSIONS_DIR = PROJECT_ROOT / "web" / "backend" / "migrations" / "versions" -HEAD_REVISION = "20261003_0042" +HEAD_REVISION = "20261004_0043" # 迁移新增、但不声明在模型里的复合查询索引。 EXTRA_INDEXES = ( diff --git a/web/backend/migrations/versions/20261004_0043_drop_events_collect_job.py b/web/backend/migrations/versions/20261004_0043_drop_events_collect_job.py new file mode 100644 index 0000000..1428485 --- /dev/null +++ b/web/backend/migrations/versions/20261004_0043_drop_events_collect_job.py @@ -0,0 +1,40 @@ +"""S5:删除已废弃的账号事件收集器作业行 + +收集器(`account_events_collect`)已被直发事件取代,代码连同注册表都已删除。 +但 `automation_jobs` 里可能还留着它的行 —— 调度器遇到没有注册实现的作业键会 +「跳过 + 每轮打一条 warning」(`scheduler.py` 里刻意这么设计,避免降级回滚后 +残留的行静默消失)。功能上无害,但会一直刷日志,且自动化页面上留着一个 +永远跑不动的作业。 + +本迁移把那行删掉。**不动 `automation_runs`**:那是「这个作业以前跑过什么」的 +历史记录,删掉没有收益,也没有 FK 约束会拦住它。 + +回退(downgrade)故意是空的:把作业行加回去并不会让收集器代码回来, +只会重新制造「有行没实现」的告警。要回退请回退代码,让 `ensure_jobs()` 自己 +把行补回来(它只补缺失的行)。 +""" + +from __future__ import annotations + +import sqlalchemy as sa +from alembic import op + +revision: str = "20261004_0043" +down_revision: str | None = "20261003_0042" +branch_labels: str | None = None +depends_on: str | None = None + +JOB_KEY = "account_events_collect" + + +def upgrade() -> None: + op.execute( + sa.text("DELETE FROM automation_jobs WHERE `key` = :key").bindparams( + key=JOB_KEY + ) + ) + + +def downgrade() -> None: + # 见模块 docstring:回退代码即可,这里刻意什么都不做。 + pass diff --git a/web/backend/routers/douyu.py b/web/backend/routers/douyu.py index f3d00f9..6f51f2c 100644 --- a/web/backend/routers/douyu.py +++ b/web/backend/routers/douyu.py @@ -51,6 +51,7 @@ from ..schemas import ( DouyuXpdGoodsOut, ) from ..services.audit_service import record_audit +from ..services.business_events import record_recharge from ..services.douyu_runner import DouyuBatchRunner, douyu_batch_registry from ..services.douyu_service import ( DOUYU_CONFIG_FIELDS, @@ -127,6 +128,10 @@ async def supplier_recharge_callback(request: Request, db: Session = Depends(get "supplier_callback_received": True, } ) + # 这笔充值到没到账,用**回调自己说的终态**判断,而不是「任务状态有没有被改」。 + # 否则会漏掉一个真实黑洞:runner 轮询超时先判了 failed(gold.py 末尾), + # 回调随后才到达成功 —— 守卫会跳过状态更新,这笔真金白银永远不进时间线。 + recharge_confirmed = status == 2 if task.status not in {"success", "failed", "stopped"}: if status == 2: task.status = "success" @@ -150,6 +155,22 @@ async def supplier_recharge_callback(request: Request, db: Session = Depends(get ) task.result = result callback_success = status not in {3, 4} + # 回调说成功了就补一条充值事件。幂等键与 runner 那条**完全一致** + # (business_events.task_ref),所以「runner 也观测到了」时不会重复记账; + # 而「runner 已判超时/失败、回调才到」这个黑洞则能被补上。 + if recharge_confirmed and task.account is not None: + record_recharge( + db, + task.account, + task_id=task.id, + amount=_to_int_or_none(result.get("buy_num")), + order_id=str(result.get("out_order_id") or ""), + supplier_order_id=str(result.get("order_id") or ""), + channel="supplier_api", + douyu_uid=str(result.get("douyu_uid") or ""), + balance_after=task.account.gold_balance, + confirmed=False, + ) record_audit( db, None, @@ -901,3 +922,11 @@ async def ws_douyu_logs(websocket: WebSocket, batch_id: str): latest = douyu_batch_registry.get(batch_id) if latest and latest.get("finished"): douyu_batch_registry.pop(batch_id) + + +def _to_int_or_none(value: object) -> int | None: + """回调里的 `buy_num` 可能是字符串/Decimal,安全转 int。""" + try: + return int(float(str(value))) + except (TypeError, ValueError): + return None diff --git a/web/backend/services/account_events_service.py b/web/backend/services/account_events_service.py index 25282da..350677b 100644 --- a/web/backend/services/account_events_service.py +++ b/web/backend/services/account_events_service.py @@ -5,13 +5,20 @@ `accounts` 存**当前状态**(积分、余额、开通标记),这张表存**发生过什么**。 账号详情页 = 状态卡(读 `accounts`)+ 时间线(读本表)。 -## 两条写入路径 +## 唯一写入路径:直接发射 -1. **直接发射**(`emit`):CK 失效/登录成功/重登、精英宝典与电竞手册开通 —— 这些 - 只在内存里出现过,不记就永远没有; -2. **增量收集**(`automation/jobs/account_events_collect.py`):充值/兑换/积分到账/绑定 —— - 这些已经是 `douyu_tasks` 的行,由收集器按水位线往后提取,不动 runner 代码。 - 本模块只提供 `_task_to_event` 映射给它用。 +业务事实**在它发生的那一刻**由业务代码发射(`emit` 或 `business_events` 里的命名 +构造函数)。曾经有过第二条路径 —— 一个作业事后扫 `douyu_tasks.result` 的 JSON、 +猜字段路径反推事件。那条路已经删掉,原因是它必然漏数据: + +- 靠 allowlist 认业务类型 → 新增类型没人想起维护,**静默丢**(电竞兑换、和平小店 + 兑换、锁单支付、电竞礼物积分因此从来没进过时间线); +- 「任务 success」不等于「业务真的发生了」(预约、已绑定、未确认到账都会被标 success); +- 作业默认关闭,生产上它一直没跑。 + +历史**不从 `douyu_tasks` 回填**。需要历史时按斗鱼侧权威接口同步 +(`query_exchange_records` / `query_xpd_purchase_records` / `query_points` 都是现成任务类型) +—— 宁可少,不可假。 ## 两条硬约束 @@ -230,102 +237,3 @@ def emit_bulk(db: Session, events: list[dict[str, Any]]) -> int: except Exception as exc: # noqa: BLE001 - 旁路,任何失败都不上抛 logger.warning(f"批量写入账号事件失败({len(rows)} 条):{exc}") return 0 - - -# ── douyu_tasks 增量收集 ────────────────────────────────── -# 只收集「业务事件」,例行轮询(query_*)一律不进时间线 —— 它们在 douyu_tasks 里 -# 占绝大多数(query_limited_goods 就 6.7 万行),收进去时间线没法看。 -COLLECTED_TASK_TYPES = ( - "exchange_goods", - "create_gold_qr", - "donate_elite_gift", - "confirm_bind", - "confirm_esports_bind", -) - - -def _task_to_event(task_type: str, result: Any) -> dict[str, Any] | None: - """把一条 douyu_tasks 行转成事件参数;解析不出返回 None(跳过并告警)。 - - ⚠️ 字段路径是**抽样实测**得到的(见设计文档 §4.2),不是猜的。 - 解析不出时必须跳过而不是抛出去 —— 一行坏数据不能卡住整条水位线。 - """ - if isinstance(result, str): - try: - result = json.loads(result) - except ValueError: - return None - if not isinstance(result, dict): - return None - - if task_type == "exchange_goods": - name = result.get("commodity_name") or "兑换" - score = result.get("score") - points_after = (result.get("points_query") or {}).get("points") - return { - "event_type": EVENT_EXCHANGE, - "title": f"兑换 {name}", - "amount": -int(score) if isinstance(score, (int, float)) else None, - "balance_after": points_after if isinstance(points_after, int) else None, - "detail": { - "commodity_name": name, - "commodity_id": result.get("commodity_id"), - "order_id": result.get("order_id"), - "exchange_id": result.get("exchange_id"), - "game_name": result.get("game_name"), - "game_channel": result.get("game_channel"), - "score": score, - }, - } - - if task_type == "create_gold_qr": - amount = result.get("buy_num") or result.get("pay_amount") - supplier = result.get("supplier_order") or {} - try: - amount_int = int(float(amount)) if amount is not None else None - except (TypeError, ValueError): - amount_int = None - return { - "event_type": EVENT_RECHARGE, - "title": f"充值 {amount_int if amount_int is not None else ''}".strip(), - "amount": amount_int, - "balance_after": None, - "detail": { - "order_id": result.get("order_id"), - "supplier_order_id": supplier.get("order_id"), - "supplier_finished_at": supplier.get("finish_time"), - "channel": result.get("recharge_channel"), - "douyu_uid": result.get("douyu_uid"), - }, - } - - if task_type == "donate_elite_gift": - gain = result.get("gift_points_expected_gain") - points = result.get("points") - will_expire = ((result.get("points_query") or {}).get("raw") or {}).get("data") - expire_at = ( - will_expire.get("willExpireTime") if isinstance(will_expire, dict) else None - ) - return { - "event_type": EVENT_POINTS_CREDIT, - "title": f"积分到账 +{gain}" if gain is not None else "积分到账", - "amount": int(gain) if isinstance(gain, (int, float)) else None, - "balance_after": int(points) if isinstance(points, (int, float)) else None, - "detail": { - "baseline": result.get("gift_points_baseline"), - "target": result.get("gift_points_target"), - "points_expire_at": expire_at, - "gold_balance": result.get("gold_balance"), - }, - } - - if task_type in ("confirm_bind", "confirm_esports_bind"): - return { - "event_type": EVENT_BIND_OK, - "title": "绑定角色" if task_type == "confirm_bind" else "电竞手册绑定", - "amount": None, - "balance_after": None, - "detail": {"raw_keys": sorted(result.keys())[:20]}, - } - - return None diff --git a/web/backend/services/automation/__init__.py b/web/backend/services/automation/__init__.py index 5afe2bd..31bd25c 100644 --- a/web/backend/services/automation/__init__.py +++ b/web/backend/services/automation/__init__.py @@ -6,12 +6,16 @@ from __future__ import annotations -from .jobs.account_events_collect import JOB_KEY as ACCOUNT_EVENTS_KEY -from .jobs.account_events_collect import run_account_events_collect from .jobs.ck_health import JOB_KEY as CK_HEALTH_KEY from .jobs.ck_health import run_ck_health from .jobs.ck_relogin import JOB_KEY as CK_RELOGIN_KEY from .jobs.ck_relogin import run_ck_relogin +from .jobs.douyu_tasks_retention import ( + JOB_KEY as DOUYU_TASKS_RETENTION_KEY, +) +from .jobs.douyu_tasks_retention import ( + run_douyu_tasks_retention, +) from .scheduler import ( JobSpec, automation_enabled, @@ -27,9 +31,9 @@ from .scheduler import ( ) __all__ = [ - "ACCOUNT_EVENTS_KEY", "CK_HEALTH_KEY", "CK_RELOGIN_KEY", + "DOUYU_TASKS_RETENTION_KEY", "JobSpec", "automation_enabled", "ensure_jobs", @@ -37,9 +41,9 @@ __all__ = [ "register_job", "registered_jobs", "reset_registry", - "run_account_events_collect", "run_ck_health", "run_ck_relogin", + "run_douyu_tasks_retention", "run_due_jobs", "run_job_now", "scheduler_status", @@ -50,6 +54,17 @@ __all__ = [ def register_default_jobs() -> None: """登记内置作业(幂等)。""" + register_job( + JobSpec( + key=DOUYU_TASKS_RETENTION_KEY, + title="任务日志清理", + # 一天一次足够:这是**清理**,不是业务;跑太勤只是白扫索引。 + interval_seconds=24 * 3600, + batch_size=2000, + concurrency=1, + run=run_douyu_tasks_retention, + ) + ) register_job( JobSpec( key=CK_RELOGIN_KEY, @@ -75,15 +90,3 @@ def register_default_jobs() -> None: run=run_ck_health, ) ) - register_job( - JobSpec( - key=ACCOUNT_EVENTS_KEY, - title="账号事件收集", - # interval_seconds 这里无实际意义(收集器每轮只按水位线往后取, - # 取不到就空转),保留一个值以复用现有调度与页面。 - interval_seconds=10 * 60, - batch_size=500, - concurrency=1, - run=run_account_events_collect, - ) - ) diff --git a/web/backend/services/automation/jobs/account_events_collect.py b/web/backend/services/automation/jobs/account_events_collect.py deleted file mode 100644 index ae22b1f..0000000 --- a/web/backend/services/automation/jobs/account_events_collect.py +++ /dev/null @@ -1,135 +0,0 @@ -"""作业:把 douyu_tasks 里的业务操作增量收集成账号事件。 - -为什么用「收集」而不是在 runner 里直接发事件:充值/兑换/积分到账这些**已经是 -`douyu_tasks` 的行**,再去改 8 个业务文件属于重复劳动;收集器只读不写业务表, -零侵入,且新老任务类型统一处理。 - -**不回填历史**(2026-10-03 已定):首次运行把水位线定在当前最大 id, -只往后收集。既有历史仍去各自页面查。 - -水位线存在本作业的 `automation_jobs.config` 里,进程重启不丢。 -""" - -from __future__ import annotations - -import json -from typing import Any - -from loguru import logger -from sqlalchemy import text -from sqlalchemy.orm import Session - -from ....models import AutomationJob -from ...account_events_service import ( - COLLECTED_TASK_TYPES, - _task_to_event, - emit, -) - -JOB_KEY = "account_events_collect" -WATERMARK_FIELD = "watermark" -DEFAULT_BATCH = 500 - - -def _load_config(job: AutomationJob) -> dict[str, Any]: - config = job.config - if isinstance(config, str): - try: - config = json.loads(config) - except ValueError: - config = {} - return dict(config) if isinstance(config, dict) else {} - - -def _save_watermark(db: Session, job: AutomationJob, watermark: int) -> None: - config = _load_config(job) - config[WATERMARK_FIELD] = int(watermark) - job.config = config - db.commit() - - -def run_account_events_collect(db: Session, job: AutomationJob) -> dict[str, Any]: - config = _load_config(job) - watermark = config.get(WATERMARK_FIELD) - - if not isinstance(watermark, int): - # 首次运行:水位线定在「当前最大 id」= 从此刻起往后收集,**不回溯历史**。 - max_id = db.execute( - text("SELECT COALESCE(MAX(id), 0) FROM douyu_tasks") - ).scalar() - watermark = int(max_id or 0) - _save_watermark(db, job, watermark) - logger.info( - f"账号事件收集:首次运行,水位线定在 douyu_tasks.id={watermark}(不回溯)" - ) - return { - "scanned": 0, - "skipped": 0, - "detail": {"written": 0, "watermark": watermark, "first_run": True}, - "message": f"首次运行,水位线定在 {watermark}(不回溯历史)", - } - - batch_size = max(int(job.batch_size or DEFAULT_BATCH), 1) - placeholders = ",".join(f":t{i}" for i in range(len(COLLECTED_TASK_TYPES))) - params: dict[str, Any] = {"wm": watermark, "limit": batch_size} - params.update({f"t{i}": t for i, t in enumerate(COLLECTED_TASK_TYPES)}) - - rows = ( - db.execute( - text( - f""" - SELECT id, account_id, task_type, status, result, created_at, finished_at - FROM douyu_tasks - WHERE id > :wm AND task_type IN ({placeholders}) - ORDER BY id ASC - LIMIT :limit - """ - ), - params, - ) - .mappings() - .all() - ) - - written = skipped = failed = 0 - max_id = watermark - for row in rows: - max_id = max(max_id, int(row["id"])) - # 失败的任务不产生业务事实:兑换失败不该记「消耗了积分」。 - if (row["status"] or "") != "success": - skipped += 1 - continue - payload = _task_to_event(row["task_type"], row["result"]) - if payload is None: - # 解析不出就跳过并告警,绝不让一行坏数据卡住整条水位线。 - skipped += 1 - logger.warning( - f"账号事件收集:无法解析 douyu_task {row['id']}({row['task_type']})" - ) - continue - ok = emit( - db, - account_id=int(row["account_id"]), - occurred_at=row["finished_at"] or row["created_at"], - source="task", - source_ref=f"douyu_task:{row['id']}", - **payload, - ) - if ok: - written += 1 - else: - failed += 1 - - if rows: - _save_watermark(db, job, max_id) - - return { - "scanned": len(rows), - "skipped": skipped, - "detail": { - "written": written, - "failed": failed, - "watermark": max_id, - }, - "message": f"本轮收集 {written} 条事件,跳过 {skipped} 条,水位线 {max_id}", - } diff --git a/web/backend/services/automation/jobs/douyu_tasks_retention.py b/web/backend/services/automation/jobs/douyu_tasks_retention.py new file mode 100644 index 0000000..1e2a87d --- /dev/null +++ b/web/backend/services/automation/jobs/douyu_tasks_retention.py @@ -0,0 +1,207 @@ +"""作业:清理 `douyu_tasks` 的历史任务日志。 + +## 为什么需要 + +`douyu_tasks` 是**执行日志**,不是账号历史。它按天增长(生产实测 222,679 行 / +687MB,且绝大多数是例行查询),而其中真正有长期价值的业务事实,S5 之后都会 +由业务代码在发生那一刻写进 `account_events`(体量小几个数量级)。 + +所以这里只做一件事:**按任务类型的价值分档,把过期的执行日志删掉。** + +## 分档依据(关键设计) + +不是「按时间一刀切」,而是按「这条日志里的 `result` 还有没有人需要」分三档: + +| 档 | 判定 | 默认保留 | 说明 | +| --- | --- | --- | --- | +| **噪声查询** | 只读、看一眼就走,且前端**不**用它的 `result` 派生账号状态 | 7 天 | 体量最大的就是这类(`query_limited_goods` 单类 6.7 万行) | +| **状态派生** | 前端从任务列表反推账号当前状态时要读它(`latest*ByAccount`) | 90 天 | 删早了工作台账号行的「已绑定/未绑定」会变成「未查」 | +| **业务事实** | 真花钱/花积分/改变账号归属的动作 | 90 天 | 90 天后仍有 `account_events` 兜底 | + +⚠️ **为什么「状态派生」类不能按 7 天删**:工作台是从任务列表**反推**账号状态的 +(`DouyuTasksPage` 的 `latestConfirmBindTaskByAccount` 等),而且明确要求 +`task.result` 非空。这是历史包袱(和 `bind_status` 同源:账号的「现在是什么」 +存在执行日志里),设计文档 §3.6 的状态列拆解才是根治。在拆解完成之前, +这里保守地把这类查询留长一点 —— 否则会把「当前显示」搞坏,那不是在清历史。 + +两类窗口都可以在 `/automation` 页面按作业改(`config.noise_days` / +`config.business_days`),不必发版。 + +## 安全约束 + +- **绝不删在途任务**(`planned` / `pending` / `running`)—— 批次线程可能正拿着它; +- 分批删(每批 `batch_size` 行、最多 `max_batches` 批),避免长事务长时间锁表; +- 不删行、只删「过期的整行」:不做「把 result 清空保留行」这种半吊子操作, + 因为部分前端派生**明确要求 result 非空**,清空等于制造「有行但读不出状态」的 + 更隐蔽的坏状态; +- 作业**默认关闭**(`ensure_jobs` 建行时 enabled=False),由运维在页面上打开。 +""" + +from __future__ import annotations + +import json +from datetime import UTC, datetime, timedelta +from typing import Any + +from loguru import logger +from sqlalchemy import text +from sqlalchemy.orm import Session + +from ....models import AutomationJob + +JOB_KEY = "douyu_tasks_retention" + +#: 只读、看一眼就走,且前端**不**拿它派生账号状态 → 短保留。 +#: 体量最大的就在这一类(`query_limited_goods` / `query_change_bind_time`)。 +NOISE_TASK_TYPES: tuple[str, ...] = ( + "query_limited_goods", + "query_change_bind_time", + "query_gold_balance", + "query_xpd_balance", + "query_xpd_fragments", + "query_points", + "refresh_goods", + "refresh_esports_goods", + "refresh_xpd_goods", + "prefetch_csrf_token", +) + +DEFAULT_NOISE_DAYS = 7 +DEFAULT_BUSINESS_DAYS = 90 +DEFAULT_BATCH = 2000 +DEFAULT_MAX_BATCHES = 50 + +#: 在途任务绝不可删:批次线程正拿着这些行。 +IN_FLIGHT_STATUSES = ("planned", "pending", "running") + + +def _load_config(job: AutomationJob) -> dict[str, Any]: + config = job.config + if isinstance(config, str): + try: + config = json.loads(config) + except ValueError: + config = {} + return dict(config) if isinstance(config, dict) else {} + + +def _positive_int(value: Any, default: int) -> int: + try: + parsed = int(value) + except (TypeError, ValueError): + return default + return parsed if parsed > 0 else default + + +def _delete_batch( + db: Session, + *, + task_types: tuple[str, ...], + cutoff: datetime, + batch: int, + negate: bool = False, +) -> int: + """删一批过期行,返回删除条数。 + + `negate=True` 时语义变成「**除这些类型之外**的都在这一档」—— 长保留档就是这么 + 表达的。这样做有个重要的安全属性:**新增的任务类型默认落在长保留档**, + 不会因为「忘了往某个列表里加」而在 7 天后被误删(收集器当年就是反过来, + 忘了加 allowlist 就静默丢数据)。 + + `LIMIT` 让单条语句不会长时间持锁。 + """ + if not task_types: + return 0 + placeholders = ", ".join(f":t{i}" for i in range(len(task_types))) + status_placeholders = ", ".join(f":s{i}" for i in range(len(IN_FLIGHT_STATUSES))) + operator = "NOT IN" if negate else "IN" + params: dict[str, Any] = { + "cutoff": cutoff, + "limit": batch, + **{f"t{i}": t for i, t in enumerate(task_types)}, + **{f"s{i}": s for i, s in enumerate(IN_FLIGHT_STATUSES)}, + } + result = db.execute( + text( + f""" + DELETE FROM douyu_tasks + WHERE task_type {operator} ({placeholders}) + AND status NOT IN ({status_placeholders}) + AND created_at < :cutoff + LIMIT :limit + """ + ), + params, + ) + return int(result.rowcount or 0) + + +def run_douyu_tasks_retention(db: Session, job: AutomationJob) -> dict[str, Any]: + config = _load_config(job) + noise_days = _positive_int(config.get("noise_days"), DEFAULT_NOISE_DAYS) + business_days = _positive_int(config.get("business_days"), DEFAULT_BUSINESS_DAYS) + batch = _positive_int(job.batch_size, DEFAULT_BATCH) + max_batches = _positive_int(config.get("max_batches"), DEFAULT_MAX_BATCHES) + + now = datetime.now(UTC) + # 传 naive UTC:created_at 是 UTCDateTime 列,裸 SQL 参数不经过类型转换, + # 带 tzinfo 会被驱动按本地时区处理(参见设计文档里那次 8 小时偏移事故)。 + noise_cutoff = (now - timedelta(days=noise_days)).replace(tzinfo=None) + business_cutoff = (now - timedelta(days=business_days)).replace(tzinfo=None) + + total_deleted = 0 + batches = 0 + + # 两档:噪声查询只留 noise_days;**其余一切**(业务事实 + 前端状态派生依赖的 + # 查询 + 以后新增的类型)留 business_days。 + passes = ( + (noise_cutoff, NOISE_TASK_TYPES, False, "noise"), + (business_cutoff, NOISE_TASK_TYPES, True, "rest"), + ) + per_pass: dict[str, int] = {} + for cutoff, task_types, negate, label in passes: + deleted_this_pass = 0 + while batches < max_batches: + deleted = _delete_batch( + db, + task_types=task_types, + cutoff=cutoff, + batch=batch, + negate=negate, + ) + batches += 1 + total_deleted += deleted + deleted_this_pass += deleted + db.commit() + if deleted < batch: + break + per_pass[label] = deleted_this_pass + + # 注意措辞:这只说明「批次预算用光了」,**不等于**「确实还有积压」—— + # 预算在上一档用完时,下一档根本没轮到执行,这里无法区分「没得删」和「没轮到」。 + # 所以做成提示而非结论,措辞上也别谎称「仍有积压」。 + batch_budget_exhausted = batches >= max_batches + if batch_budget_exhausted: + logger.warning( + "任务日志清理用光了批次预算({} 批,共删 {} 条);" + "若日志量持续很大,可调大 max_batches 或 batch_size", + max_batches, + total_deleted, + ) + + message = ( + f"清理 {total_deleted} 条任务日志" + f"(噪声>{noise_days}天 / 其余>{business_days}天,{batches} 批)" + ) + logger.info(message) + return { + "scanned": total_deleted, + "deleted": total_deleted, + "deleted_noise": per_pass.get("noise", 0), + "deleted_rest": per_pass.get("rest", 0), + "batches": batches, + "noise_days": noise_days, + "business_days": business_days, + "batch_budget_exhausted": batch_budget_exhausted, + "message": message, + } diff --git a/web/backend/services/business_events.py b/web/backend/services/business_events.py new file mode 100644 index 0000000..0200ca0 --- /dev/null +++ b/web/backend/services/business_events.py @@ -0,0 +1,251 @@ +"""业务事实的**命名构造**:充值 / 兑换 / 积分到账 / 绑定。 + +## 为什么单独一个模块 + +`emit()` 只负责「往时间线写一条」,它不知道「一次兑换」到底该长什么样。 +以前这件事由收集器从 `douyu_tasks.result` 的 JSON 里**反推**(猜字段路径), +结果是: + +- 新增业务类型没人想起改 allowlist → **静默丢数据** + (电竞兑换、和平小店兑换、锁单支付、电竞礼物积分从来没进过时间线); +- 同一件事的字段口径散在 runner 与收集器两处,容易分叉; +- `success` 不等于「业务真的发生了」时,把假成功也记成事实。 + +现在反过来:**业务代码在动作成功的那一刻,用这里的构造函数声明事实**。 +每类事实的字段、标题、金额符号只定义一次。 + +## 两条硬规则 + +1. **必须紧贴 `_mark_task(..., "success", ...)` 之前调用。** + `_mark_task` 内部 `db.commit()`,而 `emit()` 用 `db.begin_nested()` 只 RELEASE + SAVEPOINT、不提交外层事务;`douyu_runner` 的 `finally` 只 `close()`(未提交即回滚)。 + 在 `_mark_task` **之后**发事件会被静默回滚 —— 而且 `emit()` 返回 True、无任何日志。 + 放在之前还有个好处:事件与任务状态**同一次提交**,原子。 + +2. **幂等键用 `task_ref(task.id)`**,与旧收集器的 key 完全一致。 + 于是迁移窗口内「直发」与「仍在跑的收集器」会自动去重,可以安全并存后再删作业。 + ⚠️ 不要用秒级时间戳当幂等键:撞唯一键会让 `emit()` 静默吞掉事件 + (见 `account_lifecycle` 里那段复盘)。 +""" + +from __future__ import annotations + +from datetime import datetime +from typing import Any + +from sqlalchemy.orm import Session + +from ..models import Account +from .account_events_service import ( + CATEGORY_BUSINESS, + EVENT_BIND_OK, + EVENT_EXCHANGE, + EVENT_POINTS_CREDIT, + EVENT_RECHARGE, + emit, +) + + +def task_ref(task_id: int) -> str: + """业务事件的幂等键:一条任务行对应一条事实。 + + 与旧收集器(`f"douyu_task:{row['id']}"`)**完全一致**,所以直发与收集器 + 同时观测到同一笔业务时只会写一条 —— 这让「先直发、观察一段、再删收集器」 + 成为安全的迁移路径。 + """ + return f"douyu_task:{int(task_id)}" + + +def record_recharge( + db: Session, + account: Account, + *, + task_id: int, + amount: int | None, + occurred_at: datetime | None = None, + order_id: str = "", + supplier_order_id: str = "", + channel: str = "", + douyu_uid: str = "", + balance_after: int | None = None, + confirmed: bool = True, +) -> bool: + """充值鱼翅到账。 + + `amount` 用**本地已知的充值金额**,不要从 `result` 里找 —— 二维码链路的 + `result` 里根本没有金额(`create_gold_qr` 只返回 `pay_url/token/raw`), + 旧收集器因此在二维码充值上恒写 `amount=None`,导致 `accounts.recharge_total` + 系统性少计。 + + `confirmed=False` 表示只有供应商的终态回执、没有核对斗鱼余额变化。 + """ + title = f"充值 {amount}" if amount is not None else "充值到账" + if not confirmed: + title += "(供应商回执)" + return emit( + db, + account_id=account.id, + event_type=EVENT_RECHARGE, + title=title, + category=CATEGORY_BUSINESS, + occurred_at=occurred_at, + amount=amount, + balance_after=balance_after, + detail={ + "order_id": order_id, + "supplier_order_id": supplier_order_id, + "channel": channel, + "douyu_uid": douyu_uid, + "confirmed": confirmed, + }, + source="douyu_task", + subject_type="recharge", + subject_key=order_id or supplier_order_id or "", + source_ref=task_ref(task_id), + ) + + +def record_exchange( + db: Session, + account: Account, + *, + task_id: int, + name: str, + score: int | None, + occurred_at: datetime | None = None, + commodity_id: str = "", + order_id: str = "", + exchange_id: str = "", + game_name: str = "", + game_channel: str = "", + points_after: int | None = None, + scope: str = "elite", +) -> bool: + """商品/皮肤兑换成功 —— **只在真的消耗了积分并兑换成功时调用**。 + + 注意「预约到货 / 已预兑 / 预兑确认」这三类也把任务标成 `success`,但它们 + **不消耗积分、也没兑换到东西**,绝不能走这里(旧收集器只按 `status=="success"` + 过滤,于是每预约一次就生成一条金额为空的 `exchange`,把 `exchange_count` 灌大)。 + + `score` 为负写入 `amount`(消耗),`points_after` 是兑换后的积分余额。 + """ + return emit( + db, + account_id=account.id, + event_type=EVENT_EXCHANGE, + title=f"兑换 {name}" if name else "兑换", + category=CATEGORY_BUSINESS, + occurred_at=occurred_at, + amount=-int(score) if isinstance(score, (int, float)) else None, + balance_after=points_after, + detail={ + "commodity_id": commodity_id, + "commodity_name": name, + "order_id": order_id, + "exchange_id": exchange_id, + "game_name": game_name, + "game_channel": game_channel, + "score": score, + "scope": scope, + }, + source="douyu_task", + subject_type="commodity", + subject_key=commodity_id or name or "", + source_ref=task_ref(task_id), + ) + + +def record_points_credit( + db: Session, + account: Account, + *, + task_id: int, + gain: int | None, + occurred_at: datetime | None = None, + points_after: int | None = None, + expire_at: Any = None, + confirmed: bool = True, + source_label: str = "精英令", +) -> bool: + """积分到账(送礼/任务发放)。 + + `confirmed` 必须如实反映「有没有真的看到积分涨上去」。送礼任务在**没确认** + 到账时也会把任务标成 `success`(消息里加一句「(未确认涨到 X)」),旧收集器 + 照样把 `gift_points_expected_gain` 当成事实写进台账 —— 把「预期」记成了「已到账」。 + + 这里保留事件(延迟到账也是真实发生的发放),但**把 `confirmed` 明确落进 + detail**:宁可标清楚是未确认,也不假装它是确认过的数字。 + """ + title = f"积分到账 +{gain}" if gain is not None else "积分到账" + title = f"{source_label}{title}" + if not confirmed: + title += "(未确认)" + return emit( + db, + account_id=account.id, + event_type=EVENT_POINTS_CREDIT, + title=title, + category=CATEGORY_BUSINESS, + occurred_at=occurred_at, + amount=int(gain) if isinstance(gain, (int, float)) else None, + balance_after=points_after, + detail={ + "confirmed": confirmed, + "expire_at": expire_at, + "source": source_label, + }, + source="douyu_task", + subject_type="points", + source_ref=task_ref(task_id), + ) + + +def record_bind( + db: Session, + account: Account, + *, + task_id: int, + scope: str, + role_name: str = "", + occurred_at: datetime | None = None, + act_alias: str = "", + rebind: bool = False, + weak: bool = False, +) -> bool: + """角色绑定 / 换绑**真的发生了**。 + + ⚠️ 只在「确认接口成功」时调用。`confirm_bind` 里还有一条分支是 + 「检测到账号**本来就已经绑定**」——它什么都没做也把任务标成 `success`, + 旧收集器把每次重复点「确认绑定」都记成一条 `bind_ok`。 + + `weak=True` 表示确认接口成功、但回查拿不到结果(没能亲眼确认生效), + 仍然记,但标出来。 + """ + label = "换绑" if rebind else "绑定" + scope_label = {"elite": "精英宝典", "esports": "电竞手册", "peace": "和平小店"}.get( + scope, scope + ) + title = f"{scope_label}{label}" + if role_name: + title += f":{role_name}" + if weak: + title += "(未回查确认)" + return emit( + db, + account_id=account.id, + event_type=EVENT_BIND_OK, + title=title, + category=CATEGORY_BUSINESS, + occurred_at=occurred_at, + detail={ + "scope": scope, + "role_name": role_name, + "act_alias": act_alias, + "rebind": rebind, + "confirmed": not weak, + }, + source="douyu_task", + subject_type=f"{scope}_role", + subject_key=role_name or "", + source_ref=task_ref(task_id), + ) diff --git a/web/backend/services/douyu_runner_bind.py b/web/backend/services/douyu_runner_bind.py index 1923c3f..ab6d4aa 100644 --- a/web/backend/services/douyu_runner_bind.py +++ b/web/backend/services/douyu_runner_bind.py @@ -11,6 +11,7 @@ from sqlalchemy.orm import Session from core.douyu import DouyuActivityClient, DouyuActivityError from ..models import Account, DouyuTask +from .business_events import record_bind DOUYU_LEGACY_BIND_ACT_ALIAS = "20250213NQCYX" DOUYU_BIND_ROLE_POLL_SECONDS = 65 @@ -918,6 +919,18 @@ class BindMixin: else: account.bind_status = "bind_confirmed" account.updated_at = datetime.now(UTC) + # 确认接口成功了,但回查抛异常 —— 没能亲眼看到绑定生效。 + # 仍然记(确认请求确实发出且成功),但 weak=True 标出来。 + record_bind( + db, + account, + task_id=task.id, + scope="elite", + role_name=role_name, + act_alias=use_confirm_alias, + rebind=_bind_is_rebind(before_snapshot), + weak=True, + ) self._mark_task( db, task, @@ -941,6 +954,18 @@ class BindMixin: after_snapshot = self._bind_snapshot(after) final_role_name = after_snapshot["role_name"] or role_name self._apply_bind_info_to_account(account, after, "bind_confirmed") + # 强事实:确认接口成功 + 回查命中已生效绑定。 + # ⚠️ 上面还有一条 success 是「检测到账号**本来就已经绑定**」(什么都没做), + # 那条**不发事件** —— 否则用户每点一次「确认绑定」时间线就多一条假的绑定记录。 + record_bind( + db, + account, + task_id=task.id, + scope="elite", + role_name=final_role_name, + act_alias=use_confirm_alias, + rebind=_bind_is_rebind(after_snapshot), + ) self._mark_task( db, task, @@ -1089,11 +1114,23 @@ class BindMixin: return self._apply_esports_bind_info_to_account(account, after_state, "esports_bound") + # 业务事实在这一刻成立:确认接口成功 + 回查 esports_bound 为真。 + # 必须在 _mark_task 之前发 —— 它内部 commit,而 emit 只 RELEASE SAVEPOINT、 + # 不提交外层事务,runner 的 finally 只 close(未提交即回滚),之后发会被静默丢弃。 + role_text = self._esports_role_text(after_state) or "" + record_bind( + db, + account, + task_id=task.id, + scope="esports", + role_name=role_text, + act_alias=act_alias, + ) self._mark_task( db, task, "success", - f"电竞手册绑定成功: {self._esports_role_text(after_state) or '-'}", + f"电竞手册绑定成功: {role_text or '-'}", result, ) @@ -1333,3 +1370,16 @@ class BindMixin: f"{role_name} {status_text} | act={source_alias or '-'}", result, ) + + +def _bind_is_rebind(snapshot: dict) -> bool: + """这次是「换绑」还是「首次绑定」。 + + `before_bound` 命中说明确认前**该账号已有一个生效绑定**,本次是把角色换掉; + 否则是首次绑定。旧收集器不区分,时间线上看不出「换绑」这件事。 + """ + return bool( + snapshot.get("bound_act") + or snapshot.get("bind_confirmed") + or snapshot.get("role_name") + ) diff --git a/web/backend/services/douyu_runner_donate.py b/web/backend/services/douyu_runner_donate.py index 26ba8d2..d06c3d6 100644 --- a/web/backend/services/douyu_runner_donate.py +++ b/web/backend/services/douyu_runner_donate.py @@ -10,6 +10,7 @@ from sqlalchemy.orm import Session from core.douyu import DouyuActivityClient from ..models import Account, DouyuTask +from .business_events import record_points_credit DOUYU_GIFT_POINTS_REFRESH_TIMES = 3 DOUYU_GIFT_POINTS_REFRESH_INTERVAL = 2 @@ -178,11 +179,23 @@ class DonateMixin: account.esports_bind_status = "esports_gift_donated" account.updated_at = datetime.now(UTC) + confirmed = bool(result.get("esports_points_changed")) message = f"赠送{gift_name}成功: {gift_count}" if account.gold_balance is not None: message += f",鱼翅余额: {account.gold_balance}" if account.esports_points is not None: message += f",电竞积分: {account.esports_points}" + # 电竞礼物同样产生积分到账 —— 这一类此前**从来不在收集范围** + # (不在 COLLECTED_TASK_TYPES 里),时间线上一直是空白。 + record_points_credit( + db, + account, + task_id=task.id, + gain=_esports_gift_points(result, baseline_points), + points_after=account.esports_points, + confirmed=confirmed, + source_label=gift_name, + ) self._mark_task(db, task, "success", message, result) def _execute_donate_esports_chicken_gift( @@ -277,10 +290,53 @@ class DonateMixin: message = f"赠送精英令成功: {gift_count}" if account.gold_balance is not None: message += f",鱼翅余额: {account.gold_balance}" + confirmed = bool(result.get("gift_points_confirmed")) if account.points is not None: message += f",积分: {account.points}" - if result.get("gift_points_target") is not None and not result.get( - "gift_points_confirmed" - ): + if result.get("gift_points_target") is not None and not confirmed: message += f"(未确认涨到 {result['gift_points_target']})" + # 积分到账是业务事实。**未确认也记,但把 confirmed 明确写进 detail** —— + # 旧收集器把「预期涨幅」当既成事实写进台账(未确认时也记 +980), + # 那是把预期记成到账。这里宁可标清楚,也不假装它是确认过的数字。 + # 金额用本地可等价重算的值(1 个精英令 ≈ 10 分),不再依赖 result 里的字段。 + record_points_credit( + db, + account, + task_id=task.id, + gain=max(0, gift_count) * 10, + points_after=account.points, + expire_at=_points_expire_at(result), + confirmed=confirmed, + source_label="精英令", + ) self._mark_task(db, task, "success", message, result) + + +def _points_expire_at(result: dict) -> object: + """从积分查询结果里取出到期时间(同样是为了给时间线留全信息)。 + + 旧收集器走的是 `result["points_query"]["raw"]["data"]["willExpireTime"]` 这条 + 「猜出来」的路径;这里收敛成一个具名函数,只有它知道那段结构。 + """ + query = result.get("points_query") + if not isinstance(query, dict): + return None + raw = query.get("raw") + data = raw.get("data") if isinstance(raw, dict) else None + return data.get("willExpireTime") if isinstance(data, dict) else None + + +def _esports_gift_points(result: dict, baseline_points: int | None) -> int | None: + """电竞礼物带来的积分增量。 + + 以**实际观测到的积分变化**为准(`esports_points_after_gift` - 基线), + 而不是像旧收集器那样去读一个「预期涨幅」。观测不到就返回 None —— + 金额宁可空着,也不要写一个猜出来的数字进台账。 + """ + after = result.get("esports_points_after_gift") + if not isinstance(after, (int, float)) or not isinstance( + baseline_points, (int, float) + ): + return None + delta = int(after) - int(baseline_points) + return delta if delta > 0 else None diff --git a/web/backend/services/douyu_runner_gold.py b/web/backend/services/douyu_runner_gold.py index c3b5850..29b0663 100644 --- a/web/backend/services/douyu_runner_gold.py +++ b/web/backend/services/douyu_runner_gold.py @@ -18,6 +18,7 @@ from core.douyu import ( ) from ..models import Account, DouyuTask +from .business_events import record_recharge from .douyu_runner_core import DOUYU_PAYMENT_POLL_INTERVAL, DOUYU_PAYMENT_POLL_SECONDS from .douyu_service import update_account_profile_from_cookie @@ -92,8 +93,12 @@ class GoldMixin: if self._stop.wait(DOUYU_PAYMENT_POLL_INTERVAL): break continue + # ⚠️ 判据必须是 `>`(余额**增加**)而不是 `!=`。 + # 原来用 `!=`,于是余额**下降**也算「到账」:批内并发(concurrency + # 最高 10)时同账号的送礼任务会消耗鱼翅,余额一变就被这个充值任务 + # 读成「鱼翅已到账」,往台账里记一笔根本没发生的充值。 changed = last_gold is not None and ( - baseline_gold is None or last_gold != baseline_gold + baseline_gold is None or last_gold > baseline_gold ) if changed: result["payment_polling"] = False @@ -279,6 +284,18 @@ class GoldMixin: if recharged: account.bind_status = "gold_recharged" account.updated_at = datetime.now(UTC) + # 金额用**本地变量 amount**:二维码链路的 result 里没有金额 + # (create_gold_qr 只返回 pay_url/token/raw),旧收集器在这里恒写 + # amount=None,导致 accounts.recharge_total 系统性少计。 + record_recharge( + db, + account, + task_id=task.id, + amount=amount, + channel=channel, + balance_after=account.gold_balance, + confirmed=True, + ) self._mark_task( db, task, @@ -426,6 +443,22 @@ class GoldMixin: if status == 2: account.bind_status = "gold_recharged" account.updated_at = datetime.now(UTC) + # 幂等键与**供应商回调**(routers/douyu.py)完全一致 —— 两条路径都可能 + # 观测到同一笔成功(_wait_supplier_gold_order 会采纳回调结果再 mark 一次), + # 同一个 source_ref 天然去重。 + record_recharge( + db, + account, + task_id=task.id, + amount=amount, + order_id=str(result.get("out_order_id") or ""), + supplier_order_id=str(result.get("order_id") or ""), + channel="supplier_api", + douyu_uid=str(result.get("douyu_uid") or ""), + balance_after=account.gold_balance, + # 供应商终态成功 ≠ 斗鱼余额已变,如实标注 + confirmed=False, + ) self._mark_task(db, task, "success", "供应商直充成功", result) return if status in {3, 4}: diff --git a/web/backend/services/douyu_runner_goods.py b/web/backend/services/douyu_runner_goods.py index 0a3f477..8e5e9d8 100644 --- a/web/backend/services/douyu_runner_goods.py +++ b/web/backend/services/douyu_runner_goods.py @@ -12,6 +12,7 @@ from sqlalchemy.orm import Session from core.douyu import DouyuActivityClient, DouyuActivityError from ..models import Account, DouyuEsportsGoodsSnapshot, DouyuGoodsSnapshot, DouyuTask +from .business_events import record_exchange # 兑换节奏与重试(对齐 8.30 浏览器抓包:锁单->支付间隔约 2.2~4.2s;火爆类错误要长退避而不是秒级连打) DOUYU_EXCHANGE_PRE_CREATE_JITTER = (0.3, 1.2) @@ -204,6 +205,29 @@ class GoodsMixin: ) except Exception as exc: # noqa: BLE001 外部接口与任务边界需要保留宽泛异常兜底 self._push_log("warning", f"支付锁单后刷新积分失败: {exc}") + # 两步式兑换(lock_goods → pay_locked_order)的**支付**这一步才是真兑换。 + # 这一步此前从不发事件,时间线上只有「锁单」没有「支付」。 + # `pay_exchange_order` 的响应里没有 score,用商品快照上的标价。 + score = None + if goods is not None and isinstance(goods.score, (int, float)): + num = payment.get("exchange_num") + score = int(goods.score) * ( + int(num) if isinstance(num, (int, float)) else 1 + ) + record_exchange( + db, + account, + task_id=task.id, + name=(goods.name if goods else "") or commodity_id or order_id, + score=score, + commodity_id=commodity_id, + order_id=order_id, + exchange_id=str(payment.get("exchange_id") or ""), + game_name=account.game_name or "", + game_channel=account.game_channel or "", + points_after=account.points, + scope="elite", + ) self._mark_task( db, task, @@ -495,6 +519,24 @@ class GoodsMixin: ) except Exception as exc: # noqa: BLE001 外部接口与任务边界需要保留宽泛异常兜底 self._push_log("warning", f"兑换后刷新积分失败: {exc}") + # ⚠️ **只有这一条 success 是真的兑换**。同一个 handler 里还有三条 + # 「预约到货 / 已预兑 / 预兑确认」的 success —— 它们不消耗积分、也没换到东西, + # 所以**不发事件**。旧收集器只按 status=="success" 收集,导致每预约一次 + # 就生成一条金额为空的 exchange,把 accounts 的 exchange_count 灌大。 + record_exchange( + db, + account, + task_id=task.id, + name=name, + score=locked.get("score"), + commodity_id=commodity_id, + order_id=str(locked.get("order_id") or ""), + exchange_id=str(payment.get("exchange_id") or ""), + game_name=account.game_name or "", + game_channel=account.game_channel or "", + points_after=(points_refresh or {}).get("points"), + scope="elite", + ) self._mark_task( db, task, @@ -587,6 +629,28 @@ class GoodsMixin: result["account_name"] = ( account.nickname or account.username or account.uid or f"#{account.id}" ) + # 电竞皮肤兑换 —— **此前从不发事件**(不在旧收集器的 allowlist 里), + # 时间线上一直是空白。 + # 消耗的电竞积分用「兑换前后积分差」实测,而不是读猜测的字段。 + after_points = result.get("esports_points_after_exchange") + spent = ( + int(baseline_points) - int(after_points) + if isinstance(baseline_points, (int, float)) + and isinstance(after_points, (int, float)) + else None + ) + record_exchange( + db, + account, + task_id=task.id, + name=name, + score=spent if spent and spent > 0 else None, + commodity_id=commodity_id, + game_name=account.esports_game_name or "", + game_channel=account.esports_game_channel or "", + points_after=account.esports_points, + scope="esports", + ) self._mark_task(db, task, "success", message, result) def _execute_query_limited_goods( diff --git a/web/backend/services/douyu_runner_xpd.py b/web/backend/services/douyu_runner_xpd.py index 68a90d7..dcce16a 100644 --- a/web/backend/services/douyu_runner_xpd.py +++ b/web/backend/services/douyu_runner_xpd.py @@ -11,6 +11,7 @@ from sqlalchemy.orm import Session from core.douyu import DouyuActivityClient, DouyuActivityError from ..models import Account, DouyuTask, DouyuXpdGoodsSnapshot +from .business_events import record_exchange DOUYU_XPD_BIND_POLL_SECONDS = 300 DOUYU_XPD_BIND_POLL_INTERVAL = 5 @@ -581,6 +582,21 @@ class XpdMixin: "currency": currency, } ) + # 和平小店兑换 —— 此前从不发事件(不在旧收集器的 allowlist 里)。 + # 消耗的是点券或扭蛋碎片,不是积分;金额用本地已知的 price(快照标价, + # 已在上面校验过 > 0),作用域标成 peace 以便和精英/电竞区分。 + record_exchange( + db, + account, + task_id=task.id, + name=goods.name or commodity_id, + score=price, + commodity_id=commodity_id, + game_name=display_role, + game_channel=channel, + points_after=result.get("new_balance"), + scope="peace", + ) self._mark_task( db, task,