6 Commits
Author SHA1 Message Date
yml2213 7aaaeebe2b fix(automation): 修正调度口径——作业每 5 分钟跑一轮,interval_seconds 只是单行重检间隔
P1a 把两个「间隔」混成了一个:next_run_at 用的是 job.interval_seconds,于是
作业 6 小时才跑一轮、每轮 200 条 = 800 条/天,全库 12,881 条要 16 天才能过
一遍 —— 等于没巡检,与规划里「账号 6 小时内被查一次」的目标差了两个数量级。

按规划原意修正:
- TICK_SECONDS 60 → 300:作业每个 tick(5 分钟)跑一轮;
- next_run_at 改为 finished + TICK_SECONDS,不再用 interval_seconds;
- interval_seconds 只作为「同一行多久重新检测」的选行 cutoff(选行逻辑本来就
  是这么用的,无需改);
- 容量口径随之更新:5 分钟一轮 × 200 条 = 40 条/分钟 ≥ 需要的 36 条/分钟,
  冷启动约 5.4 小时覆盖全库,稳态每轮一百多条;
- 前端标签由「巡检周期(小时)」改为「单账号重检间隔(小时)」,说明文案写清
  「每 5 分钟跑一轮」,避免运维误以为是作业周期。

同步更新规划文档 §7.2/§7.3(含两个间隔不能混的说明)与附录 C 的三段上线
步骤(立即跑一次 → 每轮 50 观察 → 每轮 200 启用)。
2026-10-02 19:54:31 +08:00
yml2213 2379598800 docs(plan): 记录迁移编号不能改名的坑,并说明应用过后的对齐做法
P1a 的迁移先叫 20261002_0038,本地 dev 库应用过之后为保持链条连续改名成
0037,dev 库的 alembic_version 还记着旧编号,再启动就报
Can't locate revision identified by '20261002_0038'。

结论写进附录 C.1:编号要在被任何库应用之前定下来;已经应用过又必须改名时,
除改文件外还要把每个应用过的库 UPDATE alembic_version 对齐(前提是迁移内容
未变)。
2026-10-02 19:48:47 +08:00
yml2213 f745a61df1 feat(automation): CK 自动巡检(P1a),只检测不改 CK
后台原本靠人工逐批点「CK 检测」发现失效账号,1.2 万个账号根本盯不过来。
P1a 把「哪些账号失效了」自动化:自动挑久未检测的账号、调现成检测接口、
把结果写回 ck_check_*。**本作业不替换任何 CK**,重登仍由人工点,符合规划里
「先做只读/只提示,确认无误再做自动动作」的渐进上线原则。

实现:
- 迁移 20261002_0037:automation_jobs / automation_runs 两张表,
  login_tasks 加 relogin_attempts / relogin_next_at / relogin_last_error
  (为 P1b 备好),accounts 加 auto_managed(特殊账号不参与自动化);
- services/automation/scheduler.py:进程内调度线程,60s tick,状态全部落库
  (重启不丢调度进度、不重复跑轮);AUTOMATION_ENABLED 默认关闭,作业开关
  也默认关闭;ensure_jobs 只补缺失行、不覆盖运维改过的配置;
  作业异常只记 failed,不拖垮调度线程;单容器 --workers 1,无需分布式锁;
- services/automation/jobs/ck_health.py:选行口径与人工检测一致
  (cookie_holding_filter:持有 CK 且无在途尝试),叠加久未检测、
  auto_managed、batch_size,按 ck_checked_at 升序(最久没查的优先);
  只写 ck_check_status / ck_check_result / ck_checked_at;
- routers/automation.py:GET /jobs、PATCH /jobs/{key}、POST /jobs/{key}/run、
  GET /runs;新增权限点 automation:view(运营可看)与 automation:manage
  (仅超管),改配置与手动触发都写审计;
- 顺带补 POST /api/login/batch 的审计(此前批次是谁发起、多少账号无据可查);
- 前端 /automation 页:作业卡片(开关/周期/单轮条数/并发)、运行历史、
  「立即跑一次」;总开关未打开、调度线程未运行时给明确提示。

验证:
- 22 条新测试覆盖调度器行为(只跑到点的、异常不抛出、下轮时间、配置不被重置)、
  选行条件(在途尝试/auto_managed/间隔/batch_size)、接口与权限,其中一条
  专门断言巡检后 cookie/status/message/attempt_status 一字未动;
- 全量 257 条通过;前端 tsc 与构建通过;
- 端到端用 3 个真实账号跑真实斗鱼接口:巡检结果与直接调 check_douyu_cookie
  逐条一致(3/3 valid),CK 与尝试状态全程未变。

顺带修掉一处会炸生产的坑:DB 时间列读出来是 naive UTC,直接与
datetime.now(UTC) 比较会 TypeError,现统一走 _as_utc() 补时区。
2026-10-02 19:47:41 +08:00
yml2213 6537d77323 chore(gitignore): 忽略 .env.bak* 与 backups/,防口令和账号数据误提交
原规则只忽略 .env,改 .env 前留的备份(.env.bak-时间戳)虽然未跟踪但**未被
忽略**,一次 git add -A 就会把数据库口令提交进仓库;backups/ 目录同理,服务器
上那份 login_tasks 全量备份有 33MB 账号数据。
2026-10-02 19:34:37 +08:00
yml2213 6a62bbe70f docs(plan): 新增自动化与资料收集规划与皮肤准入映射分析
把「CK 全靠人工维护 / 资料靠人转述 / 兑换无台账」整理成分期规划,并基于两套
线上库的实测数据定下关键口径。

数据结论(2026-10-02 实测):
- order_site 271,134 单按履约路由二选一且完全不重叠:A 有 fulfillment_tasks
  85,338 单(88.3% 已 delivered,是发货平台的活)、B 有 work_orders 96,755 单
  (打手)、C 两者都无 89,542 单 —— 只有 C 是「没人管」,才是本系统的目标;
- 斗鱼可兑换商品只有 78 个(皮肤类 60)。C 桶皮肤类 923 个快手商品名映射过去:
  62 个唯一命中(27.7% 件数)、14 个命中多个(12.4%,需用户选)、847 个斗鱼
  兑不了(59.9%)—— 准入白名单以斗鱼商品目录为准,不匹配就不建单;
- order_items.spec_json 已带 sellerId/relItemId/relSkuId/skuNick,(itemId, skuId)
  才是单订单的精确定位键(4,092 个数字 skuId 中 89% 唯一),接单平台就是按
  sku_exact/sku_series 匹配的,本系统照此设计;
- CK 巡检规模 12,881 行(valid 10,079 / invalid 905 / 从未检测 1,897),按 6h
  周期反推 batch_size 应为 200 而非 50,否则实际周期被拉长到 21 小时。

规划要点:
- 与 order_site 整合选只读直连 PG:order_site 零代码改动,跨库 SQL 只允许出现在
  一个适配层文件里,另配降级开关与只读保护参数;
- P1 拆成 P1a(只检测、不改动任何 CK)与 P1b(自动重登,默认关闭),按「渐进
  上线」原则逐步减少人工;
- 用户截图一律进私有 OSS 桶 + 后端鉴权代理读图,不落本地磁盘;
- P4 的规则/模板/状态机对齐 order_site 接单平台:资料模板固定为游戏编号 +
  游戏昵称 + 主页截图,皮肤由订单号自动对应,仅多选类让用户选一件;
- 取消规则映射层(与 skin_aliases 职责重叠),规则直接挂 commodity_id,
  连带去掉 rule_ambiguous 状态。

三轮复核:① 补齐结构与缺口(术语表、P0 章节、权限表、配置清单、保留策略、
文案、测试计划、风险回退、迁移与文件清单、施工单);② 逐条核对代码事实
(路径/符号/接口/字段/迁移 head/测试基线全部命中);③ 找出并修掉 8 处实现级
矛盾(选择结果无处存、365 天命中的状态缺失、draft 悬空、多选回链断点、P4 与
P1b 排期矛盾、映射层重叠、P2 能力错位、用户回看链接缺失)。

含:docs/自动化与资料收集规划.md、docs/皮肤准入映射分析.md,并更新文档索引。
2026-10-02 19:34:37 +08:00
yml2213 2bdabc4e03 fix(compose): 补容器资源限额,并修正数据卷默认值以防错挂空卷
一、内存 / CPU 限额
douyu-login 与 mysql 此前均为 unlimited(Mem=0、NanoCpus=0)。本机 7G 且与
order_site 共用,任一方内存失控都会把整机拖垮(order_site 曾因此被 OOM kill)。
现各限 2G / 2 核;可用 DOUYU_LOGIN_MEM_LIMIT、DOUYU_LOGIN_CPUS、MYSQL_MEM_LIMIT、
MYSQL_CPUS 在 .env 覆盖。

二、修正 mysql-data 的卷名默认值(重要,防数据事故)
原默认值 live-hub-py_mysql-data 是**另一个空卷**,真实数据在项目改名前的
douyu_login_py_mysql-data(4G+)。由于该行改动之后一直没重建过容器,
隐患一直没暴露;2026-10-02 第一次重建容器即挂到空卷,
表现为数据全丢(douyu_tasks 由 228475 行变成空表)。

现默认值指向真实卷,并默认 external: true(不再由本项目创建/删除该卷,
docker compose down -v 也不会把它删掉)。修完不再依赖 .env 是否设置该变量。
2026-10-02 19:34:29 +08:00
23 changed files with 3194 additions and 7 deletions
+5
View File
@@ -8,6 +8,11 @@ __pycache__/
# 环境变量(含敏感信息,勿提交)
.env
# 改 .env 前留的备份,文件名带时间戳,同样含口令
.env.bak*
# 数据库备份(含账号数据,体积大)
backups/
# 运行时数据
data/
+10 -3
View File
@@ -8,6 +8,8 @@ services:
# 服务器默认走阿里云镜像;本地测试设为 false。
USE_CHINA_MIRRORS: ${USE_CHINA_MIRRORS:-true}
restart: unless-stopped
mem_limit: ${DOUYU_LOGIN_MEM_LIMIT:-2g}
cpus: ${DOUYU_LOGIN_CPUS:-2}
ports:
# 保持现有服务端口;可通过 APP_PORT 覆盖。
- "${APP_PORT:-8000}:8800"
@@ -90,6 +92,8 @@ services:
mysql:
image: ${MYSQL_IMAGE:-docker.m.daocloud.io/library/mysql:8.4}
restart: unless-stopped
mem_limit: ${MYSQL_MEM_LIMIT:-2g}
cpus: ${MYSQL_CPUS:-2}
ports:
# 仅绑定宿主机本地地址,供 ./dev.sh 的本地后端连接,不对外网开放。
- "${MYSQL_BIND_HOST:-127.0.0.1}:${MYSQL_HOST_PORT:-3307}:3306"
@@ -118,9 +122,12 @@ services:
volumes:
mysql-data:
# 目录/项目改名时优先复用旧项目的数据卷;新环境默认使用新项目名卷。
name: ${MYSQL_VOLUME_NAME:-live-hub-py_mysql-data}
external: ${MYSQL_VOLUME_EXTERNAL:-false}
# 复用项目改名前的旧数据卷 douyu_login_py_mysql-data(4G+ 真实数据)。
# 注意:曾用默认值 live-hub-py_mysql-data,那是**另一个空卷** —— 目录改名后第一次
# 重建容器会挂到空卷上,看起来像数据全丢(2026-10-02 实际发生过)。
# 因此默认值与 external 都指向真实卷,不依赖 .env 是否设置。
name: ${MYSQL_VOLUME_NAME:-douyu_login_py_mysql-data}
external: ${MYSQL_VOLUME_EXTERNAL:-true}
networks:
order-site-net:
+3 -1
View File
@@ -1,9 +1,11 @@
# 项目文档索引
更新时间:2026-09-02
更新时间:2026-10-02
## 当前入口
- [自动化与资料收集规划](自动化与资料收集规划.md):CK 自动管理、资料收集链接、皮肤台账与半自动工单的分期计划(规划中,未开工)。
- [皮肤准入映射分析](皮肤准入映射分析.md):C 桶皮肤类到斗鱼商品目录的归一与覆盖分析(923 个快手名 → 78 个斗鱼商品,覆盖率 40.1%)。
- [虎牙 Cookie 路线状态](HUYA_COOKIE路线状态.md):当前实现、字段状态、验证边界和后续工作。
- [虎牙网页设备字段溯源](HUYA_WEB_FIELD_TRACE.md):浏览器证据、字段来源和生成算法。
- [虎牙 Cookie 合并审计与来源矩阵](HUYA_COOKIE合并审计与来源矩阵.md):字段来源约束和合并策略。
+255
View File
@@ -0,0 +1,255 @@
# C 桶皮肤类准入映射分析
生成时间:2026-10-02 数据源:order_site 线上 PostgreSQL(2026-10-02)+ 本系统 `douyu_goods_snapshot`
## 1. 结论
范围:C 桶(`fulfillment_tasks` 与 `work_orders` 都没有的订单)中**皮肤/套装类**共 **923 个快手商品名 / 49,412 件**。
| 类别 | 快手名 | 件数 | 占比 | 处理方式 |
| --- | --- | --- | --- | --- |
| ✅ **唯一命中斗鱼商品** | 62 | 13,678 | 27.7% | 订单可自动确定皮肤 → 直接建单 |
| ⚠️ **命中多个** | 14 | 6,122 | 12.4% | 必须让用户在收集链接里**选一件** |
| ❌ **斗鱼兑不了** | 847 | 29,612 | 59.9% | 拒绝(斗鱼没有对应商品) |
**三条关键结论**
1. **锚点应该是斗鱼商品,不是快手商品名**:本系统 `douyu_goods_snapshot` 只有 **78 个可兑换商品**(皮肤类 60 个,非皮肤 18 个),全部来自手册 `20260120QYOOB`。快手那边 923 个名字最终都要落到这几十个商品上;
2. **能做的只有 19,800 件(40.1%)**:斗鱼现有商品覆盖不到 6 成的 C 桶皮肤订单。剩下 29,612 件(59.9%)斗鱼根本没有对应商品,**做不了**;
3. **归一/别名梳理的真实工作量很小**:不是梳理 923 或 615 个标准名,而是把 **76 个快手商品名**映射到 **60 个斗鱼商品**。
## 2. 斗鱼可兑换商品全清单
### 2.1 皮肤类(60 个)
| commodity_id | 商品名 | 积分 |
| --- | --- | --- |
| `20260120QYOOB_bag314` | AKM-煎蛋卷卷 | 988 |
| `20260120QYOOB_bag358` | GROZA-罗小黑 | 480 |
| `20260120QYOOB_bag351` | M416-罗小黑 | 960 |
| `20260120QYOOB_bag333` | M762-芝莓熊熊 | 688 |
| `20260120QYOOB_bag332` | MG3-芝莓熊熊 | 488 |
| `20260120QYOOB_bag334` | UZI-芝莓熊熊 | 488 |
| `20260120QYOOB_bag315` | Vector-煎蛋卷卷 | 488 |
| `20260120QYOOB_bag348` | 仙剑奇侠传大礼包 | 18000 |
| `20260120QYOOB_bag349` | 仙剑飞行器大礼包 | 8000 |
| `20260120QYOOB_bag11` | 兔兔白日梦头盔 | 1680 |
| `20260120QYOOB_bag12` | 兔兔白日梦背包 | 1680 |
| `20260120QYOOB_bag346` | 冥河战士大礼包 | 8000 |
| `20260120QYOOB_bag10` | 套装-兔兔白日梦 | 1280 |
| `20260120QYOOB_bag329` | 套装-奶油柴柴 | 988 |
| `20260120QYOOB_bag8` | 套装-宝石天鹅 | 1024 |
| `20260120QYOOB_bag319` | 套装-小鸟芋圆 | 980 |
| `20260120QYOOB_bag337` | 套装-彩虹骑师 | 988 |
| `20260120QYOOB_bag9` | 套装-星河浪客 | 1680 |
| `20260120QYOOB_bag7` | 套装-星谷精灵 | 688 |
| `20260120QYOOB_bag2` | 套装-暗影哥特 | 888 |
| `20260120QYOOB_bag354` | 套装-暴走罗小黑 | 680 |
| `20260120QYOOB_bag4` | 套装-浪漫天命 | 360 |
| `20260120QYOOB_bag353` | 套装-激萌罗小黑 | 680 |
| `20260120QYOOB_bag309` | 套装-煎蛋卷卷 | 988 |
| `20260120QYOOB_bag5` | 套装-狂战萌兔 | 480 |
| `20260120QYOOB_bag360` | 套装-猫布灵 | 988 |
| `20260120QYOOB_bag361` | 套装-糯粉咩咩 | 1680 |
| `20260120QYOOB_bag338` | 套装-绿野诗集 | 788 |
| `20260120QYOOB_bag352` | 套装-罗小黑 | 960 |
| `20260120QYOOB_bag326` | 套装-芝莓熊熊 | 988 |
| `20260120QYOOB_bag328` | 套装-蓝莓兔兔 | 988 |
| `20260120QYOOB_bag6` | 套装-银月精灵 | 688 |
| `20260120QYOOB_bag347` | 女娲后裔-赵灵儿大礼包 | 8000 |
| `20260120QYOOB_bag14` | 暗夜武装大礼包 | 8000 |
| `20260120QYOOB_bag325` | 极夜星羊大礼包 | 2800 |
| `20260120QYOOB_bag308` | 果冻柠檬大礼包 | 2800 |
| `20260120QYOOB_bag364` | 烟雾弹-糯粉咩咩 | 488 |
| `20260120QYOOB_bag310` | 煎蛋卷卷头盔 | 900 |
| `20260120QYOOB_bag311` | 煎蛋卷卷背包 | 900 |
| `20260120QYOOB_bag312` | 煎蛋卷卷降落伞 | 288 |
| `20260120QYOOB_bag317` | 熔岩游骑兵礼包 | 600 |
| `20260120QYOOB_bag318` | 玩偶服-卖崽青蛙 | 980 |
| `20260120QYOOB_bag13` | 破片手榴弹-兔兔白日梦 | 488 |
| `20260120QYOOB_bag316` | 破片手榴弹-煎蛋卷卷 | 488 |
| `20260120QYOOB_bag365` | 破片手榴弹-糯粉咩咩 | 488 |
| `20260120QYOOB_bag362` | 糯粉咩咩头盔 | 1680 |
| `20260120QYOOB_bag363` | 糯粉咩咩背包 | 1680 |
| `20260120QYOOB_bag366` | 糯粉咩咩降落伞 | 488 |
| `20260120QYOOB_bag355` | 罗小黑头盔 | 1280 |
| `20260120QYOOB_bag356` | 罗小黑背包 | 1280 |
| `20260120QYOOB_bag357` | 罗小黑降落伞 | 360 |
| `20260120QYOOB_bag313` | 背包挂件-煎蛋卷卷 | 288 |
| `20260120QYOOB_bag367` | 背包挂件-糯粉咩咩 | 488 |
| `20260120QYOOB_bag359` | 背包挂件-罗小黑 | 480 |
| `20260120QYOOB_bag336` | 背包挂件-芝莓熊熊 | 388 |
| `20260120QYOOB_bag330` | 芝莓熊熊头盔 | 1280 |
| `20260120QYOOB_bag331` | 芝莓熊熊背包 | 1280 |
| `20260120QYOOB_bag335` | 芝莓熊熊降落伞 | 388 |
| `20260120QYOOB_bag350` | 蜀山剑侠-李逍遥大礼包 | 8000 |
| `20260120QYOOB_bag3` | 黑色高级特训官上衣 | 360 |
### 2.2 非皮肤类(18 个,不进皮肤白名单)
| commodity_id | 商品名 | 积分 |
| --- | --- | --- |
| `20260120QYOOB_bag55` | 伯爵体验卡(7天) | 4650 |
| `20260120QYOOB_bag54` | 公爵体验卡(3天) | 5000 |
| `20260120QYOOB_bag53` | 国王体验卡(3天) | 10000 |
| `20260120QYOOB_bag59` | 子爵体验卡(7天) | 1200 |
| `20260120QYOOB_bag19` | 幸运币礼包(中) | 300 |
| `20260120QYOOB_bag20` | 幸运币礼包(大) | 900 |
| `20260120QYOOB_bag18` | 幸运币礼包(小) | 20 |
| `20260120QYOOB_bag56` | 改名卡 | 5000 |
| `20260120QYOOB_bag58` | 游侠体验卡(14天) | 100 |
| `20260120QYOOB_bag62` | 精英卡 | 60 |
| `20260120QYOOB_bag61` | 精英星光 | 10 |
| `20260120QYOOB_bag64` | 精英火箭 | 5000 |
| `20260120QYOOB_bag63` | 精英超火 | 20000 |
| `20260120QYOOB_bag65` | 精英飞机 | 1000 |
| `20260120QYOOB_bag16` | 荣耀勋章礼包(中) | 300 |
| `20260120QYOOB_bag17` | 荣耀勋章礼包(大) | 900 |
| `20260120QYOOB_bag15` | 荣耀勋章礼包(小) | 20 |
| `20260120QYOOB_bag57` | 骑士体验卡(14天) | 900 |
### 2.3 归一层用的核心词(30 个)
```
女娲后裔-赵灵儿大、蜀山剑侠-李逍遥大、黑色高级特训官上衣、仙剑飞行器大、仙剑奇侠传大、暴走罗小黑、兔兔白日梦、暗夜武装大、果冻柠檬大、激萌罗小黑、冥河战士大、熔岩游骑兵、极夜星羊大、星谷精灵、蓝莓兔兔、彩虹骑师、宝石天鹅、奶油柴柴、银月精灵、卖崽青蛙、浪漫天命、芝莓熊熊、绿野诗集、狂战萌兔、星河浪客、糯粉咩咩、暗影哥特、煎蛋卷卷、小鸟芋圆、猫布灵
```
## 3. 可映射的快手商品名
### 3.1 唯一命中(订单可直接确定皮肤)
| 快手商品名 | 件数 | → 斗鱼商品关键词 |
| --- | --- | --- |
| 兔兔白日梦 | 4,854 | 兔兔白日梦 |
| 狂战萌兔 | 3,340 | 狂战萌兔 |
| 冥河战士大礼包 | 1,207 | 冥河战士大 |
| 猫布灵 | 1,020 | 猫布灵 |
| 激萌罗小黑 | 611 | 激萌罗小黑 |
| 狂战萌兔-套装 | 527 | 狂战萌兔 |
| 【穿搭】浪漫天命+黑色上衣 | 255 | 浪漫天命 |
| 和平精英皮肤小店入口出售返场吃鸡新皮肤暗影哥特红眼罩百搭眼罩 | 244 | 暗影哥特 |
| 兔兔白日梦套装 | 219 | 兔兔白日梦 |
| 浪漫天命 | 170 | 浪漫天命 |
| 和平精英皮肤小店入口出售返场吃鸡皮肤浪漫天命 | 168 | 浪漫天命 |
| 和平精英皮肤小店返场吃鸡新皮肤暗影哥特红眼罩百搭眼罩游戏充值 | 156 | 暗影哥特 |
| 和平精英皮肤小店入口出售返场吃鸡皮肤兔兔白日梦 | 123 | 兔兔白日梦 |
| 暗影哥特 | 107 | 暗影哥特 |
| 和平精英皮肤小店入口暗影哥特皮肤限时返厂 | 96 | 暗影哥特 |
| 熔岩游骑兵 | 68 | 熔岩游骑兵 |
| 极夜星羊大礼包 | 58 | 极夜星羊大 |
| 芝莓熊熊 | 45 | 芝莓熊熊 |
| 和平精英皮肤小店入口返场集市新皮肤出售浪漫天命 | 40 | 浪漫天命 |
| 兔兔白日梦手雷 | 39 | 兔兔白日梦 |
| 糯粉咩咩套装 | 37 | 糯粉咩咩 |
| 和平精英皮肤小店入口出售返场吃鸡皮肤狂战萌兔 | 36 | 狂战萌兔 |
| 彩虹骑师 | 29 | 彩虹骑师 |
| 浪漫天命+黑色上衣 | 27 | 浪漫天命 |
| 熔岩游骑兵+夹克 | 19 | 熔岩游骑兵 |
| 熔岩游骑兵口罩+夹克 | 16 | 熔岩游骑兵 |
| 星河浪客 | 15 | 星河浪客 |
| (4选1)魔性柴柴-奶油庭院-芝莓熊熊-纯白悦动 | 12 | 芝莓熊熊 |
| 套装-浪漫天命 | 11 | 浪漫天命 |
| 套装-糯粉咩咩 | 9 | 糯粉咩咩 |
| 宝石天鹅 | 9 | 宝石天鹅 |
| 暗影哥特+红眼罩 | 8 | 暗影哥特 |
| (4选1)魔性柴柴-奶油庭院-芝莓熊熊-纯白跃动 | 8 | 芝莓熊熊 |
| 蓝莓兔兔 | 8 | 蓝莓兔兔 |
| 和平精英皮肤小店入口返场集市新皮肤出售糯粉咩咩套装 | 8 | 糯粉咩咩 |
| 熔岩游骑兵+面罩 | 6 | 熔岩游骑兵 |
| 兔兔白日梦【秒到无差价】 | 6 | 兔兔白日梦 |
| 绿野诗集 | 6 | 绿野诗集 |
| 套装-暗影哥特 | 5 | 暗影哥特 |
| (4选1)魔性柴柴/奶油庭院/芝莓熊熊/纯白悦动 | 5 | 芝莓熊熊 |
| 奶油柴柴 | 5 | 奶油柴柴 |
| 熔岩游骑兵礼包 | 4 | 熔岩游骑兵 |
| 和平精英皮肤小店入口出售返场吃鸡皮肤糯粉咩咩游戏充值 | 4 | 糯粉咩咩 |
| 和平精英皮肤小店入口出售返场吃鸡皮肤糯粉咩咩 | 4 | 糯粉咩咩 |
| 和平精英皮肤小店入口出售返场吃鸡皮肤熔岩游骑兵 | 3 | 熔岩游骑兵 |
| 果冻柠檬大礼包 | 3 | 果冻柠檬大 |
| 浪漫天命【秒到无差价】 | 3 | 浪漫天命 |
| 破片手榴弹-糯粉咩咩 | 3 | 糯粉咩咩 |
| 和平精英皮肤小店入口返场集市新皮肤出售糯粉咩咩可爱小羊面具 | 3 | 糯粉咩咩 |
| 糯粉咩咩 | 2 | 糯粉咩咩 |
| 糯粉咩咩套装【秒到无差价】 | 2 | 糯粉咩咩 |
| 暗影哥特【秒到无差价】 | 2 | 暗影哥特 |
| 和平精英皮肤小店入口出售返场吃鸡皮肤黑色高级特训官上衣 | 2 | 黑色高级特训官上衣 |
| 和平精英皮肤小店返场吃鸡新皮肤糯粉咩咩手榴弹百搭游戏充值 | 2 | 糯粉咩咩 |
| 和平精英皮肤小店入口出售返场皮肤糯粉咩咩系列烟雾dan游戏充值 | 2 | 糯粉咩咩 |
| 黑色高级特训官上衣 | 1 | 黑色高级特训官上衣 |
| 兔兔白日梦背包 | 1 | 兔兔白日梦 |
| 烟雾弹-糯粉咩咩 | 1 | 糯粉咩咩 |
| 熔岩游骑兵口罩+夹克【秒到无差价】 | 1 | 熔岩游骑兵 |
| 狂战萌兔【秒到无差价】 | 1 | 狂战萌兔 |
| (4选1)魔性柴柴-奶油庭院-芝莓熊熊-纯白跃动( | 1 | 芝莓熊熊 |
| 魔性柴柴-奶油庭院-芝莓熊熊-纯白跃动 | 1 | 芝莓熊熊 |
### 3.2 命中多个(必须在链接里让用户选)
| 快手商品名 | 件数 | 候选 |
| --- | --- | --- |
| 和平精英皮肤小店入口浪漫天命黑色上衣暗影哥特雪国小丑木乃伊 | 3,739 | 暗影哥特 / 浪漫天命 |
| 和平精英熔岩游骑兵胖达团团圆圆萌熊伴侣背包雪山精英兔兔白日梦 | 1,606 | 兔兔白日梦 / 熔岩游骑兵 |
| 【穿搭】暗影哥特+黑色上衣+浪漫天命 | 236 | 暗影哥特 / 浪漫天命 |
| 【穿搭】暗影哥特+狂沙舞者+浪漫天命 | 170 | 暗影哥特 / 浪漫天命 |
| (3选1)芝莓熊熊-蓝莓兔兔-奶油柴柴 | 125 | 奶油柴柴 / 芝莓熊熊 / 蓝莓兔兔 |
| (3选1)煎蛋卷卷/卖崽青蛙/小鸟芋圆 | 86 | 卖崽青蛙 / 小鸟芋圆 / 煎蛋卷卷 |
| 暗影哥特+黑色上衣+浪漫天命 | 83 | 暗影哥特 / 浪漫天命 |
| 暗影哥特+狂杀舞者+浪漫天命 | 34 | 暗影哥特 / 浪漫天命 |
| (3选1)芝莓熊熊-蓝莓熊熊-奶油柴柴 | 14 | 奶油柴柴 / 芝莓熊熊 |
| (2选1)银月精灵-星谷精灵 | 13 | 星谷精灵 / 银月精灵 |
| 暗影哥特+狂沙舞者+浪漫天命 | 8 | 暗影哥特 / 浪漫天命 |
| 暗影哥特+浪漫天命+狂沙舞者 | 5 | 暗影哥特 / 浪漫天命 |
| 银月精灵/星谷精灵【2选1】 | 2 | 星谷精灵 / 银月精灵 |
| 2选1)银月精灵-星谷精灵 | 1 | 星谷精灵 / 银月精灵 |
### 3.3 斗鱼兑不了(847 个名字 / 29,612 件,仅列件数 Top 40)
| 快手商品名 | 件数 |
| --- | --- |
| 青涩年华 | 2,068 |
| 破片手榴弹-粉粉猫爪大礼包 | 1,779 |
| 和平精英小店皮肤返场红熊茄茄雪国幻梦放客神游怪诞乔克猪妹脸 | 1,340 |
| 风蚀行者 | 1,088 |
| 雪山精英 | 1,008 |
| 和平精英地铁逃生装备雪隼七级套耐火轩辕双弹头4词条aug低价便宜 | 870 |
| AUG-魔性柴柴 | 768 |
| 浪漫波比 | 646 |
| 套装-暮夜私语 | 630 |
| (3选1)甜心娇蕊-清雅山茶-紫梦罗兰 | 625 |
| 天网终结者大礼包 | 603 |
| 暮夜私语 | 560 |
| 和平精英人气值王私人飞机梦幻烟花爱心熊金色三级头充值 | 556 |
| 顽皮橙 | 500 |
| 风蚀行者-套装 | 435 |
| (3选1)顽皮玉兔-机械兔法师-浪漫假日【自行兑换】 | 429 |
| 烟雾弹-猫咪绅士大礼包 | 410 |
| 和平精英地铁逃生动力装甲加特林肩炮星铸壁垒外观皮肤装备机甲 | 402 |
| 幸运萌趣趣 | 398 |
| (3选1)软软马卡龙/软软小魔女/软软绵绵云 | 397 |
| 软软马卡龙 | 315 |
| 雪山精英-老六服 | 299 |
| (3选1)甜橙豚豚/芝士鼠鼠/摩卡猫猫 | 297 |
| 黑色rock长袖 | 292 |
| TT战队服 | 279 |
| (3选1)甜橙豚豚/芝士鼠鼠/摩卡猫猫 | 278 |
| (18选1)战队服 | 254 |
| 灰烬之翎全包系列 | 248 |
| 【壁垒】动力装甲 | 244 |
| 纯白跃动 | 243 |
| (3选1)炽情鼓音-旭辉号角-星乐指令 | 232 |
| 晶辉守护者【眼罩可拆】 | 227 |
| 萌熊伴侣背包 | 226 |
| 一把随机金枪 | 214 |
| 碎梦浮光全包系列 | 206 |
| 和平精英吃鸡全区全服点卷充值代充点卷直充代充198/328游戏充值 | 206 |
| 主页点赞1000个 | 194 |
| 赵灵儿大礼包 | 192 |
| 套装-墨青刃影 | 173 |
| (3选1)甜橙豚豚/芝士鼠鼠/摩卡猫猫 | 172 |
## 4. 对规划的影响
1. **`skin_catalog` 改为以斗鱼商品为主键**(78 行),快手商品名作为别名挂上去 —— 数据量从「615 个标准名」降到「78 个商品 + 89 条别名」;
2. **准入白名单 = 能映射到斗鱼商品**(第一层 `route=unhandled` + 第二层命中映射),未命中的订单直接拒绝;
3. **决策 8 需要补一个例外**:唯一命中时皮肤由订单决定(用户不用选);命中多个时必须让用户在链接里选一件;
4. **29,612 件兑不了的订单**需要业务决策:是不接、还是在斗鱼侧开通对应商品。
File diff suppressed because it is too large Load Diff
+453
View File
@@ -0,0 +1,453 @@
"""P1a 回归:CK 自动巡检。
两块约束必须一起守住:
1. **调度器行为**:只跑到点的、打开开关的作业;作业抛异常只记 failed,不能
把调度线程带崩;跑完排下一轮时间;已有配置不能被重启重置。
2. **本作业绝不改 CK**:P1a 的定位是「只检测、只标记」,所以巡检后
``cookie`` / ``status`` / ``message`` / ``attempt_*`` 必须一字未动 ——
线上全靠这条保证「自动化上线了但人工流程照旧」,回归时一旦有人顺手写了
``status``,这里要立刻红。
"""
from datetime import UTC, datetime, timedelta
import pytest
from fastapi import HTTPException
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from web.backend.database import Base
from web.backend.deps import require_permission
from web.backend.models import Account, AutomationJob, AutomationRun, LoginTask, User
from web.backend.services.automation import (
JobSpec,
register_default_jobs,
register_job,
reset_registry,
run_due_jobs,
run_job_now,
)
from web.backend.services.automation.jobs import ck_health
class _Base:
def setup_method(self):
self.engine = create_engine("sqlite://")
Base.metadata.create_all(self.engine)
self.session = sessionmaker(bind=self.engine)()
self.admin = User(
username="admin", password_hash="h", role="super_admin", is_active=True
)
self.support = User(
username="kf", password_hash="h", role="support", is_active=True
)
self.session.add_all([self.admin, self.support])
self.session.commit()
# 注册表是模块级的,不重置会被上一个用例注册的探针作业污染。
reset_registry()
register_default_jobs()
def teardown_method(self):
self.session.close()
def make_account(self, name="acc", *, auto_managed=True, assigned_to=None):
account = Account(
username=name,
password="p",
email=f"{name}@example.com",
email_password="ep",
assigned_to=assigned_to,
auto_managed=auto_managed,
)
self.session.add(account)
self.session.commit()
return account
def make_task(
self,
account,
*,
cookie="acf_uid=1",
status="success",
attempt_status="",
ck_checked_at=None,
ck_check_status="",
):
task = LoginTask(
batch_id="b1",
account_id=account.id,
created_by=self.admin.id,
status=status,
cookie=cookie,
attempt_status=attempt_status,
ck_checked_at=ck_checked_at,
ck_check_status=ck_check_status,
)
self.session.add(task)
self.session.commit()
return task
class TestCkHealthSelection(_Base):
def _selected_ids(self, job):
return [
t.id for t in ck_health._select_tasks(self.session, job, datetime.now(UTC))
]
def test_picks_only_rows_that_still_hold_cookie(self):
job = self._job()
with_cookie = self.make_task(self.make_account("a"))
self.make_task(self.make_account("b"), cookie="")
assert self._selected_ids(job) == [with_cookie.id]
def test_skips_accounts_with_inflight_attempt(self):
job = self._job()
idle = self.make_task(self.make_account("a"))
self.make_task(self.make_account("b"), attempt_status="running")
self.make_task(self.make_account("c"), attempt_status="pending")
assert self._selected_ids(job) == [idle.id]
def test_skips_accounts_opted_out_of_automation(self):
job = self._job()
managed = self.make_task(self.make_account("a"))
self.make_task(self.make_account("b", auto_managed=False))
assert self._selected_ids(job) == [managed.id]
def test_respects_interval_and_prefers_stalest(self):
job = self._job()
fresh = self.make_task(
self.make_account("fresh"), ck_checked_at=datetime.now(UTC)
)
stale = self.make_task(
self.make_account("stale"),
ck_checked_at=datetime.now(UTC) - timedelta(hours=30),
)
never = self.make_task(self.make_account("never"), ck_checked_at=None)
ids = self._selected_ids(job)
assert fresh.id not in ids
# 从没检测过的排最前,其次是久未检测的
assert ids[0] == never.id
assert stale.id in ids
def test_batch_size_caps_the_round(self):
job = self._job()
job.batch_size = 2
for index in range(5):
self.make_task(self.make_account(f"a{index}"))
assert len(self._selected_ids(job)) == 2
def _job(self):
return AutomationJob(
key=ck_health.JOB_KEY, interval_seconds=6 * 3600, batch_size=200
)
class TestCkHealthNeverTouchesCookie(_Base):
def test_patrol_only_writes_check_fields(self):
job = self._job()
account = self.make_account("acc")
task = self.make_task(account, attempt_status="failed", ck_check_status="")
before = (task.cookie, task.status, task.message, task.attempt_status)
ck_health.check_douyu_cookie = _fake_check(valid=True)
try:
counters = ck_health.run_ck_health(self.session, job)
finally:
ck_health.check_douyu_cookie = _REAL_CHECK
self.session.refresh(task)
assert counters["scanned"] == 1 and counters["valid"] == 1
assert task.ck_check_status == "valid"
assert task.ck_checked_at is not None
assert task.ck_check_result["nickname"] == "测试"
# 核心断言:CK 本体与尝试状态一字未动
assert (task.cookie, task.status, task.message, task.attempt_status) == before
def test_invalid_result_marks_invalid_but_keeps_cookie(self):
job = self._job()
task = self.make_task(self.make_account("acc"))
ck_health.check_douyu_cookie = _fake_check(valid=False)
try:
counters = ck_health.run_ck_health(self.session, job)
finally:
ck_health.check_douyu_cookie = _REAL_CHECK
self.session.refresh(task)
assert counters["invalid"] == 1
assert task.ck_check_status == "invalid"
assert task.cookie == "acf_uid=1"
def test_detection_exception_does_not_abort_round(self):
job = self._job()
self.make_task(self.make_account("a"))
self.make_task(self.make_account("b"))
ck_health.check_douyu_cookie = _fake_check(raises=True)
try:
counters = ck_health.run_ck_health(self.session, job)
finally:
ck_health.check_douyu_cookie = _REAL_CHECK
assert counters["scanned"] == 2
assert counters["invalid"] == 2
def _job(self):
return AutomationJob(
key=ck_health.JOB_KEY, interval_seconds=6 * 3600, batch_size=200
)
class TestScheduler(_Base):
def _register_probe(self, calls):
def run(db, job):
calls.append(job.key)
return {"scanned": 3, "valid": 2, "invalid": 1, "message": "ok"}
register_job(
JobSpec(
key="probe",
title="探针",
interval_seconds=3600,
batch_size=1,
concurrency=1,
run=run,
)
)
def test_only_enabled_jobs_run(self):
from web.backend.services.automation import ensure_jobs
calls: list[str] = []
self._register_probe(calls)
ensure_jobs(self.session)
self.session.query(AutomationJob).filter(AutomationJob.key == "probe").update(
{"enabled": True}
)
self.session.commit()
run_due_jobs(self.session)
assert calls == ["probe"]
calls.clear()
run_due_jobs(self.session)
# 上一轮已排了 next_run_at,未到点不再跑
assert calls == []
def test_job_failure_is_recorded_not_raised(self):
from web.backend.services.automation import ensure_jobs
def boom(db, job):
raise RuntimeError("接口挂了")
register_job(
JobSpec(
key="boom",
title="会炸的作业",
interval_seconds=3600,
batch_size=1,
concurrency=1,
run=boom,
)
)
ensure_jobs(self.session)
self.session.query(AutomationJob).filter(AutomationJob.key == "boom").update(
{"enabled": True}
)
self.session.commit()
outcomes = run_due_jobs(self.session)
assert outcomes[0]["status"] == "failed"
run = self.session.query(AutomationRun).one()
assert run.status == "failed" and "接口挂了" in run.message
def test_next_run_follows_tick_and_counters_persist(self):
from web.backend.services.automation import ensure_jobs
calls: list[str] = []
self._register_probe(calls)
ensure_jobs(self.session)
self.session.query(AutomationJob).filter(AutomationJob.key == "probe").update(
{"enabled": True}
)
self.session.commit()
run_due_jobs(self.session)
job = (
self.session.query(AutomationJob).filter(AutomationJob.key == "probe").one()
)
assert job.last_status == "success"
# 下一轮在一个 tick 之后(作业每个 tick 都跑),与 interval_seconds 无关
from web.backend.services.automation.scheduler import TICK_SECONDS
assert _aware(job.next_run_at) - _aware(job.last_run_at) == timedelta(
seconds=TICK_SECONDS
)
run = self.session.query(AutomationRun).one()
assert (run.scanned, run.valid, run.invalid) == (3, 2, 1)
def test_ensure_jobs_keeps_operator_settings(self):
from web.backend.services.automation import ensure_jobs
ensure_jobs(self.session)
job = (
self.session.query(AutomationJob)
.filter(AutomationJob.key == ck_health.JOB_KEY)
.one()
)
job.enabled = True
job.batch_size = 7
self.session.commit()
ensure_jobs(self.session) # 模拟重启
self.session.refresh(job)
assert job.enabled is True and job.batch_size == 7
def test_run_job_now_ignores_switch(self):
from web.backend.services.automation import ensure_jobs
ensure_jobs(self.session)
outcome = run_job_now(self.session, ck_health.JOB_KEY)
assert outcome["job_key"] == ck_health.JOB_KEY
assert self.session.query(AutomationRun).count() == 1
def test_run_job_now_unknown_key(self):
with pytest.raises(KeyError):
run_job_now(self.session, "nope")
class TestAutomationPermissions(_Base):
def test_view_allows_operation_denies_support(self):
from web.backend.permissions import ROLE_PERMISSIONS
assert "automation:view" in ROLE_PERMISSIONS["operation"]
assert "automation:view" not in ROLE_PERMISSIONS["support"]
assert "automation:manage" in ROLE_PERMISSIONS["super_admin"]
assert "automation:manage" not in ROLE_PERMISSIONS["operation"]
def test_manage_dependency_rejects_operator(self):
operator = User(
username="op", password_hash="h", role="operation", is_active=True
)
checker = require_permission("automation:manage")
with pytest.raises(HTTPException) as exc:
checker(current_user=operator)
assert exc.value.status_code == 403
def test_view_dependency_accepts_operator(self):
operator = User(
username="op", password_hash="h", role="operation", is_active=True
)
checker = require_permission("automation:view")
assert checker(current_user=operator).id == operator.id
def _aware(value):
"""库里读出的是 naive UTC,比较前补时区。"""
return value if value.tzinfo else value.replace(tzinfo=UTC)
_REAL_CHECK = ck_health.check_douyu_cookie
def _fake_check(*, valid=True, raises=False):
def fake(cookie: str) -> dict:
if raises:
raise RuntimeError("网络异常")
return {
"valid": valid,
"message": "ok" if valid else "Cookie 已失效",
"fish_ball": 1,
"nickname": "测试",
"level": 30,
"checked_at": datetime.now(UTC),
}
return fake
class TestAutomationApi(_Base):
"""接口层:序列化字段、开关落库、手动触发不受开关限制。"""
def _api(self):
from web.backend.routers import automation as api
return api
def test_list_jobs_exposes_scheduler_and_job_state(self):
api = self._api()
payload = api.list_jobs(db=self.session, current=self.admin)
assert payload["scheduler"]["tick_seconds"] == 300
assert payload["scheduler"]["running"] is False
keys = [item["key"] for item in payload["jobs"]]
assert keys == [ck_health.JOB_KEY]
job = payload["jobs"][0]
assert job["enabled"] is False
assert job["interval_seconds"] == 6 * 3600
assert job["batch_size"] == 200
assert job["latest_run"] is None
def test_update_job_persists_and_schedules_next_run(self):
api = self._api()
api.update_job(
ck_health.JOB_KEY,
api.JobUpdate(
enabled=True, interval_seconds=3600, batch_size=5, concurrency=2
),
db=self.session,
current=self.admin,
)
job = self.session.query(AutomationJob).one()
assert job.enabled is True
assert job.interval_seconds == 3600
assert job.batch_size == 5
assert job.next_run_at is not None # 打开开关就排了一轮
# 审计留痕
from web.backend.models import AuditLog
entry = (
self.session.query(AuditLog)
.filter(AuditLog.action == "automation:update")
.one()
)
assert entry.target == ck_health.JOB_KEY
def test_run_endpoint_records_run_and_audit(self):
api = self._api()
self.make_task(self.make_account("a"))
ck_health.check_douyu_cookie = _fake_check(valid=True)
try:
outcome = api.run_job(
ck_health.JOB_KEY, db=self.session, current=self.admin
)
finally:
ck_health.check_douyu_cookie = _REAL_CHECK
assert outcome["status"] == "success"
assert outcome["scanned"] == 1
assert self.session.query(AutomationRun).count() == 1
from web.backend.models import AuditLog
assert (
self.session.query(AuditLog)
.filter(AuditLog.action == "automation:run")
.count()
== 1
)
def test_run_endpoint_unknown_job_returns_404(self):
api = self._api()
with pytest.raises(HTTPException) as exc:
api.run_job("nope", db=self.session, current=self.admin)
assert exc.value.status_code == 404
def test_runs_endpoint_filters_by_job_key(self):
api = self._api()
self.session.add(
AutomationRun(job_key=ck_health.JOB_KEY, status="success", scanned=1)
)
self.session.add(AutomationRun(job_key="other", status="failed"))
self.session.commit()
payload = api.list_runs(
job_key=ck_health.JOB_KEY, limit=50, db=self.session, current=self.admin
)
assert len(payload["items"]) == 1
assert payload["items"][0]["job_key"] == ck_health.JOB_KEY
+1 -1
View File
@@ -22,7 +22,7 @@ TESTS_DIR = Path(__file__).resolve().parent
PROJECT_ROOT = TESTS_DIR.parent
VERSIONS_DIR = PROJECT_ROOT / "web" / "backend" / "migrations" / "versions"
HEAD_REVISION = "20261001_0036"
HEAD_REVISION = "20261002_0037"
# 迁移新增、但不声明在模型里的复合查询索引。
EXTRA_INDEXES = (
+16 -1
View File
@@ -20,6 +20,7 @@ from .routers import (
accounts,
audit,
auth,
automation,
cookies,
dashboard,
douyu,
@@ -75,7 +76,20 @@ async def lifespan(app: FastAPI):
logger.info(f"启动清理应用宝残留任务: {cleaned_yyb} 条")
finally:
db.close()
yield
# 自动化调度器:AUTOMATION_ENABLED 未开启时不会起线程(默认关闭)。
from .services.automation import (
register_default_jobs,
start_scheduler,
stop_scheduler,
)
register_default_jobs()
start_scheduler()
try:
yield
finally:
stop_scheduler()
app = FastAPI(
@@ -135,6 +149,7 @@ app.include_router(huya.router)
app.include_router(douyu.router)
app.include_router(yyb.router)
app.include_router(audit.router)
app.include_router(automation.router)
@app.get("/api/health")
@@ -0,0 +1,135 @@
"""自动化作业:作业定义表、运行记录表,以及 CK 巡检所需的退避字段。
P1a 只落地「CK 自动巡检」需要的表结构;重登退避(relogin_*)与账号级开关
(auto_managed)也一并建好,供 P1b 使用,避免之后再改一次表。
上一版登录任务把 attempt_* 与 CK 本体拆开,这里延续同一原则:巡检只写
ck_check_*,绝不碰 cookie / status / attempt_*。
"""
from collections.abc import Sequence
import sqlalchemy as sa
from alembic import op
revision: str = "20261002_0037"
down_revision: str | None = "20261001_0036"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _columns(bind, table: str) -> set[str]:
return {item["name"] for item in sa.inspect(bind).get_columns(table)}
def _tables(bind) -> set[str]:
return set(sa.inspect(bind).get_table_names())
def upgrade() -> None:
bind = op.get_bind()
existing_tables = _tables(bind)
if "automation_jobs" not in existing_tables:
op.create_table(
"automation_jobs",
sa.Column("key", sa.String(length=64), primary_key=True),
sa.Column(
"title", sa.String(length=128), nullable=False, server_default=""
),
sa.Column(
"enabled", sa.Boolean(), nullable=False, server_default=sa.text("0")
),
sa.Column(
"interval_seconds",
sa.Integer(),
nullable=False,
server_default="21600",
),
sa.Column("batch_size", sa.Integer(), nullable=False, server_default="200"),
sa.Column("concurrency", sa.Integer(), nullable=False, server_default="5"),
sa.Column("config", sa.JSON(), nullable=True),
sa.Column("last_run_at", sa.DateTime(), nullable=True),
sa.Column("next_run_at", sa.DateTime(), nullable=True),
sa.Column(
"last_status", sa.String(length=16), nullable=False, server_default=""
),
sa.Column(
"last_message", sa.String(length=512), nullable=False, server_default=""
),
sa.Column("updated_at", sa.DateTime(), nullable=False),
)
if "automation_runs" not in existing_tables:
op.create_table(
"automation_runs",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column("job_key", sa.String(length=64), nullable=False),
sa.Column("started_at", sa.DateTime(), nullable=False),
sa.Column("finished_at", sa.DateTime(), nullable=True),
sa.Column(
"status", sa.String(length=16), nullable=False, server_default="running"
),
sa.Column("scanned", sa.Integer(), nullable=False, server_default="0"),
sa.Column("valid", sa.Integer(), nullable=False, server_default="0"),
sa.Column("invalid", sa.Integer(), nullable=False, server_default="0"),
sa.Column(
"relogin_started", sa.Integer(), nullable=False, server_default="0"
),
sa.Column(
"relogin_success", sa.Integer(), nullable=False, server_default="0"
),
sa.Column("skipped", sa.Integer(), nullable=False, server_default="0"),
sa.Column(
"message", sa.String(length=512), nullable=False, server_default=""
),
sa.Column("detail", sa.JSON(), nullable=True),
)
op.create_index("ix_automation_runs_job_key", "automation_runs", ["job_key"])
login_columns = _columns(bind, "login_tasks")
if "relogin_attempts" not in login_columns:
op.add_column(
"login_tasks",
sa.Column(
"relogin_attempts", sa.Integer(), nullable=False, server_default="0"
),
)
if "relogin_next_at" not in login_columns:
op.add_column(
"login_tasks", sa.Column("relogin_next_at", sa.DateTime(), nullable=True)
)
if "relogin_last_error" not in login_columns:
op.add_column(
"login_tasks",
sa.Column(
"relogin_last_error",
sa.String(length=512),
nullable=False,
server_default="",
),
)
account_columns = _columns(bind, "accounts")
if "auto_managed" not in account_columns:
op.add_column(
"accounts",
sa.Column(
"auto_managed",
sa.Boolean(),
nullable=False,
server_default=sa.text("1"),
),
)
def downgrade() -> None:
bind = op.get_bind()
for table in ("automation_runs", "automation_jobs"):
if table in _tables(bind):
op.drop_table(table)
login_columns = _columns(bind, "login_tasks")
for name in ("relogin_last_error", "relogin_next_at", "relogin_attempts"):
if name in login_columns:
op.drop_column("login_tasks", name)
if "auto_managed" in _columns(bind, "accounts"):
op.drop_column("accounts", "auto_managed")
+60
View File
@@ -111,6 +111,10 @@ class Account(Base):
esports_can_change_time: Mapped[int | None] = mapped_column(Integer, nullable=True)
xpd_game_name: Mapped[str] = mapped_column(String(128), default="")
xpd_openid: Mapped[str] = mapped_column(String(128), default="")
# 自动化后门:false 时巡检与自动重登都跳过这个账号(特殊账号人工维护)。
auto_managed: Mapped[bool] = mapped_column(
Boolean, default=True, server_default="1"
)
xpd_role_id: Mapped[str] = mapped_column(String(64), default="")
xpd_plat_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
xpd_area_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
@@ -171,6 +175,14 @@ class LoginTask(Base):
String(64), default="", server_default=""
)
attempt_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
# 自动重登(P1b)用的退避状态。P1a 的巡检不碰这几个字段。
relogin_attempts: Mapped[int] = mapped_column(
Integer, default=0, server_default="0"
) # 连续重登尝试次数,成功即清零
relogin_next_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
relogin_last_error: Mapped[str] = mapped_column(
String(512), default="", server_default=""
)
account: Mapped[Account] = relationship("Account", back_populates="login_tasks")
@@ -673,3 +685,51 @@ class AuditLog(Base):
# 旧记录未标记结果时为 NULL,避免把历史操作误判成失败。
success: Mapped[bool | None] = mapped_column(Boolean, nullable=True, index=True)
created_at: Mapped[datetime] = mapped_column(DateTime, default=_utcnow)
class AutomationJob(Base):
"""自动化作业定义。状态落库,进程重启不丢调度进度。"""
__tablename__ = "automation_jobs"
key: Mapped[str] = mapped_column(String(64), primary_key=True)
title: Mapped[str] = mapped_column(String(128), default="")
# 默认关闭:按「渐进上线」原则,先观察再打开。
enabled: Mapped[bool] = mapped_column(Boolean, default=False, server_default="0")
interval_seconds: Mapped[int] = mapped_column(
Integer, default=21600, server_default="21600"
)
batch_size: Mapped[int] = mapped_column(Integer, default=200, server_default="200")
concurrency: Mapped[int] = mapped_column(Integer, default=5, server_default="5")
config: Mapped[dict | None] = mapped_column(JSON, nullable=True)
last_run_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
next_run_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
last_status: Mapped[str] = mapped_column(String(16), default="", server_default="")
last_message: Mapped[str] = mapped_column(
String(512), default="", server_default=""
)
updated_at: Mapped[datetime] = mapped_column(
DateTime, default=_utcnow, onupdate=_utcnow
)
class AutomationRun(Base):
"""自动化作业每一轮的执行记录,用于页面展示与排障。"""
__tablename__ = "automation_runs"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
job_key: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
started_at: Mapped[datetime] = mapped_column(DateTime, default=_utcnow)
finished_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
status: Mapped[str] = mapped_column(
String(16), default="running", server_default="running"
) # running / success / failed / skipped
scanned: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
valid: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
invalid: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
relogin_started: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
relogin_success: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
skipped: Mapped[int] = mapped_column(Integer, default=0, server_default="0")
message: Mapped[str] = mapped_column(String(512), default="", server_default="")
detail: Mapped[dict | None] = mapped_column(JSON, nullable=True)
+4
View File
@@ -51,6 +51,9 @@ PERMISSIONS = {
"proxy:manage": "代理配置管理",
"whitelist:manage": "白名单配置管理",
"whitelist:test": "测试白名单",
# 自动化
"automation:view": "查看自动化作业与运行历史",
"automation:manage": "修改自动化配置、手动触发作业",
# 系统
"system:settings": "系统设置",
"audit:view": "查看审计日志",
@@ -90,6 +93,7 @@ ROLE_PERMISSIONS = {
"proxy:manage",
"whitelist:manage",
"whitelist:test",
"automation:view",
],
"support": [
"account:view_assigned",
+191
View File
@@ -0,0 +1,191 @@
"""自动化作业接口。
设计上「先能看、再能开」:查看需要 ``automation:view``(运营也有),改配置与
手动触发需要 ``automation:manage``(仅超管)。作业默认关闭,打开后由进程内
调度器按间隔执行。
"""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from fastapi import APIRouter, Depends, HTTPException, Query
from pydantic import BaseModel, Field
from sqlalchemy.orm import Session
from ..database import get_db
from ..deps import require_permission
from ..models import AutomationJob, AutomationRun, User
from ..services.audit_service import record_audit
from ..services.automation import (
ensure_jobs,
registered_jobs,
run_job_now,
scheduler_status,
)
router = APIRouter(prefix="/api/automation", tags=["自动化"])
MAX_BATCH_SIZE = 2000
MIN_INTERVAL_SECONDS = 300
class JobUpdate(BaseModel):
"""作业配置更新;只传要改的字段。"""
enabled: bool | None = None
interval_seconds: int | None = Field(default=None, ge=MIN_INTERVAL_SECONDS)
batch_size: int | None = Field(default=None, ge=1, le=MAX_BATCH_SIZE)
concurrency: int | None = Field(default=None, ge=1, le=16)
def _job_out(job: AutomationJob) -> dict[str, Any]:
return {
"key": job.key,
"title": job.title,
"enabled": bool(job.enabled),
"interval_seconds": int(job.interval_seconds or 0),
"batch_size": int(job.batch_size or 0),
"concurrency": int(job.concurrency or 0),
"last_run_at": job.last_run_at.isoformat() if job.last_run_at else None,
"next_run_at": job.next_run_at.isoformat() if job.next_run_at else None,
"last_status": job.last_status or "",
"last_message": job.last_message or "",
"registered": job.key in registered_jobs(),
}
def _run_out(run: AutomationRun) -> dict[str, Any]:
return {
"id": run.id,
"job_key": run.job_key,
"started_at": run.started_at.isoformat() if run.started_at else None,
"finished_at": run.finished_at.isoformat() if run.finished_at else None,
"status": run.status or "",
"scanned": int(run.scanned or 0),
"valid": int(run.valid or 0),
"invalid": int(run.invalid or 0),
"skipped": int(run.skipped or 0),
"message": run.message or "",
"detail": run.detail or None,
}
@router.get("/jobs")
def list_jobs(
db: Session = Depends(get_db),
current: User = Depends(require_permission("automation:view")),
):
"""列出作业配置、最近一次结果与调度器状态。"""
ensure_jobs(db)
jobs = db.query(AutomationJob).order_by(AutomationJob.key.asc()).all()
latest: dict[str, AutomationRun] = {}
for run in (
db.query(AutomationRun).order_by(AutomationRun.id.desc()).limit(200).all()
):
latest.setdefault(run.job_key, run)
return {
"scheduler": scheduler_status(),
"jobs": [
{
**_job_out(job),
"latest_run": _run_out(latest[job.key]) if job.key in latest else None,
}
for job in jobs
],
}
@router.patch("/jobs/{key}")
def update_job(
key: str,
payload: JobUpdate,
db: Session = Depends(get_db),
current: User = Depends(require_permission("automation:manage")),
):
"""改开关 / 间隔 / 批量 / 并发。打开开关时排定下一轮时间。"""
ensure_jobs(db)
job = db.query(AutomationJob).filter(AutomationJob.key == key).first()
if job is None:
raise HTTPException(status_code=404, detail="作业不存在")
changed: dict[str, Any] = {}
if payload.enabled is not None and bool(payload.enabled) != bool(job.enabled):
job.enabled = bool(payload.enabled)
changed["enabled"] = job.enabled
# 刚打开时排一轮:不早于一个 tick,避免打开瞬间就压满 CPU。
job.next_run_at = datetime.now(UTC) + _tick_delta() if job.enabled else None
if payload.interval_seconds is not None:
changed["interval_seconds"] = payload.interval_seconds
job.interval_seconds = payload.interval_seconds
if payload.batch_size is not None:
changed["batch_size"] = payload.batch_size
job.batch_size = payload.batch_size
if payload.concurrency is not None:
changed["concurrency"] = payload.concurrency
job.concurrency = payload.concurrency
if changed:
job.updated_at = datetime.now(UTC)
record_audit(
db,
current,
action="automation:update",
target=key,
detail=changed,
)
db.commit()
return _job_out(job)
@router.post("/jobs/{key}/run")
def run_job(
key: str,
db: Session = Depends(get_db),
current: User = Depends(require_permission("automation:manage")),
):
"""立刻跑一轮(不看 enabled)。用于正式打开开关前先试跑。"""
ensure_jobs(db)
try:
outcome = run_job_now(db, key)
except KeyError:
raise HTTPException(status_code=404, detail="作业不存在") from None
record_audit(
db,
current,
action="automation:run",
target=key,
detail={
"status": outcome.get("status"),
"scanned": outcome.get("scanned"),
"valid": outcome.get("valid"),
"invalid": outcome.get("invalid"),
"message": outcome.get("message"),
},
)
db.commit()
return outcome
@router.get("/runs")
def list_runs(
job_key: str | None = Query(default=None),
limit: int = Query(default=50, ge=1, le=200),
db: Session = Depends(get_db),
current: User = Depends(require_permission("automation:view")),
):
"""运行历史,最近的在前。"""
query = db.query(AutomationRun)
if job_key:
query = query.filter(AutomationRun.job_key == job_key)
runs = query.order_by(AutomationRun.id.desc()).limit(limit).all()
return {"items": [_run_out(run) for run in runs]}
def _tick_delta():
from datetime import timedelta
from ..services.automation.scheduler import TICK_SECONDS
return timedelta(seconds=TICK_SECONDS)
+21
View File
@@ -13,6 +13,7 @@ from ..models import Account, LoginTask, User
from ..models import ProxyConfig as ProxyConfigModel
from ..permissions import get_user_permissions, user_has_permission
from ..schemas import LoginBatchRequest, LoginTaskOut
from ..services.audit_service import record_audit
from ..services.login_service import LoginBatchRunner, batch_registry
router = APIRouter(prefix="/api/login", tags=["登录任务"])
@@ -111,6 +112,26 @@ async def create_batch(
thread = threading.Thread(target=runner.run, daemon=True)
thread.start()
# 审计:批次是谁在什么时候发起、多少个账号、什么模式。
# 自动重登与人工登录共用同一条链路,没有这条记录就无法追溯来源。
try:
record_audit(
db,
current,
action="login:batch",
target=batch_id,
detail={
"mode": req.mode,
"requested": len(req.account_ids),
"accepted": len(valid_ids),
"concurrency": req.concurrency,
"api_strategy": req.api_strategy,
},
)
db.commit()
except Exception: # noqa: BLE001 审计失败不应影响已启动的批次
db.rollback()
return {"batch_id": batch_id, "count": len(valid_ids), "success": True}
@@ -0,0 +1,57 @@
"""自动化调度:作业注册、调度线程与各作业实现。
登记作业请放在 ``register_default_jobs()``,由 ``main.py`` 在启动时调用一次,
避免 import 顺序影响注册结果。
"""
from __future__ import annotations
from .jobs.ck_health import JOB_KEY as CK_HEALTH_KEY
from .jobs.ck_health import run_ck_health
from .scheduler import (
JobSpec,
automation_enabled,
ensure_jobs,
register_job,
registered_jobs,
reset_registry,
run_due_jobs,
run_job_now,
scheduler_status,
start_scheduler,
stop_scheduler,
)
__all__ = [
"CK_HEALTH_KEY",
"JobSpec",
"automation_enabled",
"ensure_jobs",
"register_default_jobs",
"register_job",
"registered_jobs",
"reset_registry",
"run_ck_health",
"run_due_jobs",
"run_job_now",
"scheduler_status",
"start_scheduler",
"stop_scheduler",
]
def register_default_jobs() -> None:
"""登记内置作业(幂等)。"""
register_job(
JobSpec(
key=CK_HEALTH_KEY,
title="CK 自动巡检",
# interval_seconds = 单账号多久重新检测一次(不是作业周期;
# 作业每个 tick 都跑)。全库 12,881 条按 5 分钟一轮、单轮 200 条
# 反推,吞吐 40 条/分钟,够 6 小时内覆盖一遍(规划 §7.3)。
interval_seconds=6 * 3600,
batch_size=200,
concurrency=5,
run=run_ck_health,
)
)
@@ -0,0 +1 @@
"""自动化作业实现。"""
@@ -0,0 +1,125 @@
"""CK 巡检作业(P1a)。
只做一件事:把「持有 CK 但久未检测」的账号挑出来,调现成的
``check_douyu_cookie()`` 检测,把结果写回 ``ck_check_*``。
**本作业不修改 CK 本身**:不写 ``cookie`` / ``status`` / ``message``,也不碰
``attempt_*``。P1a 上线后人工照旧可以点检测和重登,自动重登是 P1b 的事。
"""
from __future__ import annotations
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import UTC, datetime, timedelta
from typing import Any
from sqlalchemy import or_
from sqlalchemy.orm import Session
from ....models import Account, AutomationJob, LoginTask
from ...cookie_check_service import check_douyu_cookie
from ...douyu_service import cookie_operable_filter
JOB_KEY = "ck_health"
FLUSH_EVERY = 25
def _select_tasks(db: Session, job: AutomationJob, now: datetime) -> list[LoginTask]:
"""挑本轮要检测的记录。
口径与人工点检测一致(``cookie_operable_filter``:持有 CK 且没有在途尝试),
再加三条:距上次检测超过作业间隔、账号允许自动管理、最久没查的优先。
"""
cutoff = now - timedelta(seconds=max(int(job.interval_seconds or 0), 60))
return (
db.query(LoginTask)
.join(Account, LoginTask.account_id == Account.id)
.filter(cookie_operable_filter())
.filter(Account.auto_managed.is_(True))
.filter(
or_(
LoginTask.ck_checked_at.is_(None),
LoginTask.ck_checked_at < cutoff,
)
)
.order_by(LoginTask.ck_checked_at.asc().nullsfirst(), LoginTask.id.asc())
.limit(max(int(job.batch_size or 0), 1))
.all()
)
def _detect(task: LoginTask) -> dict[str, Any]:
try:
return check_douyu_cookie(task.cookie or "")
except Exception as exc: # noqa: BLE001 单个账号失败不影响整轮
return {
"valid": False,
"message": f"检测异常: {exc}",
"fish_ball": None,
"nickname": None,
"level": None,
"checked_at": datetime.now(UTC),
}
def run_ck_health(db: Session, job: AutomationJob) -> dict[str, Any]:
now = datetime.now(UTC)
tasks = _select_tasks(db, job, now)
if not tasks:
return {"scanned": 0, "valid": 0, "invalid": 0, "skipped": 0}
results: dict[int, dict[str, Any]] = {}
concurrency = max(min(int(job.concurrency or 1), 16), 1)
with ThreadPoolExecutor(max_workers=concurrency) as pool:
futures = {pool.submit(_detect, task): task for task in tasks}
for future in as_completed(futures):
task = futures[future]
results[task.id] = future.result()
# 只写检测结果三件套;cookie / status / attempt_* 一律不动。
valid = invalid = 0
for index, task in enumerate(tasks, start=1):
result = results.get(task.id)
if result is None:
continue
is_valid = bool(result.get("valid"))
if is_valid:
valid += 1
else:
invalid += 1
task.ck_check_status = "valid" if is_valid else "invalid"
task.ck_check_result = {
"fish_ball": result.get("fish_ball"),
"nickname": result.get("nickname"),
"level": result.get("level"),
"message": result.get("message", ""),
}
task.ck_checked_at = now
if index % FLUSH_EVERY == 0:
db.commit()
db.commit()
# 剩余待检数量只用于提示,不参与调度。
remaining = (
db.query(LoginTask)
.join(Account, LoginTask.account_id == Account.id)
.filter(cookie_operable_filter())
.filter(Account.auto_managed.is_(True))
.filter(
or_(
LoginTask.ck_checked_at.is_(None),
LoginTask.ck_checked_at
< now - timedelta(seconds=max(int(job.interval_seconds or 0), 60)),
)
)
.count()
)
message = f"本轮检测 {len(tasks)} 条,剩余待检 {remaining} 条"
return {
"scanned": len(tasks),
"valid": valid,
"invalid": invalid,
"skipped": 0,
"message": message,
"detail": {"remaining": remaining, "concurrency": concurrency},
}
@@ -0,0 +1,275 @@
"""进程内自动化调度器。
背景:后台的日常动作原本全靠人工点(判断 CK 失效、重登、收集资料、记台账)。
自动化按「渐进上线」推进,因此:
- 调度器总开关 ``AUTOMATION_ENABLED`` **默认关闭**;
- 每个作业自己还有开关,**默认关闭**,在 ``/automation`` 页面打开;
- 作业状态全部落库(``automation_jobs`` / ``automation_runs``),进程重启不丢
调度进度,也不会因为重启而重复跑一轮。
部署是单容器 ``--workers 1``,所以一个进程内线程就够,不需要分布式锁。
"""
from __future__ import annotations
import os
import threading
from collections.abc import Callable
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from typing import Any
from loguru import logger
from sqlalchemy.orm import Session
from ...database import SessionLocal
from ...models import AutomationJob, AutomationRun
# 调度轮询间隔,也是作业的运行节奏:每个 tick 跑一轮。
#
# 注意区分两个「间隔」:
# - **本常量**:作业多久跑一轮(5 分钟);
# - **job.interval_seconds**:同一行数据多久重新检测一次(默认 6h),只用于选行 cutoff。
#
# 分开的原因:全库 12,881 条的 CK 若想「6 小时内都被查一遍」,需要约 36 条/分钟的
# 吞吐;按 5 分钟一轮、单轮 200 条算 = 40 条/分钟,刚好够。若把作业周期也设成 6h,
# 就变成 200 条/6h = 800 条/天,全库要 16 天才能过一轮,等于没巡检。
TICK_SECONDS = 300
TRUTHY = {"1", "true", "yes", "on"}
@dataclass(frozen=True)
class JobSpec:
"""一个可调度的作业。``run`` 接收会话与作业行,返回本轮计数。"""
key: str
title: str
interval_seconds: int
batch_size: int
concurrency: int
run: Callable[[Session, AutomationJob], dict[str, Any]]
_REGISTRY: dict[str, JobSpec] = {}
_thread: threading.Thread | None = None
_stop_event = threading.Event()
_lock = threading.Lock()
def register_job(spec: JobSpec) -> None:
_REGISTRY[spec.key] = spec
def registered_jobs() -> dict[str, JobSpec]:
return dict(_REGISTRY)
def reset_registry() -> None:
"""清空作业注册表(测试隔离用;正常启动只调用 register_default_jobs)。"""
_REGISTRY.clear()
def automation_enabled() -> bool:
return os.getenv("AUTOMATION_ENABLED", "").strip().lower() in TRUTHY
def ensure_jobs(db: Session) -> None:
"""把注册表里的作业补进 ``automation_jobs``(幂等)。
只补缺失的行,**不覆盖已有配置** —— 运维在页面上改过的开关、间隔、批量
不能因为重启被重置回默认值。
"""
existing = {row.key for row in db.query(AutomationJob.key).all()}
for spec in _REGISTRY.values():
if spec.key in existing:
continue
db.add(
AutomationJob(
key=spec.key,
title=spec.title,
enabled=False,
interval_seconds=spec.interval_seconds,
batch_size=spec.batch_size,
concurrency=spec.concurrency,
config=None,
next_run_at=None,
last_status="",
last_message="",
updated_at=datetime.now(UTC),
)
)
db.commit()
def _as_utc(value: datetime | None) -> datetime | None:
"""DB 时间列存的是 naive UTC,读出来要补时区标记才能跟 now 比较。"""
if value is None:
return None
if value.tzinfo is None:
return value.replace(tzinfo=UTC)
return value.astimezone(UTC)
def _due(job: AutomationJob, now: datetime) -> bool:
"""到点判定:没排过时间视为到点(刚打开开关就会跑一轮)。"""
next_run_at = _as_utc(job.next_run_at)
return next_run_at is None or next_run_at <= now
def run_due_jobs(db: Session, *, now: datetime | None = None) -> list[dict[str, Any]]:
"""跑一轮「到点的作业」,返回每轮结果。
抽成普通函数是为了可测:测试直接调用即可,不必启动线程或等 60 秒。
作业串行执行 —— 目前只有 CK 巡检,单轮约 20~40 秒,远小于 tick,串行
反而省掉了并发调度的复杂度。
"""
now = now or datetime.now(UTC)
outcomes: list[dict[str, Any]] = []
jobs = (
db.query(AutomationJob)
.filter(AutomationJob.enabled.is_(True))
.order_by(AutomationJob.key.asc())
.all()
)
for job in jobs:
spec = _REGISTRY.get(job.key)
if spec is None or not _due(job, now):
continue
outcomes.append(_run_one(db, job, spec))
# 到期但没注册实现的作业(例如降级回滚后残留的行)只留日志,不静默消失。
for job in jobs:
if job.key not in _REGISTRY:
logger.warning("自动化作业 {} 没有注册实现,已跳过", job.key)
return outcomes
def _run_one(db: Session, job: AutomationJob, spec: JobSpec) -> dict[str, Any]:
started = datetime.now(UTC)
run = AutomationRun(job_key=job.key, started_at=started, status="running")
db.add(run)
db.commit()
counters: dict[str, Any] = {}
status = "success"
message = ""
try:
counters = spec.run(db, job) or {}
message = str(counters.pop("message", "") or "")
except Exception as exc: # noqa: BLE001 作业边界要兜住任何异常,不能拖垮调度线程
db.rollback()
status = "failed"
message = f"{type(exc).__name__}: {exc}"[:500]
logger.exception("自动化作业 {} 执行失败", job.key)
finished = datetime.now(UTC)
run.finished_at = finished
run.status = status
run.message = message
run.scanned = int(counters.get("scanned", 0) or 0)
run.valid = int(counters.get("valid", 0) or 0)
run.invalid = int(counters.get("invalid", 0) or 0)
run.relogin_started = int(counters.get("relogin_started", 0) or 0)
run.relogin_success = int(counters.get("relogin_success", 0) or 0)
run.skipped = int(counters.get("skipped", 0) or 0)
run.detail = counters.get("detail") or None
job.last_run_at = finished
# 下一轮固定在一个 tick 之后;interval_seconds 是「同一行多久重检」,
# 由作业自己在选行时用,不参与调度。
job.next_run_at = finished + timedelta(seconds=TICK_SECONDS)
job.last_status = status
job.last_message = message
job.updated_at = finished
db.commit()
logger.info(
"自动化作业 {} 完成:status={} scanned={} valid={} invalid={} skipped={}",
job.key,
status,
run.scanned,
run.valid,
run.invalid,
run.skipped,
)
return {
"job_key": job.key,
"status": status,
"message": message,
"scanned": run.scanned,
"valid": run.valid,
"invalid": run.invalid,
"skipped": run.skipped,
}
def run_job_now(db: Session, key: str) -> dict[str, Any]:
"""立刻跑一个作业,**不看 enabled 开关**。
页面上的「立即跑一次」用这个:运维需要能在正式打开开关之前先试一轮。
"""
job = db.query(AutomationJob).filter(AutomationJob.key == key).first()
spec = _REGISTRY.get(key)
if job is None or spec is None:
raise KeyError(key)
return _run_one(db, job, spec)
def _loop() -> None:
db = SessionLocal()
try:
ensure_jobs(db)
except Exception: # noqa: BLE001 初始化失败不应让线程静默退出
logger.exception("自动化作业初始化失败")
finally:
db.close()
logger.info("自动化调度器已启动,tick={}s", TICK_SECONDS)
while not _stop_event.wait(TICK_SECONDS):
db = SessionLocal()
try:
run_due_jobs(db)
except Exception: # noqa: BLE001 单轮异常不能让调度线程死掉
db.rollback()
logger.exception("自动化调度轮询异常")
finally:
db.close()
logger.info("自动化调度器已停止")
def start_scheduler() -> bool:
"""启动调度线程。返回是否真的启动了(未开启总开关时返回 False)。"""
global _thread
if not automation_enabled():
logger.info("AUTOMATION_ENABLED 未开启,自动化调度器不启动")
return False
with _lock:
if _thread is not None and _thread.is_alive():
return False
_stop_event.clear()
_thread = threading.Thread(
target=_loop, name="automation-scheduler", daemon=True
)
_thread.start()
return True
def stop_scheduler() -> None:
global _thread
with _lock:
if _thread is None:
return
_stop_event.set()
_thread.join(timeout=5)
_thread = None
def scheduler_status() -> dict[str, Any]:
return {
"enabled": automation_enabled(),
"running": _thread is not None and _thread.is_alive(),
"tick_seconds": TICK_SECONDS,
"registered": sorted(_REGISTRY),
}
+2
View File
@@ -15,6 +15,7 @@ const AccountCheckPage = lazy(() => import('./pages/AccountCheckPage'));
const AssignmentsPage = lazy(() => import('./pages/AssignmentsPage'));
const LoginTasksPage = lazy(() => import('./pages/LoginTasksPage'));
const ProxyPage = lazy(() => import('./pages/ProxyPage'));
const AutomationPage = lazy(() => import('./pages/AutomationPage'));
const UsersPage = lazy(() => import('./pages/UsersPage'));
const CookiePage = lazy(() => import('./pages/CookiePage'));
const CookieOperationsPage = lazy(() => import('./pages/CookieOperationsPage'));
@@ -100,6 +101,7 @@ function AppContent() {
<Route path="huya/browser-exchange" element={lazyRoute(<HuyaBrowserExchangePage />)} />
<Route path="yyb/recharge" element={lazyRoute(<YybRechargePage />)} />
<Route path="proxy" element={lazyRoute(<ProxyPage />)} />
<Route path="automation" element={lazyRoute(<AutomationPage />)} />
<Route path="users" element={lazyRoute(<UsersPage />)} />
<Route path="audit-logs" element={isSuperAdmin ? lazyRoute(<AuditLogsPage />) : <Navigate to="/" replace />} />
</Route>
+18
View File
@@ -0,0 +1,18 @@
import api from './client';
import type {
AutomationJobItem,
AutomationJobUpdate,
AutomationJobsResponse,
AutomationRunOutcome,
AutomationRunsResponse,
} from './types';
export const automationApi = {
jobs: () => api.get<AutomationJobsResponse, AutomationJobsResponse>('/automation/jobs'),
updateJob: (key: string, payload: AutomationJobUpdate) =>
api.patch<AutomationJobItem, AutomationJobItem>(`/automation/jobs/${key}`, payload),
runJob: (key: string) =>
api.post<AutomationRunOutcome, AutomationRunOutcome>(`/automation/jobs/${key}/run`),
runs: (params: { job_key?: string; limit?: number }) =>
api.get<AutomationRunsResponse, AutomationRunsResponse>('/automation/runs', { params }),
};
+1
View File
@@ -4,6 +4,7 @@ export { accountApi } from './accounts';
export { appApi } from './app';
export { auditApi } from './audit';
export { authApi } from './auth';
export { automationApi } from './automation';
export { cookieApi } from './cookies';
export { dashboardApi } from './dashboard';
export { douyuApi } from './douyu';
+64
View File
@@ -827,3 +827,67 @@ export interface PermissionsListResult {
permissions: Record<string, string>;
role_permissions: Record<string, string[]>;
}
// ==================== 自动化作业 ====================
export interface AutomationRunItem {
id: number;
job_key: string;
started_at: string | null;
finished_at: string | null;
status: string;
scanned: number;
valid: number;
invalid: number;
skipped: number;
message: string;
detail: Record<string, unknown> | null;
}
export interface AutomationJobItem {
key: string;
title: string;
enabled: boolean;
interval_seconds: number;
batch_size: number;
concurrency: number;
last_run_at: string | null;
next_run_at: string | null;
last_status: string;
last_message: string;
registered: boolean;
latest_run: AutomationRunItem | null;
}
export interface AutomationSchedulerStatus {
enabled: boolean;
running: boolean;
tick_seconds: number;
registered: string[];
}
export interface AutomationJobsResponse {
scheduler: AutomationSchedulerStatus;
jobs: AutomationJobItem[];
}
export interface AutomationRunsResponse {
items: AutomationRunItem[];
}
export interface AutomationJobUpdate {
enabled?: boolean;
interval_seconds?: number;
batch_size?: number;
concurrency?: number;
}
export interface AutomationRunOutcome {
job_key: string;
status: string;
message: string;
scanned: number;
valid: number;
invalid: number;
skipped: number;
}
+2 -1
View File
@@ -7,7 +7,7 @@ import {
MenuFoldOutlined, MenuUnfoldOutlined, SwapOutlined,
SunOutlined, MoonOutlined, DesktopOutlined, GiftOutlined, ApartmentOutlined, MobileOutlined, ShoppingCartOutlined,
BookOutlined, TrophyOutlined, ShopOutlined, GlobalOutlined,
SafetyCertificateOutlined, FileSearchOutlined,
SafetyCertificateOutlined, FileSearchOutlined, RobotOutlined,
} from '@ant-design/icons';
import { useNavigate, useLocation, Outlet } from 'react-router-dom';
import { getUser, clearAuth, type AuthUser } from '../store/auth';
@@ -75,6 +75,7 @@ export default function MainLayout({ onLogout }: { onLogout?: () => void }) {
}
if (can('cookie:operate')) {
douyuItems.push({ key: '/cookie-operations', label: 'CK 检测与重登', icon: <SafetyCertificateOutlined /> });
douyuItems.push({ key: '/automation', label: '自动巡检', icon: <RobotOutlined /> });
}
if (can('douyu:task')) {
douyuItems.push({ key: '/douyu/elite', label: '精英宝典', icon: <BookOutlined /> });
+281
View File
@@ -0,0 +1,281 @@
import { useCallback, useEffect, useState } from 'react';
import {
Alert,
Button,
Card,
Descriptions,
Form,
InputNumber,
Popconfirm,
Space,
Switch,
Table,
Tag,
Typography,
} from 'antd';
import type { ColumnsType } from 'antd/es/table';
import { ReloadOutlined, ThunderboltOutlined } from '@ant-design/icons';
import { message } from '../utils/antdMessage';
import { getErrorMessage } from '../utils/error';
import { automationApi, type AutomationJobItem, type AutomationRunItem } from '../api/modules';
import { usePermissions } from '../hooks/usePermissions';
import { formatTime } from '../utils/time';
const STATUS_TAG: Record<string, { color: string; text: string }> = {
success: { color: 'green', text: '成功' },
failed: { color: 'red', text: '失败' },
running: { color: 'blue', text: '执行中' },
skipped: { color: 'default', text: '已跳过' },
};
function secondsToHours(seconds: number): number {
return Math.round((seconds / 3600) * 10) / 10;
}
export default function AutomationPage() {
const [jobs, setJobs] = useState<AutomationJobItem[]>([]);
const [schedulerEnabled, setSchedulerEnabled] = useState(true);
const [schedulerRunning, setSchedulerRunning] = useState(false);
const [runs, setRuns] = useState<AutomationRunItem[]>([]);
const [loading, setLoading] = useState(false);
const [savingKey, setSavingKey] = useState<string | null>(null);
const [runningKey, setRunningKey] = useState<string | null>(null);
const [forms] = Form.useForm();
const { can } = usePermissions();
const canManage = can('automation:manage');
const load = useCallback(async () => {
setLoading(true);
try {
const data = await automationApi.jobs();
setJobs(data.jobs);
setSchedulerEnabled(data.scheduler.enabled);
setSchedulerRunning(data.scheduler.running);
const history = await automationApi.runs({ limit: 50 });
setRuns(history.items);
// 统一回填表单:initialValue 只在首次渲染生效,刷新后不会跟着变。
forms.setFieldsValue(
Object.fromEntries(
data.jobs.map((job) => [
job.key,
{
enabled: job.enabled,
interval_hours: secondsToHours(job.interval_seconds),
batch_size: job.batch_size,
concurrency: job.concurrency,
},
]),
),
);
} catch (e: unknown) {
message.error(getErrorMessage(e));
} finally {
setLoading(false);
}
}, [forms]);
useEffect(() => {
load();
}, [load]);
const saveJob = async (job: AutomationJobItem) => {
const values = forms.getFieldValue(job.key) || {};
setSavingKey(job.key);
try {
await automationApi.updateJob(job.key, {
enabled: values.enabled,
interval_seconds: Math.max(Math.round((values.interval_hours ?? 6) * 3600), 300),
batch_size: values.batch_size,
concurrency: values.concurrency,
});
message.success(`${job.title} 配置已保存`);
await load();
} catch (e: unknown) {
message.error(getErrorMessage(e));
} finally {
setSavingKey(null);
}
};
const runOnce = async (job: AutomationJobItem) => {
setRunningKey(job.key);
try {
const outcome = await automationApi.runJob(job.key);
message.success(
`跑完一轮:扫描 ${outcome.scanned} 条,有效 ${outcome.valid},失效 ${outcome.invalid}`,
);
await load();
} catch (e: unknown) {
message.error(getErrorMessage(e));
} finally {
setRunningKey(null);
}
};
const runColumns: ColumnsType<AutomationRunItem> = [
{
title: '开始时间',
dataIndex: 'started_at',
width: 180,
render: (value: string | null) => formatTime(value),
},
{
title: '作业',
dataIndex: 'job_key',
width: 120,
},
{
title: '结果',
dataIndex: 'status',
width: 90,
render: (value: string) => {
const tag = STATUS_TAG[value] ?? { color: 'default', text: value || '-' };
return <Tag color={tag.color}>{tag.text}</Tag>;
},
},
{ title: '扫描', dataIndex: 'scanned', width: 80 },
{ title: '有效', dataIndex: 'valid', width: 80 },
{
title: '失效',
dataIndex: 'invalid',
width: 80,
render: (value: number) => (value > 0 ? <Typography.Text type="warning">{value}</Typography.Text> : 0),
},
{ title: '耗时', width: 90, render: (_, row) => formatDuration(row.started_at, row.finished_at) },
{ title: '说明', dataIndex: 'message', ellipsis: true },
];
return (
<Space direction="vertical" size="middle" style={{ width: '100%' }}>
<Card
title="自动巡检"
extra={
<Button icon={<ReloadOutlined />} onClick={load} loading={loading}>
刷新
</Button>
}
>
{!schedulerEnabled && (
<Alert
type="warning"
showIcon
style={{ marginBottom: 16 }}
message="调度器总开关未打开"
description="服务端环境变量 AUTOMATION_ENABLED 未开启,打开下面的作业开关不会真正执行。需要先打开总开关并重启服务,再回到本页打开作业。"
/>
)}
{schedulerEnabled && !schedulerRunning && (
<Alert
type="warning"
showIcon
style={{ marginBottom: 16 }}
message="调度线程未运行"
description="总开关已开但调度线程不在运行状态,请检查服务日志或重启服务。"
/>
)}
<Alert
type="info"
showIcon
style={{ marginBottom: 16 }}
message="巡检只检测、不改 CK"
description="每 5 分钟跑一轮,每轮按「每轮条数」上限挑最久没检测的账号;「单账号重检间隔」指同一个账号最多多久重新检测一次。本作业只标记失效账号,不会替换任何 Cookie、也不会改账号状态,重登仍由人工在「CK 检测与重登」页面操作。"
/>
{jobs.map((job) => (
<Card key={job.key} type="inner" title={job.title} style={{ marginBottom: 16 }}>
<Descriptions size="small" column={2} style={{ marginBottom: 16 }}>
<Descriptions.Item label="最近运行">
{formatTime(job.last_run_at)}
</Descriptions.Item>
<Descriptions.Item label="下次运行">
{formatTime(job.next_run_at)}
</Descriptions.Item>
<Descriptions.Item label="最近结果">
{job.last_status ? (
<Tag color={STATUS_TAG[job.last_status]?.color ?? 'default'}>
{STATUS_TAG[job.last_status]?.text ?? job.last_status}
</Tag>
) : (
'-'
)}
</Descriptions.Item>
<Descriptions.Item label="最近说明">{job.last_message || '-'}</Descriptions.Item>
</Descriptions>
<Form form={forms} layout="inline" onFinish={() => saveJob(job)}>
<Form.Item
name={[job.key, 'enabled']}
label="启用"
valuePropName="checked"
>
<Switch disabled={!canManage} />
</Form.Item>
<Form.Item
name={[job.key, 'interval_hours']}
label="单账号重检间隔(小时)"
>
<InputNumber min={0.1} step={1} disabled={!canManage} />
</Form.Item>
<Form.Item name={[job.key, 'batch_size']} label="每轮条数">
<InputNumber min={1} max={2000} disabled={!canManage} />
</Form.Item>
<Form.Item name={[job.key, 'concurrency']} label="并发">
<InputNumber min={1} max={16} disabled={!canManage} />
</Form.Item>
<Form.Item>
<Space>
<Button
type="primary"
htmlType="submit"
disabled={!canManage}
loading={savingKey === job.key}
>
保存
</Button>
<Popconfirm
title="立刻跑一轮?"
description="不看启用开关,立即执行一次巡检。"
onConfirm={() => runOnce(job)}
disabled={!canManage}
>
<Button
icon={<ThunderboltOutlined />}
disabled={!canManage}
loading={runningKey === job.key}
>
立即跑一次
</Button>
</Popconfirm>
</Space>
</Form.Item>
</Form>
{!canManage && (
<Typography.Text type="secondary">
当前账号只能查看,修改配置需要 automation:manage 权限。
</Typography.Text>
)}
</Card>
))}
</Card>
<Card title="运行历史">
<Table
rowKey="id"
size="small"
loading={loading}
columns={runColumns}
dataSource={runs}
pagination={{ pageSize: 20, showSizeChanger: false }}
/>
</Card>
</Space>
);
}
function formatDuration(start: string | null, end: string | null): string {
if (!start || !end) return '-';
const ms = new Date(end).getTime() - new Date(start).getTime();
if (!Number.isFinite(ms) || ms < 0) return '-';
if (ms < 1000) return `${ms}ms`;
return `${(ms / 1000).toFixed(1)}s`;
}