fix(sizeof,parse): 尺寸链去静默兜底，解析报错可定位，补 C 解析器缺口
StructGen: _get_struct_size/_get_align 去 opaque→8/ptr_size、未知→4 兜底并补 VectorType；LiteralStructType.fields→.elements（llvmlite 无 fields，必然 AttributeError） HandlesClassDef: union 载荷尺寸经不存在的 ir.DataLayout 静默退化成 8 字节；HandlesExprCall/ExprBuiltin: sizeof 各静默 0 分支改 fail-fast，数组元素 sizeof 统一走 _get_struct_size HandlesFor: nonlocal struct/array 堆分配不再退化成 malloc(8) vast/vast_c: 补一元 * / & 解析，装饰器后 extern/export 分派修复，C 侧报错带 token 与行号；StubGen.Converter: 宏展开丢尾随空行修 .pyi 缩进非法 其余为 rawptr 迁移 / fail-fast 口径等历史改动一并入库
This commit is contained in:
TermiNexus committed
1 parent
53a345a59a
commit
c83fc752c6
53 files changed
+6238
-887
No files matched your search
@@ -0,0 +1,840 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""作用域感知静态扫描: `rawptr[<族>.<X>]` 变量上访问 <X> 不存在的字段（fast-fail 潜在点）。
|
||||
|
||||
族（Family）= 一个"类名限定前缀 + 类定义来源"。默认族表见 BuildFamilies；
|
||||
`rawptr[ast.X]` / `rawptr[llvmlite.X]` / `rawptr[GenTypes.X]` / `rawptr[HT.X]` /
|
||||
`rawptr[memhub.X]` 均可解析。另加载 linkedlist.vp 作为**支撑族**（空前缀 ''），
|
||||
供 `class ParamNode(GSListNode[ParamNode])` 这类**裸基类名**解析（否则 `.Next` 会误报）。
|
||||
|
||||
相对旧版的关键修正（旧版按“变量名全文件聚合 + 正则硬匹配”导致大量误报）:
|
||||
1) 作用域建模: 为 模块 / 函数(方法/嵌套函数) / lambda 建立独立作用域,
|
||||
类体作用域按 Python 语义对方法“穿透”; 属性访问按“该访问点所在作用域链”找
|
||||
**最近一次、且位置在访问点之前** 的声明（不是“全文件第一条”）。
|
||||
2) 形参（含 self 与 rawptr[X] 形参）计入声明; 无注解形参视为“遮蔽”外层同名变量。
|
||||
3) 重新赋值/重声明: AnnAssign / Assign(cast 形态) / 赋值继承(Name RHS) 都参与;
|
||||
非指针赋值记录为“已解析但非指针”=> 不再继承更早的类。
|
||||
4) 跳过非代码: 解析源码后, 注释/字符串/docstring 内容天然不产生 Attribute 节点
|
||||
（如 "%s.__iter__" 之类的字符串与三引号注释块不再命中）。
|
||||
5) 类成员表按族隔离: 基类先在**本族**找, 再找**支撑族**(''), 最后全局唯一名;
|
||||
同一基类名跨族重复时报 AMBIG（宁可 UNRESOLVED 也不猜）。
|
||||
6) `t.REnum`（tagged union）派生类: 变体字段合并进父类成员集
|
||||
（TPV 允许在父类指针上访问变体字段, 不合并会产生大量误报）。
|
||||
|
||||
前端 = **TPC 自研 vast**（`lib/vast`，与 StubGen/Converter.py 用的 `import vast as ast`
|
||||
是同一源文件; 不可用 CPython `ast`, 否则含 `list[X*]` 等 vast 专有写法的文件会整体
|
||||
SyntaxError 被跳过）。vast 与 CPython ast 的差异及本脚本的适配:
|
||||
A) `X*` / `X**` 被 vast 脱糖成 `X | t.CPtr`（`**` 加两层）, 其中合成的
|
||||
`t.CPtr` Attribute 节点**连 lineno 属性都没有**（编译器内部脱糖产物, 不是源码里的
|
||||
访问; 全树实测 555 个, 全部是 ('t','CPtr')）=> 计入 Uses 时用
|
||||
`getattr(Node,'lineno',None) is not None` 排除。
|
||||
B) vast 的 SyntaxError 可能不带 lineno（C 路径）=> 从报文 `at line N` 兜底提取。
|
||||
C) vast 的装饰器 `@t.NoVTable` 的 col_offset 是 `@` 的位置（CPython 是其后 1 列）;
|
||||
该 Attribute 的接收者 `t` 不是族指针, 判据不受影响。
|
||||
D) 其余节点类名/属性名/walk/iter_child_nodes 与 CPython ast 同构（已逐节点对照）。
|
||||
|
||||
输出: 文件:行号 / 变量 / 推导出的类 / 字段 / 声明来源行 / 判定, 并统计
|
||||
NOWHERE / OTHER / RESOLVED / UNRESOLVED, 并按族（ast/llvmlite/其它）分桶。
|
||||
|
||||
路径: 全部按**脚本自身位置**推导（ScriptDir = <RepoRoot>/TransPyC/Tools），
|
||||
不依赖 cwd；可用 --app-root / --includes / --out 覆盖。
|
||||
|
||||
用法:
|
||||
python TransPyC/Tools/ScanAstFieldMiss.py [--app-root DIR] [--includes DIR]
|
||||
[--out FILE] [--verbose] [--strict]
|
||||
python TransPyC/Tools/ScanAstFieldMiss.py --selftest [--includes DIR] [--out FILE]
|
||||
|
||||
默认值:
|
||||
--app-root <RepoRoot>/TransPyV/App
|
||||
--includes <RepoRoot>/includes
|
||||
--out <RepoRoot>/Debugger/ScanAstFieldMiss.txt
|
||||
|
||||
退出码:
|
||||
0 非 --strict 的正常扫描; 或 --strict 下 NOWHERE / OTHER / parsefail 三者全 0
|
||||
1 --strict 下 有 NOWHERE/OTHER 命中, **或 parsefail > 0**
|
||||
（parsefail 也算门禁失败: 文件被前端静默跳过 = 假绿, 见下）
|
||||
2 参数错误
|
||||
--selftest: 0 = 断言全通过; 1 = 有断言失败
|
||||
|
||||
--selftest 为什么必须有（历史教训）:
|
||||
上一版门禁的前端是 CPython `ast`, 对含 `list[X*]` / `rawptr[X]*` 等 vast 专有写法的
|
||||
文件整体 SyntaxError 跳过 ⇒ 曾出现 "OTHER=0 全绿" 而工作树里实际有 **18 处**
|
||||
`rawptr[llvmlite.Function].Ty` 真错（整批文件根本没被检查）。
|
||||
故 --selftest 内置合成用例（生成到临时目录、跑完即删）, 断言"**报出集合 == 预期集合**",
|
||||
覆盖 4 类正例（含 vast 专有 `rawptr[X]*` / `rawptr[X]**`）与 4 类负例
|
||||
（合法字段 / 跨函数同名各自声明 / docstring 与注释 / 返回值推断生效）,
|
||||
用来证明扫描器不是空转。
|
||||
"""
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
# ============================================================
|
||||
# 0. 路径推导（全部按脚本自身位置, 不依赖 cwd）
|
||||
# ScriptDir = <RepoRoot>/TransPyC/Tools
|
||||
# TpcRoot = <RepoRoot>/TransPyC （TransPyC 仓库根）
|
||||
# RepoRoot = <RepoRoot> （工作区根）
|
||||
# ============================================================
|
||||
ScriptDir = os.path.dirname(os.path.abspath(__file__))
|
||||
TpcRoot = os.path.dirname(ScriptDir)
|
||||
RepoRoot = os.path.dirname(TpcRoot)
|
||||
|
||||
DefaultAppRoot = os.path.join(RepoRoot, 'TransPyV', 'App')
|
||||
DefaultIncludesDir = os.path.join(RepoRoot, 'includes')
|
||||
DefaultOutFile = os.path.join(RepoRoot, 'Debugger', 'ScanAstFieldMiss.txt')
|
||||
|
||||
# ---- 前端: TPC 自研 vast（非 CPython ast）。需要 TransPyC 包根在 sys.path,
|
||||
# lib._bootstrap 会再补齐 lib/ 与 lib/includes（使 `import vast` 可用）。
|
||||
if TpcRoot not in sys.path:
|
||||
sys.path.insert(0, TpcRoot)
|
||||
from lib import vast as ast # noqa: E402
|
||||
from lib.core.SymbolUtils import ExpandSoftKeywords # noqa: E402
|
||||
|
||||
SupportPrefix = ''
|
||||
|
||||
# ============================================================
|
||||
# 1. 类成员表: 字段 / 方法 / 基类（按族隔离；每次扫描重建）
|
||||
# ============================================================
|
||||
FamilyTable = []
|
||||
SupportFiles = []
|
||||
FamilyNames = set()
|
||||
|
||||
ClassFields = {}
|
||||
ClassMethods = {}
|
||||
ClassBase = {}
|
||||
ClassLoc = {} # 裸类名 -> [族, ...]（用于跨族基类解析）
|
||||
EnumClass = {} # (族, 类名) -> 1 表示 t.REnum/t.CEnum 派生（变体字段合并）
|
||||
REnumClass = {} # (族, 类名) -> 1 表示 t.REnum 派生（额外有编译器内建 .kind()）
|
||||
|
||||
MemberCache = {}
|
||||
FamilyFields = {} # 族 -> 该族所有字段并集（OTHER/NOWHERE 判定用）
|
||||
AmbiguousBases = set()
|
||||
|
||||
Uses = [] # [(AttrNode, Scope)]；Walk 现场填充, 每个文件扫描前 clear
|
||||
|
||||
# ---- 返回注解表（同文件内 def f(...) -> rawptr[族.X]）：供 x = f() 形态的类型推导。
|
||||
# 仅补充覆盖（收紧），不改变任何既有判据：名字在文件内返回注解不一致时报歧义并放弃。
|
||||
FuncRet = {}
|
||||
|
||||
|
||||
def ResetTables():
|
||||
"""清空全部类表/缓存（供同一进程内多轮扫描复用, 如 --selftest）。"""
|
||||
global FamilyTable, SupportFiles, FamilyNames
|
||||
global ClassFields, ClassMethods, ClassBase, ClassLoc, EnumClass, REnumClass
|
||||
global MemberCache, FamilyFields, AmbiguousBases
|
||||
FamilyTable = []
|
||||
SupportFiles = []
|
||||
FamilyNames = set()
|
||||
ClassFields = {}
|
||||
ClassMethods = {}
|
||||
ClassBase = {}
|
||||
ClassLoc = {}
|
||||
EnumClass = {}
|
||||
REnumClass = {}
|
||||
MemberCache = {}
|
||||
FamilyFields = {}
|
||||
AmbiguousBases = set()
|
||||
FuncRet.clear()
|
||||
|
||||
|
||||
def AddClass(Fam, Name, Bases):
|
||||
ClassFields.setdefault((Fam, Name), set())
|
||||
ClassMethods.setdefault((Fam, Name), set())
|
||||
ClassLoc.setdefault(Name, [])
|
||||
if Fam not in ClassLoc[Name]:
|
||||
ClassLoc[Name].append(Fam)
|
||||
if Bases:
|
||||
ClassBase.setdefault((Fam, Name), set()).update(Bases)
|
||||
|
||||
|
||||
def BaseName(Node):
|
||||
if isinstance(Node, ast.Name):
|
||||
return Node.id
|
||||
if isinstance(Node, ast.Attribute):
|
||||
return Node.attr
|
||||
if isinstance(Node, ast.Subscript): # GSListNode[ParamNode] -> GSListNode
|
||||
return BaseName(Node.value)
|
||||
return None
|
||||
|
||||
|
||||
def CollectBody(Fam, Name, Body, Enum):
|
||||
"""收集类体成员；Enum=True 时把嵌套变体类的字段并入本类（REnum 语义）。"""
|
||||
for Item in Body:
|
||||
if isinstance(Item, ast.AnnAssign) and isinstance(Item.target, ast.Name):
|
||||
ClassFields[(Fam, Name)].add(Item.target.id)
|
||||
elif isinstance(Item, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||||
ClassMethods[(Fam, Name)].add(Item.name)
|
||||
elif Enum and isinstance(Item, ast.ClassDef):
|
||||
CollectBody(Fam, Name, Item.body, Enum) # 变体字段并入父类
|
||||
|
||||
|
||||
def RegisterClass(Fam, Node):
|
||||
Bases = set()
|
||||
for B in Node.bases:
|
||||
Nm = BaseName(B)
|
||||
if Nm:
|
||||
Bases.add(Nm)
|
||||
AddClass(Fam, Node.name, Bases)
|
||||
Enum = 1 if (Bases & {'REnum', 'CEnum'}) else 0
|
||||
if Enum:
|
||||
EnumClass[(Fam, Node.name)] = 1
|
||||
if 'REnum' in Bases:
|
||||
# REnum.kind() 是编译器内建（HandlesExprCall.py:3126 'kind' → __tag 读取）
|
||||
REnumClass[(Fam, Node.name)] = 1
|
||||
CollectBody(Fam, Node.name, Node.body, Enum)
|
||||
|
||||
|
||||
def ParseSrc(Src, Path):
|
||||
"""vast 前端解析入口（全脚本单一前端; 失败抛异常, 由调用方记账）。
|
||||
|
||||
`.vp` 源码须先过 Viper 软关键字层（`lib/core/SymbolUtils.ExpandSoftKeywords`：
|
||||
`fn/fun/func/function` → `def`、`enum X:` → `class X(t.CEnum):`），与
|
||||
`StubGen/Converter.py` / `Shell/TransPyC.py` 等既有前端入口同口径。
|
||||
否则含 `enum` 的 `.vp`（如 `includes/ast/base.vp:168`）会在此整体 SyntaxError，
|
||||
被 `LoadInclude` 的正则兜底或 `RunScan` 的 `parsefail` 记成"解析失败"——两者
|
||||
都会**静默漏掉该文件的类/字段**（本脚本的假绿来源）。`.py` 无糖，展开为空操作
|
||||
（`'enum' in Code` 门禁 + 行首锚定正则）。
|
||||
"""
|
||||
return ast.parse(ExpandSoftKeywords(Src), Path)
|
||||
|
||||
|
||||
def ErrLine(E):
|
||||
"""vast 的 SyntaxError 可能无 lineno（C 路径）=> 从报文 `at line N` 兜底提取。"""
|
||||
L = getattr(E, 'lineno', None)
|
||||
if L:
|
||||
return L
|
||||
M = re.search(r'at line (\d+)', str(E))
|
||||
return int(M.group(1)) if M else -1
|
||||
|
||||
|
||||
def LoadIncludeRegex(Src, Fam):
|
||||
"""ast.parse 失败（docstring 含 \\xNN 等）时的回退：缩进感知的类/字段/方法收集。"""
|
||||
Indent = {}
|
||||
Enum = {}
|
||||
for Ln in Src.splitlines():
|
||||
M = re.match(r'(\s*)class (\w+)(?:\[[^\]]*\])?(?:\(([^)]*)\))?\s*:', Ln)
|
||||
if M:
|
||||
Depth = len(M.group(1))
|
||||
Name = M.group(2)
|
||||
Bases = set()
|
||||
if M.group(3):
|
||||
for B in M.group(3).split(','):
|
||||
B = B.strip().split('.')[-1].split('[')[0].strip()
|
||||
if B:
|
||||
Bases.add(B)
|
||||
AddClass(Fam, Name, Bases)
|
||||
Indent[Name] = Depth
|
||||
Enum[Name] = 1 if (Bases & {'REnum', 'CEnum'}) else 0
|
||||
if Enum[Name]:
|
||||
EnumClass[(Fam, Name)] = 1
|
||||
continue
|
||||
M2 = re.match(r'(\s+)(\w+)\s*:\s', Ln)
|
||||
if M2 and Indent:
|
||||
Depth = len(M2.group(1))
|
||||
Owner = [N for N, D in Indent.items() if D < Depth]
|
||||
if Owner:
|
||||
P = max(Owner, key=lambda N: Indent[N])
|
||||
ClassFields[(Fam, P)].add(M2.group(2))
|
||||
continue
|
||||
M3 = re.match(r'(\s+)def (\w+)\(', Ln)
|
||||
if M3 and Indent:
|
||||
Depth = len(M3.group(1))
|
||||
Owner = [N for N, D in Indent.items() if D < Depth]
|
||||
if Owner:
|
||||
P = max(Owner, key=lambda N: Indent[N])
|
||||
ClassMethods[(Fam, P)].add(M3.group(2))
|
||||
|
||||
|
||||
def LoadInclude(Path, Fam):
|
||||
try:
|
||||
Src = open(Path, encoding='utf-8').read()
|
||||
except Exception:
|
||||
return
|
||||
try:
|
||||
Tree = ParseSrc(Src, Path)
|
||||
except Exception: # noqa: BLE001 （vast 失败可能是 SyntaxError/ValueError）
|
||||
LoadIncludeRegex(Src, Fam)
|
||||
return
|
||||
for Node in Tree.body:
|
||||
if isinstance(Node, ast.ClassDef):
|
||||
RegisterClass(Fam, Node)
|
||||
|
||||
|
||||
def BuildFamilies(AppRoot, IncludesDir):
|
||||
"""按 AppRoot/IncludesDir 建族表并加载类定义（含支撑族）。"""
|
||||
global FamilyTable, SupportFiles, FamilyNames
|
||||
FamilyTable = [
|
||||
('ast', [os.path.join(IncludesDir, 'ast')]),
|
||||
('llvmlite', [os.path.join(IncludesDir, 'llvmlite')]),
|
||||
('GenTypes', [os.path.join(AppRoot, 'lib', 'core', 'Gen', 'GenTypes.vp')]),
|
||||
('HT', [os.path.join(AppRoot, 'lib', 'core', 'Gen', 'Translator.vp')]),
|
||||
('memhub', [os.path.join(IncludesDir, 'memhub.vp')]),
|
||||
]
|
||||
SupportFiles = [os.path.join(IncludesDir, 'linkedlist.vp')]
|
||||
FamilyNames = set(P for P, _ in FamilyTable) | {''}
|
||||
for Prefix, Paths in FamilyTable:
|
||||
for P in Paths:
|
||||
if os.path.isdir(P):
|
||||
for F in sorted(glob.glob(os.path.join(P, '*.vp'))):
|
||||
LoadInclude(F, Prefix)
|
||||
else:
|
||||
LoadInclude(P, Prefix)
|
||||
for F in SupportFiles:
|
||||
LoadInclude(F, SupportPrefix)
|
||||
|
||||
|
||||
def ComputeFamilyFields():
|
||||
global FamilyFields
|
||||
FamilyFields = {}
|
||||
for (_F, _N) in ClassFields:
|
||||
FamilyFields.setdefault(_F, set()).update(ClassFields[(_F, _N)])
|
||||
|
||||
|
||||
def FindClass(Name, Prefer):
|
||||
"""裸类名 -> (族, 类名)。优先本族, 再支撑族, 最后全局唯一名。"""
|
||||
if (Prefer, Name) in ClassFields:
|
||||
return (Prefer, Name)
|
||||
if (SupportPrefix, Name) in ClassFields:
|
||||
return (SupportPrefix, Name)
|
||||
Locs = ClassLoc.get(Name, [])
|
||||
if len(Locs) == 1:
|
||||
return (Locs[0], Name)
|
||||
if len(Locs) > 1:
|
||||
AmbiguousBases.add(Name)
|
||||
return None
|
||||
|
||||
|
||||
def Members(Fam, Name, Seen=None):
|
||||
if Seen is None:
|
||||
Seen = set()
|
||||
Key = (Fam, Name)
|
||||
if Key in Seen:
|
||||
return set()
|
||||
Seen.add(Key)
|
||||
Out = set(ClassFields.get(Key, ())) | set(ClassMethods.get(Key, ()))
|
||||
if Key in REnumClass:
|
||||
Out.add('kind')
|
||||
for B in ClassBase.get(Key, ()):
|
||||
Loc = FindClass(B, Fam)
|
||||
if Loc is not None:
|
||||
if Loc in REnumClass:
|
||||
Out.add('kind')
|
||||
Out |= Members(Loc[0], Loc[1], Seen)
|
||||
return Out
|
||||
|
||||
|
||||
def Member(Fam, Name):
|
||||
Key = (Fam, Name)
|
||||
if Key not in MemberCache:
|
||||
MemberCache[Key] = Members(Fam, Name)
|
||||
return MemberCache[Key]
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 2. 类型推导: 从注解 / cast 调用里取 (族, 类名)
|
||||
# ============================================================
|
||||
def AnnClass(Ann):
|
||||
"""注解里若为 rawptr[<族>.<X>]（可被 `| t.CPtr` 包裹）返回 (族, X), 否则 None。"""
|
||||
if Ann is None:
|
||||
return None
|
||||
if isinstance(Ann, ast.BinOp) and isinstance(Ann.op, ast.BitOr):
|
||||
return AnnClass(Ann.left) or AnnClass(Ann.right)
|
||||
if isinstance(Ann, ast.Subscript):
|
||||
if isinstance(Ann.value, ast.Name) and Ann.value.id == 'rawptr':
|
||||
Sl = Ann.slice
|
||||
if isinstance(Sl, ast.Attribute) and isinstance(Sl.value, ast.Name) \
|
||||
and Sl.value.id in FamilyNames and Sl.value.id != '':
|
||||
return (Sl.value.id, Sl.attr)
|
||||
return None
|
||||
|
||||
|
||||
def CastClass(Val):
|
||||
"""`rawptr[<族>.<X>](...)` 形态的 cast 调用返回 (族, X), 否则 None。"""
|
||||
if isinstance(Val, ast.Call) and isinstance(Val.func, ast.Subscript):
|
||||
return AnnClass(Val.func)
|
||||
return None
|
||||
|
||||
|
||||
def CollectFuncRets(Tree):
|
||||
FuncRet.clear()
|
||||
Ambig = set()
|
||||
for Node in ast.walk(Tree):
|
||||
if isinstance(Node, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||||
C = AnnClass(Node.returns)
|
||||
if Node.name in FuncRet and FuncRet[Node.name] != C:
|
||||
Ambig.add(Node.name)
|
||||
else:
|
||||
FuncRet[Node.name] = C
|
||||
for N in Ambig:
|
||||
FuncRet.pop(N, None)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 3. 作用域模型
|
||||
# ============================================================
|
||||
class Scope:
|
||||
__slots__ = ('Kind', 'Parent', 'Decls')
|
||||
|
||||
def __init__(self, Kind, Parent):
|
||||
self.Kind = Kind # 'module' / 'func' / 'class'
|
||||
self.Parent = Parent
|
||||
self.Decls = {} # name -> [(Lineno, Col, Cls, DeclLine)]
|
||||
|
||||
def Add(self, Name, Lineno, Col, Cls):
|
||||
self.Decls.setdefault(Name, []).append((Lineno, Col, Cls, Lineno))
|
||||
|
||||
|
||||
def Resolve(Sc, Name, Lineno, Col):
|
||||
"""沿作用域链找 Name 在 (Lineno, Col) 之前最近一次的声明。
|
||||
|
||||
返回 (Cls, DeclLine); Cls 为 None 表示“已解析但非族指针”或“未找到”。
|
||||
类体作用域对嵌套函数穿透（Python 语义）。
|
||||
"""
|
||||
Cur = Sc
|
||||
First = True
|
||||
while Cur is not None:
|
||||
if Cur.Kind != 'class' or First:
|
||||
if Name in Cur.Decls:
|
||||
Best = None
|
||||
for D in Cur.Decls[Name]:
|
||||
if (D[0], D[1]) <= (Lineno, Col):
|
||||
if Best is None or (D[0], D[1]) > (Best[0], Best[1]):
|
||||
Best = D
|
||||
if Best is not None:
|
||||
return Best[2], Best[3]
|
||||
First = False
|
||||
Cur = Cur.Parent
|
||||
return None, None
|
||||
|
||||
|
||||
def Walk(Node, Sc):
|
||||
if Node is None:
|
||||
return
|
||||
if isinstance(Node, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||||
for D in Node.decorator_list:
|
||||
Walk(D, Sc)
|
||||
A = Node.args
|
||||
Formals = list(A.posonlyargs) + list(A.args) + list(A.kwonlyargs)
|
||||
for P in Formals:
|
||||
Walk(P.annotation, Sc)
|
||||
for D in A.defaults:
|
||||
Walk(D, Sc)
|
||||
for D in A.kw_defaults:
|
||||
Walk(D, Sc)
|
||||
if A.vararg is not None:
|
||||
Walk(A.vararg.annotation, Sc)
|
||||
if A.kwarg is not None:
|
||||
Walk(A.kwarg.annotation, Sc)
|
||||
Walk(Node.returns, Sc)
|
||||
FS = Scope('func', Sc)
|
||||
for P in Formals:
|
||||
FS.Add(P.arg, Node.lineno, 0, AnnClass(P.annotation))
|
||||
if A.vararg is not None:
|
||||
FS.Add(A.vararg.arg, Node.lineno, 0, AnnClass(A.vararg.annotation))
|
||||
if A.kwarg is not None:
|
||||
FS.Add(A.kwarg.arg, Node.lineno, 0, AnnClass(A.kwarg.annotation))
|
||||
for S in Node.body:
|
||||
Walk(S, FS)
|
||||
return
|
||||
if isinstance(Node, ast.Lambda):
|
||||
A = Node.args
|
||||
Formals = list(A.posonlyargs) + list(A.args) + list(A.kwonlyargs)
|
||||
for P in Formals:
|
||||
Walk(P.annotation, Sc)
|
||||
for D in A.defaults:
|
||||
Walk(D, Sc)
|
||||
for D in A.kw_defaults:
|
||||
Walk(D, Sc)
|
||||
LS = Scope('func', Sc)
|
||||
for P in Formals:
|
||||
LS.Add(P.arg, Node.lineno, 0, AnnClass(P.annotation))
|
||||
Walk(Node.body, LS)
|
||||
return
|
||||
if isinstance(Node, ast.ClassDef):
|
||||
for D in Node.decorator_list:
|
||||
Walk(D, Sc)
|
||||
for B in Node.bases:
|
||||
Walk(B, Sc)
|
||||
for K in Node.keywords:
|
||||
Walk(K.value, Sc)
|
||||
CS = Scope('class', Sc)
|
||||
for S in Node.body:
|
||||
Walk(S, CS)
|
||||
return
|
||||
if isinstance(Node, ast.Attribute):
|
||||
# 只收源码里真实存在的属性访问: vast 脱糖 `X*` 会合成 `t.CPtr` Attribute,
|
||||
# 该节点**连 lineno 属性都没有**（全树 555 个, 全为 ('t','CPtr')）, 不是源码
|
||||
# 访问、也不能参与位置比较 => 用 getattr 显式排除。
|
||||
if isinstance(Node.value, ast.Name) and getattr(Node, 'lineno', None) is not None:
|
||||
Uses.append((Node, Sc))
|
||||
Walk(Node.value, Sc)
|
||||
return
|
||||
if isinstance(Node, ast.AnnAssign):
|
||||
if isinstance(Node.target, ast.Name):
|
||||
Sc.Add(Node.target.id, Node.target.lineno, Node.target.col_offset,
|
||||
AnnClass(Node.annotation))
|
||||
elif isinstance(Node.target, ast.Attribute):
|
||||
Walk(Node.target, Sc) # `x.y: T = ...` 的属性访问
|
||||
Walk(Node.annotation, Sc)
|
||||
Walk(Node.value, Sc)
|
||||
return
|
||||
if isinstance(Node, ast.Assign):
|
||||
Cls = None
|
||||
V = Node.value
|
||||
if isinstance(V, ast.Call):
|
||||
Cls = CastClass(V)
|
||||
if Cls is None and isinstance(V.func, ast.Name):
|
||||
Cls = FuncRet.get(V.func.id)
|
||||
elif isinstance(V, ast.Name):
|
||||
Cls, _ = Resolve(Sc, V.id, V.lineno, V.col_offset)
|
||||
for T in Node.targets:
|
||||
if isinstance(T, ast.Name):
|
||||
Sc.Add(T.id, T.lineno, T.col_offset, Cls)
|
||||
# 赋值**目标**位置的属性访问也要计入（如 `vtable_node.Ty = ...`）
|
||||
Walk(T, Sc)
|
||||
Walk(Node.value, Sc)
|
||||
return
|
||||
if isinstance(Node, ast.AugAssign):
|
||||
if isinstance(Node.target, ast.Name):
|
||||
Sc.Add(Node.target.id, Node.target.lineno, Node.target.col_offset, None)
|
||||
Walk(Node.target, Sc)
|
||||
Walk(Node.value, Sc)
|
||||
return
|
||||
if isinstance(Node, (ast.For, ast.AsyncFor)):
|
||||
for T in ast.walk(Node.target):
|
||||
if isinstance(T, ast.Name):
|
||||
Sc.Add(T.id, Node.lineno, 0, None)
|
||||
Walk(Node.iter, Sc)
|
||||
for S in Node.body:
|
||||
Walk(S, Sc)
|
||||
for S in Node.orelse:
|
||||
Walk(S, Sc)
|
||||
return
|
||||
if isinstance(Node, ast.NamedExpr):
|
||||
if isinstance(Node.target, ast.Name):
|
||||
Sc.Add(Node.target.id, Node.target.lineno, Node.target.col_offset, None)
|
||||
Walk(Node.value, Sc)
|
||||
return
|
||||
for Ch in ast.iter_child_nodes(Node):
|
||||
Walk(Ch, Sc)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 4. 扫描（一轮 = ResetTables + BuildFamilies + 遍历 AppRoot）
|
||||
# ============================================================
|
||||
def BucketOf(Fam):
|
||||
return Fam if Fam in ('ast', 'llvmlite') else 'OTHER:' + (Fam or 'support')
|
||||
|
||||
|
||||
def Bump(D, Key):
|
||||
D[Key] = D.get(Key, 0) + 1
|
||||
|
||||
|
||||
def RunScan(AppRoot, IncludesDir, OutFile, Verbose):
|
||||
"""跑一轮扫描: 写报告文件 + 返回统计/命中集合（不打印）。"""
|
||||
ResetTables()
|
||||
BuildFamilies(AppRoot, IncludesDir)
|
||||
ComputeFamilyFields()
|
||||
|
||||
Nowhere = []
|
||||
Other = []
|
||||
ResolvedCount = 0
|
||||
UnresolvedCount = 0
|
||||
Files = sorted(glob.glob(os.path.join(AppRoot, '**', '*.vp'), recursive=True))
|
||||
ParseFail = []
|
||||
|
||||
for Path in Files:
|
||||
try:
|
||||
Src = open(Path, encoding='utf-8').read()
|
||||
except Exception:
|
||||
continue
|
||||
try:
|
||||
Tree = ParseSrc(Src, Path)
|
||||
except Exception as E: # noqa: BLE001 （vast 失败可能是 SyntaxError/ValueError）
|
||||
ParseFail.append((os.path.relpath(Path, AppRoot), ErrLine(E), str(E)[:80]))
|
||||
continue
|
||||
RawLines = Src.splitlines()
|
||||
Uses.clear()
|
||||
CollectFuncRets(Tree)
|
||||
MS = Scope('module', None)
|
||||
for S in Tree.body:
|
||||
Walk(S, MS)
|
||||
|
||||
for AttrNode, Sc in Uses:
|
||||
Var = AttrNode.value.id
|
||||
Field = AttrNode.attr
|
||||
Cls, DeclLine = Resolve(Sc, Var, AttrNode.lineno, AttrNode.col_offset)
|
||||
if Cls is None:
|
||||
UnresolvedCount += 1
|
||||
continue
|
||||
Fam, CName = Cls
|
||||
if (Fam, CName) not in ClassFields:
|
||||
UnresolvedCount += 1
|
||||
continue
|
||||
ResolvedCount += 1
|
||||
if Field in Member(Fam, CName):
|
||||
continue
|
||||
Rel = os.path.relpath(Path, AppRoot)
|
||||
SrcLine = RawLines[AttrNode.lineno - 1].strip()[:96] if AttrNode.lineno <= len(RawLines) else ''
|
||||
Row = (Rel, AttrNode.lineno, Var, CName, Field, DeclLine, SrcLine, Fam)
|
||||
if Field in FamilyFields.get(Fam, set()):
|
||||
Other.append(Row)
|
||||
else:
|
||||
Nowhere.append(Row)
|
||||
|
||||
# 稳定排序（便于 diff / 门禁）
|
||||
SortKey = lambda R: (R[0], R[1], R[2], R[4], R[7], R[3], R[5])
|
||||
Nowhere.sort(key=SortKey)
|
||||
Other.sort(key=SortKey)
|
||||
|
||||
BucketsN = {}
|
||||
BucketsO = {}
|
||||
for R in Nowhere:
|
||||
Bump(BucketsN, BucketOf(R[7]))
|
||||
for R in Other:
|
||||
Bump(BucketsO, BucketOf(R[7]))
|
||||
|
||||
ParsedN = len(Files) - len(ParseFail)
|
||||
UsesN = ResolvedCount + UnresolvedCount
|
||||
ClassLine = ' '.join('%s=%d' % (f, sum(1 for (x, _n) in ClassFields if x == f))
|
||||
for f in sorted(set(x for x, _n in ClassFields)))
|
||||
|
||||
def W(Fh, Rows, Tag):
|
||||
for R in Rows:
|
||||
Fh.write('%-46s:%-5d var=%s cls=%s.%s field=%s decl@%d 判定=%s %s\n' %
|
||||
(R[0], R[1], R[2], R[7], R[3], R[4], R[5], Tag, R[6]))
|
||||
|
||||
OutDir = os.path.dirname(os.path.abspath(OutFile))
|
||||
if OutDir:
|
||||
os.makedirs(OutDir, exist_ok=True)
|
||||
with open(OutFile, 'w', encoding='utf-8') as Fh:
|
||||
Fh.write('files=%d parsed=%d parsefail=%d uses=%d NOWHERE=%d OTHER=%d RESOLVED=%d UNRESOLVED=%d\n' %
|
||||
(len(Files), ParsedN, len(ParseFail), UsesN,
|
||||
len(Nowhere), len(Other), ResolvedCount, UnresolvedCount))
|
||||
Fh.write('classes: %s\n' % ClassLine)
|
||||
Fh.write('NOWHERE buckets: %s\n' % sorted(BucketsN.items()))
|
||||
Fh.write('OTHER buckets: %s\n' % sorted(BucketsO.items()))
|
||||
if AmbiguousBases:
|
||||
Fh.write('AMBIGUOUS BASES (跨族重名, 已不解析): %s\n' % sorted(AmbiguousBases))
|
||||
Fh.write('\n=== NOWHERE (本族任何类都没有该字段) ===\n')
|
||||
W(Fh, Nowhere, 'NOWHERE')
|
||||
Fh.write('\n=== OTHER (字段只存在于本族别的类) ===\n')
|
||||
W(Fh, Other, 'OTHER')
|
||||
if ParseFail:
|
||||
Fh.write('\n=== PARSE FAIL (vast 无法解析, 已跳过) ===\n')
|
||||
for P in ParseFail:
|
||||
Fh.write('%s:%s %s\n' % P)
|
||||
|
||||
return {
|
||||
'Files': len(Files), 'Parsed': ParsedN, 'ParseFail': ParseFail,
|
||||
'Uses': UsesN, 'Nowhere': Nowhere, 'Other': Other,
|
||||
'Resolved': ResolvedCount, 'Unresolved': UnresolvedCount,
|
||||
'BucketsN': BucketsN, 'BucketsO': BucketsO,
|
||||
'Ambiguous': sorted(AmbiguousBases), 'Classes': ClassLine,
|
||||
'OutFile': OutFile,
|
||||
}
|
||||
|
||||
|
||||
def PrintSummary(Stats, Verbose):
|
||||
print('files=%d parsed=%d parsefail=%d uses=%d NOWHERE=%d OTHER=%d RESOLVED=%d UNRESOLVED=%d -> %s' %
|
||||
(Stats['Files'], Stats['Parsed'], len(Stats['ParseFail']), Stats['Uses'],
|
||||
len(Stats['Nowhere']), len(Stats['Other']),
|
||||
Stats['Resolved'], Stats['Unresolved'], Stats['OutFile']))
|
||||
print('NOWHERE buckets: %s' % sorted(Stats['BucketsN'].items()))
|
||||
print('OTHER buckets: %s' % sorted(Stats['BucketsO'].items()))
|
||||
if Stats['Ambiguous']:
|
||||
print('AMBIGUOUS BASES: %s' % Stats['Ambiguous'])
|
||||
if Stats['ParseFail']:
|
||||
print('PARSE FAIL: %s' % Stats['ParseFail'])
|
||||
Limit = 25 if not Verbose else 100000
|
||||
for R in Stats['Nowhere'][:Limit]:
|
||||
print('N %-40s:%-5d %s.%s.%s [decl@%d]' % (R[0], R[1], R[7], R[3], R[4], R[5]))
|
||||
for R in Stats['Other'][:Limit]:
|
||||
print('O %-40s:%-5d %s.%s.%s [decl@%d]' % (R[0], R[1], R[7], R[3], R[4], R[5]))
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 5. --selftest: 内置合成用例（临时目录生成, 跑完即删）
|
||||
# 断言"报出集合 == 预期集合"；正例 6 条（含 vast 专有写法）、负例 4 个文件。
|
||||
# ============================================================
|
||||
SelftestCases = [
|
||||
('CaseP1AstNameOther.vp', """import ast
|
||||
|
||||
|
||||
def ProbeP1(x_r: rawptr[ast.AST]) -> int:
|
||||
return x_r.name
|
||||
"""),
|
||||
('CaseP2LlvFunctionTyOther.vp', """import llvmlite
|
||||
|
||||
|
||||
def ProbeP2(x_r: rawptr[llvmlite.Function]) -> int:
|
||||
return x_r.Ty
|
||||
"""),
|
||||
('CaseP3AstNowhere.vp', """import ast
|
||||
|
||||
|
||||
def ProbeP3(x_r: rawptr[ast.AST]) -> int:
|
||||
return x_r.zzz_not_a_field
|
||||
"""),
|
||||
('CaseP4AstStarOther.vp', """import ast
|
||||
|
||||
|
||||
def ProbeP4(x_r: rawptr[ast.AST]*) -> int:
|
||||
return x_r.name
|
||||
"""),
|
||||
('CaseP5AstStarStarOther.vp', """import ast
|
||||
|
||||
|
||||
def ProbeP5(x_r: rawptr[ast.AST]**) -> int:
|
||||
return x_r.name
|
||||
"""),
|
||||
('CaseP6ReturnInferOther.vp', """import llvmlite
|
||||
|
||||
|
||||
def MkP6(x_r: rawptr[llvmlite.Function]) -> rawptr[llvmlite.Function]:
|
||||
return x_r
|
||||
|
||||
|
||||
def ProbeP6(x_r: rawptr[llvmlite.Function]) -> int:
|
||||
y_r = MkP6(x_r)
|
||||
return y_r.Ty
|
||||
"""),
|
||||
('CaseN1ParamNodeTy.vp', """import llvmlite
|
||||
|
||||
|
||||
def ProbeN1(x_r: rawptr[llvmlite.ParamNode]) -> int:
|
||||
return x_r.Ty
|
||||
"""),
|
||||
('CaseN2SameNameScopes.vp', """import llvmlite
|
||||
|
||||
|
||||
def ProbeN2A(x_r: rawptr[llvmlite.ParamNode]) -> int:
|
||||
return x_r.Ty
|
||||
|
||||
|
||||
def ProbeN2B(x_r: rawptr[llvmlite.Function]) -> int:
|
||||
return x_r.Ty
|
||||
"""),
|
||||
('CaseN3DocComment.vp', """import ast
|
||||
|
||||
|
||||
def ProbeN3(x_r: rawptr[ast.AST]) -> int:
|
||||
\"\"\"docstring: x_r.name / x_r.zzz_not_a_field 只是说明文字, 不是代码。\"\"\"
|
||||
# 注释: x_r.name / x_r.zzz_not_a_field
|
||||
return 0
|
||||
"""),
|
||||
('CaseN4ReturnInferOk.vp', """import llvmlite
|
||||
|
||||
|
||||
def MkN4(x_r: rawptr[llvmlite.ParamNode]) -> rawptr[llvmlite.ParamNode]:
|
||||
return x_r
|
||||
|
||||
|
||||
def ProbeN4(x_r: rawptr[llvmlite.ParamNode]) -> int:
|
||||
y_r = MkN4(x_r)
|
||||
return y_r.Ty
|
||||
"""),
|
||||
]
|
||||
|
||||
# (文件, 变量, 族.类, 字段) -> 判定
|
||||
SelftestExpected = {
|
||||
('CaseP1AstNameOther.vp', 'x_r', 'ast.AST', 'name'): 'OTHER',
|
||||
('CaseP2LlvFunctionTyOther.vp', 'x_r', 'llvmlite.Function', 'Ty'): 'OTHER',
|
||||
('CaseP3AstNowhere.vp', 'x_r', 'ast.AST', 'zzz_not_a_field'): 'NOWHERE',
|
||||
('CaseP4AstStarOther.vp', 'x_r', 'ast.AST', 'name'): 'OTHER',
|
||||
('CaseP5AstStarStarOther.vp', 'x_r', 'ast.AST', 'name'): 'OTHER',
|
||||
('CaseP6ReturnInferOther.vp', 'y_r', 'llvmlite.Function', 'Ty'): 'OTHER',
|
||||
('CaseN2SameNameScopes.vp', 'x_r', 'llvmlite.Function', 'Ty'): 'OTHER',
|
||||
}
|
||||
|
||||
|
||||
def SelfTest(IncludesDir, OutFile):
|
||||
AppRoot = tempfile.mkdtemp(prefix='ScanAstFieldMissSelftest.')
|
||||
try:
|
||||
for Name, Src in SelftestCases:
|
||||
with open(os.path.join(AppRoot, Name), 'w', encoding='utf-8', newline='\n') as Fh:
|
||||
Fh.write(Src)
|
||||
Report = OutFile or os.path.join(AppRoot, 'Selftest.txt')
|
||||
Stats = RunScan(AppRoot, IncludesDir, Report, False)
|
||||
|
||||
Reported = {}
|
||||
for R in Stats['Nowhere']:
|
||||
Reported[(R[0], R[2], '%s.%s' % (R[7], R[3]), R[4])] = 'NOWHERE'
|
||||
for R in Stats['Other']:
|
||||
Reported[(R[0], R[2], '%s.%s' % (R[7], R[3]), R[4])] = 'OTHER'
|
||||
|
||||
Ok = True
|
||||
if Stats['ParseFail']:
|
||||
Ok = False
|
||||
print('[selftest] 解析失败（预期 parsefail=0）:')
|
||||
for P in Stats['ParseFail']:
|
||||
print(' %s:%s %s' % P)
|
||||
Missing = sorted(set(SelftestExpected) - set(Reported))
|
||||
Extra = sorted(set(Reported) - set(SelftestExpected))
|
||||
TagDiff = sorted(K for K in set(Reported) & set(SelftestExpected)
|
||||
if Reported[K] != SelftestExpected[K])
|
||||
for K in Missing:
|
||||
Ok = False
|
||||
print('[selftest] 漏报: %s -> 预期 %s' % (K, SelftestExpected[K]))
|
||||
for K in Extra:
|
||||
Ok = False
|
||||
print('[selftest] 误报: %s -> %s' % (K, Reported[K]))
|
||||
for K in TagDiff:
|
||||
Ok = False
|
||||
print('[selftest] 判定不符: %s -> %s（预期 %s）' % (K, Reported[K], SelftestExpected[K]))
|
||||
|
||||
print('[selftest] files=%d parsed=%d parsefail=%d NOWHERE=%d OTHER=%d 报出=%d 预期=%d' %
|
||||
(Stats['Files'], Stats['Parsed'], len(Stats['ParseFail']),
|
||||
len(Stats['Nowhere']), len(Stats['Other']), len(Reported), len(SelftestExpected)))
|
||||
print('[selftest] %s' % ('PASS' if Ok else 'FAIL'))
|
||||
return 0 if Ok else 1
|
||||
finally:
|
||||
shutil.rmtree(AppRoot, ignore_errors=True)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 6. CLI
|
||||
# ============================================================
|
||||
def ParseArgs(Argv):
|
||||
Opt = {'AppRoot': DefaultAppRoot, 'Includes': DefaultIncludesDir, 'Out': None,
|
||||
'Strict': False, 'Verbose': False, 'SelfTest': False}
|
||||
Need = {'--app-root': 'AppRoot', '--includes': 'Includes', '--out': 'Out'}
|
||||
Flags = {'--strict': 'Strict', '--verbose': 'Verbose', '--selftest': 'SelfTest'}
|
||||
I = 0
|
||||
while I < len(Argv):
|
||||
A = Argv[I]
|
||||
if A in Need:
|
||||
if I + 1 >= len(Argv):
|
||||
sys.stderr.write('参数缺值: %s\n' % A)
|
||||
return None
|
||||
Opt[Need[A]] = Argv[I + 1]
|
||||
I += 2
|
||||
continue
|
||||
if A in Flags:
|
||||
Opt[Flags[A]] = True
|
||||
I += 1
|
||||
continue
|
||||
sys.stderr.write('未知参数: %s（用法见本脚本 docstring）\n' % A)
|
||||
return None
|
||||
return Opt
|
||||
|
||||
|
||||
def main():
|
||||
Opt = ParseArgs(sys.argv[1:])
|
||||
if Opt is None:
|
||||
return 2
|
||||
if Opt['SelfTest']:
|
||||
return SelfTest(Opt['Includes'], Opt['Out'])
|
||||
Stats = RunScan(Opt['AppRoot'], Opt['Includes'], Opt['Out'] or DefaultOutFile, Opt['Verbose'])
|
||||
PrintSummary(Stats, Opt['Verbose'])
|
||||
if Opt['Strict'] and (Stats['Nowhere'] or Stats['Other'] or Stats['ParseFail']):
|
||||
sys.stderr.write('[strict] 门禁失败: NOWHERE=%d OTHER=%d parsefail=%d\n' %
|
||||
(len(Stats['Nowhere']), len(Stats['Other']), len(Stats['ParseFail'])))
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user