TermiNexus
c83fc752c6
StructGen: _get_struct_size/_get_align 去 opaque→8/ptr_size、未知→4 兜底并补 VectorType；LiteralStructType.fields→.elements（llvmlite 无 fields，必然 AttributeError） HandlesClassDef: union 载荷尺寸经不存在的 ir.DataLayout 静默退化成 8 字节；HandlesExprCall/ExprBuiltin: sizeof 各静默 0 分支改 fail-fast，数组元素 sizeof 统一走 _get_struct_size HandlesFor: nonlocal struct/array 堆分配不再退化成 malloc(8) vast/vast_c: 补一元 * / & 解析，装饰器后 extern/export 分派修复，C 侧报错带 token 与行号；StubGen.Converter: 宏展开丢尾随空行修 .pyi 缩进非法 其余为 rawptr 迁移 / fail-fast 口径等历史改动一并入库
841 lines
32 KiB
Python
841 lines
32 KiB
Python
# -*- coding: utf-8 -*-
|
||
"""作用域感知静态扫描: `rawptr[<族>.<X>]` 变量上访问 <X> 不存在的字段（fast-fail 潜在点）。
|
||
|
||
族（Family）= 一个"类名限定前缀 + 类定义来源"。默认族表见 BuildFamilies；
|
||
`rawptr[ast.X]` / `rawptr[llvmlite.X]` / `rawptr[GenTypes.X]` / `rawptr[HT.X]` /
|
||
`rawptr[memhub.X]` 均可解析。另加载 linkedlist.vp 作为**支撑族**（空前缀 ''），
|
||
供 `class ParamNode(GSListNode[ParamNode])` 这类**裸基类名**解析（否则 `.Next` 会误报）。
|
||
|
||
相对旧版的关键修正（旧版按“变量名全文件聚合 + 正则硬匹配”导致大量误报）:
|
||
1) 作用域建模: 为 模块 / 函数(方法/嵌套函数) / lambda 建立独立作用域,
|
||
类体作用域按 Python 语义对方法“穿透”; 属性访问按“该访问点所在作用域链”找
|
||
**最近一次、且位置在访问点之前** 的声明（不是“全文件第一条”）。
|
||
2) 形参（含 self 与 rawptr[X] 形参）计入声明; 无注解形参视为“遮蔽”外层同名变量。
|
||
3) 重新赋值/重声明: AnnAssign / Assign(cast 形态) / 赋值继承(Name RHS) 都参与;
|
||
非指针赋值记录为“已解析但非指针”=> 不再继承更早的类。
|
||
4) 跳过非代码: 解析源码后, 注释/字符串/docstring 内容天然不产生 Attribute 节点
|
||
（如 "%s.__iter__" 之类的字符串与三引号注释块不再命中）。
|
||
5) 类成员表按族隔离: 基类先在**本族**找, 再找**支撑族**(''), 最后全局唯一名;
|
||
同一基类名跨族重复时报 AMBIG（宁可 UNRESOLVED 也不猜）。
|
||
6) `t.REnum`（tagged union）派生类: 变体字段合并进父类成员集
|
||
（TPV 允许在父类指针上访问变体字段, 不合并会产生大量误报）。
|
||
|
||
前端 = **TPC 自研 vast**（`lib/vast`，与 StubGen/Converter.py 用的 `import vast as ast`
|
||
是同一源文件; 不可用 CPython `ast`, 否则含 `list[X*]` 等 vast 专有写法的文件会整体
|
||
SyntaxError 被跳过）。vast 与 CPython ast 的差异及本脚本的适配:
|
||
A) `X*` / `X**` 被 vast 脱糖成 `X | t.CPtr`（`**` 加两层）, 其中合成的
|
||
`t.CPtr` Attribute 节点**连 lineno 属性都没有**（编译器内部脱糖产物, 不是源码里的
|
||
访问; 全树实测 555 个, 全部是 ('t','CPtr')）=> 计入 Uses 时用
|
||
`getattr(Node,'lineno',None) is not None` 排除。
|
||
B) vast 的 SyntaxError 可能不带 lineno（C 路径）=> 从报文 `at line N` 兜底提取。
|
||
C) vast 的装饰器 `@t.NoVTable` 的 col_offset 是 `@` 的位置（CPython 是其后 1 列）;
|
||
该 Attribute 的接收者 `t` 不是族指针, 判据不受影响。
|
||
D) 其余节点类名/属性名/walk/iter_child_nodes 与 CPython ast 同构（已逐节点对照）。
|
||
|
||
输出: 文件:行号 / 变量 / 推导出的类 / 字段 / 声明来源行 / 判定, 并统计
|
||
NOWHERE / OTHER / RESOLVED / UNRESOLVED, 并按族（ast/llvmlite/其它）分桶。
|
||
|
||
路径: 全部按**脚本自身位置**推导（ScriptDir = <RepoRoot>/TransPyC/Tools），
|
||
不依赖 cwd；可用 --app-root / --includes / --out 覆盖。
|
||
|
||
用法:
|
||
python TransPyC/Tools/ScanAstFieldMiss.py [--app-root DIR] [--includes DIR]
|
||
[--out FILE] [--verbose] [--strict]
|
||
python TransPyC/Tools/ScanAstFieldMiss.py --selftest [--includes DIR] [--out FILE]
|
||
|
||
默认值:
|
||
--app-root <RepoRoot>/TransPyV/App
|
||
--includes <RepoRoot>/includes
|
||
--out <RepoRoot>/Debugger/ScanAstFieldMiss.txt
|
||
|
||
退出码:
|
||
0 非 --strict 的正常扫描; 或 --strict 下 NOWHERE / OTHER / parsefail 三者全 0
|
||
1 --strict 下 有 NOWHERE/OTHER 命中, **或 parsefail > 0**
|
||
（parsefail 也算门禁失败: 文件被前端静默跳过 = 假绿, 见下）
|
||
2 参数错误
|
||
--selftest: 0 = 断言全通过; 1 = 有断言失败
|
||
|
||
--selftest 为什么必须有（历史教训）:
|
||
上一版门禁的前端是 CPython `ast`, 对含 `list[X*]` / `rawptr[X]*` 等 vast 专有写法的
|
||
文件整体 SyntaxError 跳过 ⇒ 曾出现 "OTHER=0 全绿" 而工作树里实际有 **18 处**
|
||
`rawptr[llvmlite.Function].Ty` 真错（整批文件根本没被检查）。
|
||
故 --selftest 内置合成用例（生成到临时目录、跑完即删）, 断言"**报出集合 == 预期集合**",
|
||
覆盖 4 类正例（含 vast 专有 `rawptr[X]*` / `rawptr[X]**`）与 4 类负例
|
||
（合法字段 / 跨函数同名各自声明 / docstring 与注释 / 返回值推断生效）,
|
||
用来证明扫描器不是空转。
|
||
"""
|
||
import glob
|
||
import os
|
||
import re
|
||
import shutil
|
||
import sys
|
||
import tempfile
|
||
|
||
# ============================================================
|
||
# 0. 路径推导（全部按脚本自身位置, 不依赖 cwd）
|
||
# ScriptDir = <RepoRoot>/TransPyC/Tools
|
||
# TpcRoot = <RepoRoot>/TransPyC （TransPyC 仓库根）
|
||
# RepoRoot = <RepoRoot> （工作区根）
|
||
# ============================================================
|
||
ScriptDir = os.path.dirname(os.path.abspath(__file__))
|
||
TpcRoot = os.path.dirname(ScriptDir)
|
||
RepoRoot = os.path.dirname(TpcRoot)
|
||
|
||
DefaultAppRoot = os.path.join(RepoRoot, 'TransPyV', 'App')
|
||
DefaultIncludesDir = os.path.join(RepoRoot, 'includes')
|
||
DefaultOutFile = os.path.join(RepoRoot, 'Debugger', 'ScanAstFieldMiss.txt')
|
||
|
||
# ---- 前端: TPC 自研 vast（非 CPython ast）。需要 TransPyC 包根在 sys.path,
|
||
# lib._bootstrap 会再补齐 lib/ 与 lib/includes（使 `import vast` 可用）。
|
||
if TpcRoot not in sys.path:
|
||
sys.path.insert(0, TpcRoot)
|
||
from lib import vast as ast # noqa: E402
|
||
from lib.core.SymbolUtils import ExpandSoftKeywords # noqa: E402
|
||
|
||
SupportPrefix = ''
|
||
|
||
# ============================================================
|
||
# 1. 类成员表: 字段 / 方法 / 基类（按族隔离；每次扫描重建）
|
||
# ============================================================
|
||
FamilyTable = []
|
||
SupportFiles = []
|
||
FamilyNames = set()
|
||
|
||
ClassFields = {}
|
||
ClassMethods = {}
|
||
ClassBase = {}
|
||
ClassLoc = {} # 裸类名 -> [族, ...]（用于跨族基类解析）
|
||
EnumClass = {} # (族, 类名) -> 1 表示 t.REnum/t.CEnum 派生（变体字段合并）
|
||
REnumClass = {} # (族, 类名) -> 1 表示 t.REnum 派生（额外有编译器内建 .kind()）
|
||
|
||
MemberCache = {}
|
||
FamilyFields = {} # 族 -> 该族所有字段并集（OTHER/NOWHERE 判定用）
|
||
AmbiguousBases = set()
|
||
|
||
Uses = [] # [(AttrNode, Scope)]；Walk 现场填充, 每个文件扫描前 clear
|
||
|
||
# ---- 返回注解表（同文件内 def f(...) -> rawptr[族.X]）：供 x = f() 形态的类型推导。
|
||
# 仅补充覆盖（收紧），不改变任何既有判据：名字在文件内返回注解不一致时报歧义并放弃。
|
||
FuncRet = {}
|
||
|
||
|
||
def ResetTables():
|
||
"""清空全部类表/缓存（供同一进程内多轮扫描复用, 如 --selftest）。"""
|
||
global FamilyTable, SupportFiles, FamilyNames
|
||
global ClassFields, ClassMethods, ClassBase, ClassLoc, EnumClass, REnumClass
|
||
global MemberCache, FamilyFields, AmbiguousBases
|
||
FamilyTable = []
|
||
SupportFiles = []
|
||
FamilyNames = set()
|
||
ClassFields = {}
|
||
ClassMethods = {}
|
||
ClassBase = {}
|
||
ClassLoc = {}
|
||
EnumClass = {}
|
||
REnumClass = {}
|
||
MemberCache = {}
|
||
FamilyFields = {}
|
||
AmbiguousBases = set()
|
||
FuncRet.clear()
|
||
|
||
|
||
def AddClass(Fam, Name, Bases):
|
||
ClassFields.setdefault((Fam, Name), set())
|
||
ClassMethods.setdefault((Fam, Name), set())
|
||
ClassLoc.setdefault(Name, [])
|
||
if Fam not in ClassLoc[Name]:
|
||
ClassLoc[Name].append(Fam)
|
||
if Bases:
|
||
ClassBase.setdefault((Fam, Name), set()).update(Bases)
|
||
|
||
|
||
def BaseName(Node):
|
||
if isinstance(Node, ast.Name):
|
||
return Node.id
|
||
if isinstance(Node, ast.Attribute):
|
||
return Node.attr
|
||
if isinstance(Node, ast.Subscript): # GSListNode[ParamNode] -> GSListNode
|
||
return BaseName(Node.value)
|
||
return None
|
||
|
||
|
||
def CollectBody(Fam, Name, Body, Enum):
|
||
"""收集类体成员；Enum=True 时把嵌套变体类的字段并入本类（REnum 语义）。"""
|
||
for Item in Body:
|
||
if isinstance(Item, ast.AnnAssign) and isinstance(Item.target, ast.Name):
|
||
ClassFields[(Fam, Name)].add(Item.target.id)
|
||
elif isinstance(Item, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||
ClassMethods[(Fam, Name)].add(Item.name)
|
||
elif Enum and isinstance(Item, ast.ClassDef):
|
||
CollectBody(Fam, Name, Item.body, Enum) # 变体字段并入父类
|
||
|
||
|
||
def RegisterClass(Fam, Node):
|
||
Bases = set()
|
||
for B in Node.bases:
|
||
Nm = BaseName(B)
|
||
if Nm:
|
||
Bases.add(Nm)
|
||
AddClass(Fam, Node.name, Bases)
|
||
Enum = 1 if (Bases & {'REnum', 'CEnum'}) else 0
|
||
if Enum:
|
||
EnumClass[(Fam, Node.name)] = 1
|
||
if 'REnum' in Bases:
|
||
# REnum.kind() 是编译器内建（HandlesExprCall.py:3126 'kind' → __tag 读取）
|
||
REnumClass[(Fam, Node.name)] = 1
|
||
CollectBody(Fam, Node.name, Node.body, Enum)
|
||
|
||
|
||
def ParseSrc(Src, Path):
|
||
"""vast 前端解析入口（全脚本单一前端; 失败抛异常, 由调用方记账）。
|
||
|
||
`.vp` 源码须先过 Viper 软关键字层（`lib/core/SymbolUtils.ExpandSoftKeywords`：
|
||
`fn/fun/func/function` → `def`、`enum X:` → `class X(t.CEnum):`），与
|
||
`StubGen/Converter.py` / `Shell/TransPyC.py` 等既有前端入口同口径。
|
||
否则含 `enum` 的 `.vp`（如 `includes/ast/base.vp:168`）会在此整体 SyntaxError，
|
||
被 `LoadInclude` 的正则兜底或 `RunScan` 的 `parsefail` 记成"解析失败"——两者
|
||
都会**静默漏掉该文件的类/字段**（本脚本的假绿来源）。`.py` 无糖，展开为空操作
|
||
（`'enum' in Code` 门禁 + 行首锚定正则）。
|
||
"""
|
||
return ast.parse(ExpandSoftKeywords(Src), Path)
|
||
|
||
|
||
def ErrLine(E):
|
||
"""vast 的 SyntaxError 可能无 lineno（C 路径）=> 从报文 `at line N` 兜底提取。"""
|
||
L = getattr(E, 'lineno', None)
|
||
if L:
|
||
return L
|
||
M = re.search(r'at line (\d+)', str(E))
|
||
return int(M.group(1)) if M else -1
|
||
|
||
|
||
def LoadIncludeRegex(Src, Fam):
|
||
"""ast.parse 失败（docstring 含 \\xNN 等）时的回退：缩进感知的类/字段/方法收集。"""
|
||
Indent = {}
|
||
Enum = {}
|
||
for Ln in Src.splitlines():
|
||
M = re.match(r'(\s*)class (\w+)(?:\[[^\]]*\])?(?:\(([^)]*)\))?\s*:', Ln)
|
||
if M:
|
||
Depth = len(M.group(1))
|
||
Name = M.group(2)
|
||
Bases = set()
|
||
if M.group(3):
|
||
for B in M.group(3).split(','):
|
||
B = B.strip().split('.')[-1].split('[')[0].strip()
|
||
if B:
|
||
Bases.add(B)
|
||
AddClass(Fam, Name, Bases)
|
||
Indent[Name] = Depth
|
||
Enum[Name] = 1 if (Bases & {'REnum', 'CEnum'}) else 0
|
||
if Enum[Name]:
|
||
EnumClass[(Fam, Name)] = 1
|
||
continue
|
||
M2 = re.match(r'(\s+)(\w+)\s*:\s', Ln)
|
||
if M2 and Indent:
|
||
Depth = len(M2.group(1))
|
||
Owner = [N for N, D in Indent.items() if D < Depth]
|
||
if Owner:
|
||
P = max(Owner, key=lambda N: Indent[N])
|
||
ClassFields[(Fam, P)].add(M2.group(2))
|
||
continue
|
||
M3 = re.match(r'(\s+)def (\w+)\(', Ln)
|
||
if M3 and Indent:
|
||
Depth = len(M3.group(1))
|
||
Owner = [N for N, D in Indent.items() if D < Depth]
|
||
if Owner:
|
||
P = max(Owner, key=lambda N: Indent[N])
|
||
ClassMethods[(Fam, P)].add(M3.group(2))
|
||
|
||
|
||
def LoadInclude(Path, Fam):
|
||
try:
|
||
Src = open(Path, encoding='utf-8').read()
|
||
except Exception:
|
||
return
|
||
try:
|
||
Tree = ParseSrc(Src, Path)
|
||
except Exception: # noqa: BLE001 （vast 失败可能是 SyntaxError/ValueError）
|
||
LoadIncludeRegex(Src, Fam)
|
||
return
|
||
for Node in Tree.body:
|
||
if isinstance(Node, ast.ClassDef):
|
||
RegisterClass(Fam, Node)
|
||
|
||
|
||
def BuildFamilies(AppRoot, IncludesDir):
|
||
"""按 AppRoot/IncludesDir 建族表并加载类定义（含支撑族）。"""
|
||
global FamilyTable, SupportFiles, FamilyNames
|
||
FamilyTable = [
|
||
('ast', [os.path.join(IncludesDir, 'ast')]),
|
||
('llvmlite', [os.path.join(IncludesDir, 'llvmlite')]),
|
||
('GenTypes', [os.path.join(AppRoot, 'lib', 'core', 'Gen', 'GenTypes.vp')]),
|
||
('HT', [os.path.join(AppRoot, 'lib', 'core', 'Gen', 'Translator.vp')]),
|
||
('memhub', [os.path.join(IncludesDir, 'memhub.vp')]),
|
||
]
|
||
SupportFiles = [os.path.join(IncludesDir, 'linkedlist.vp')]
|
||
FamilyNames = set(P for P, _ in FamilyTable) | {''}
|
||
for Prefix, Paths in FamilyTable:
|
||
for P in Paths:
|
||
if os.path.isdir(P):
|
||
for F in sorted(glob.glob(os.path.join(P, '*.vp'))):
|
||
LoadInclude(F, Prefix)
|
||
else:
|
||
LoadInclude(P, Prefix)
|
||
for F in SupportFiles:
|
||
LoadInclude(F, SupportPrefix)
|
||
|
||
|
||
def ComputeFamilyFields():
|
||
global FamilyFields
|
||
FamilyFields = {}
|
||
for (_F, _N) in ClassFields:
|
||
FamilyFields.setdefault(_F, set()).update(ClassFields[(_F, _N)])
|
||
|
||
|
||
def FindClass(Name, Prefer):
|
||
"""裸类名 -> (族, 类名)。优先本族, 再支撑族, 最后全局唯一名。"""
|
||
if (Prefer, Name) in ClassFields:
|
||
return (Prefer, Name)
|
||
if (SupportPrefix, Name) in ClassFields:
|
||
return (SupportPrefix, Name)
|
||
Locs = ClassLoc.get(Name, [])
|
||
if len(Locs) == 1:
|
||
return (Locs[0], Name)
|
||
if len(Locs) > 1:
|
||
AmbiguousBases.add(Name)
|
||
return None
|
||
|
||
|
||
def Members(Fam, Name, Seen=None):
|
||
if Seen is None:
|
||
Seen = set()
|
||
Key = (Fam, Name)
|
||
if Key in Seen:
|
||
return set()
|
||
Seen.add(Key)
|
||
Out = set(ClassFields.get(Key, ())) | set(ClassMethods.get(Key, ()))
|
||
if Key in REnumClass:
|
||
Out.add('kind')
|
||
for B in ClassBase.get(Key, ()):
|
||
Loc = FindClass(B, Fam)
|
||
if Loc is not None:
|
||
if Loc in REnumClass:
|
||
Out.add('kind')
|
||
Out |= Members(Loc[0], Loc[1], Seen)
|
||
return Out
|
||
|
||
|
||
def Member(Fam, Name):
|
||
Key = (Fam, Name)
|
||
if Key not in MemberCache:
|
||
MemberCache[Key] = Members(Fam, Name)
|
||
return MemberCache[Key]
|
||
|
||
|
||
# ============================================================
|
||
# 2. 类型推导: 从注解 / cast 调用里取 (族, 类名)
|
||
# ============================================================
|
||
def AnnClass(Ann):
|
||
"""注解里若为 rawptr[<族>.<X>]（可被 `| t.CPtr` 包裹）返回 (族, X), 否则 None。"""
|
||
if Ann is None:
|
||
return None
|
||
if isinstance(Ann, ast.BinOp) and isinstance(Ann.op, ast.BitOr):
|
||
return AnnClass(Ann.left) or AnnClass(Ann.right)
|
||
if isinstance(Ann, ast.Subscript):
|
||
if isinstance(Ann.value, ast.Name) and Ann.value.id == 'rawptr':
|
||
Sl = Ann.slice
|
||
if isinstance(Sl, ast.Attribute) and isinstance(Sl.value, ast.Name) \
|
||
and Sl.value.id in FamilyNames and Sl.value.id != '':
|
||
return (Sl.value.id, Sl.attr)
|
||
return None
|
||
|
||
|
||
def CastClass(Val):
|
||
"""`rawptr[<族>.<X>](...)` 形态的 cast 调用返回 (族, X), 否则 None。"""
|
||
if isinstance(Val, ast.Call) and isinstance(Val.func, ast.Subscript):
|
||
return AnnClass(Val.func)
|
||
return None
|
||
|
||
|
||
def CollectFuncRets(Tree):
|
||
FuncRet.clear()
|
||
Ambig = set()
|
||
for Node in ast.walk(Tree):
|
||
if isinstance(Node, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||
C = AnnClass(Node.returns)
|
||
if Node.name in FuncRet and FuncRet[Node.name] != C:
|
||
Ambig.add(Node.name)
|
||
else:
|
||
FuncRet[Node.name] = C
|
||
for N in Ambig:
|
||
FuncRet.pop(N, None)
|
||
|
||
|
||
# ============================================================
|
||
# 3. 作用域模型
|
||
# ============================================================
|
||
class Scope:
|
||
__slots__ = ('Kind', 'Parent', 'Decls')
|
||
|
||
def __init__(self, Kind, Parent):
|
||
self.Kind = Kind # 'module' / 'func' / 'class'
|
||
self.Parent = Parent
|
||
self.Decls = {} # name -> [(Lineno, Col, Cls, DeclLine)]
|
||
|
||
def Add(self, Name, Lineno, Col, Cls):
|
||
self.Decls.setdefault(Name, []).append((Lineno, Col, Cls, Lineno))
|
||
|
||
|
||
def Resolve(Sc, Name, Lineno, Col):
|
||
"""沿作用域链找 Name 在 (Lineno, Col) 之前最近一次的声明。
|
||
|
||
返回 (Cls, DeclLine); Cls 为 None 表示“已解析但非族指针”或“未找到”。
|
||
类体作用域对嵌套函数穿透（Python 语义）。
|
||
"""
|
||
Cur = Sc
|
||
First = True
|
||
while Cur is not None:
|
||
if Cur.Kind != 'class' or First:
|
||
if Name in Cur.Decls:
|
||
Best = None
|
||
for D in Cur.Decls[Name]:
|
||
if (D[0], D[1]) <= (Lineno, Col):
|
||
if Best is None or (D[0], D[1]) > (Best[0], Best[1]):
|
||
Best = D
|
||
if Best is not None:
|
||
return Best[2], Best[3]
|
||
First = False
|
||
Cur = Cur.Parent
|
||
return None, None
|
||
|
||
|
||
def Walk(Node, Sc):
|
||
if Node is None:
|
||
return
|
||
if isinstance(Node, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||
for D in Node.decorator_list:
|
||
Walk(D, Sc)
|
||
A = Node.args
|
||
Formals = list(A.posonlyargs) + list(A.args) + list(A.kwonlyargs)
|
||
for P in Formals:
|
||
Walk(P.annotation, Sc)
|
||
for D in A.defaults:
|
||
Walk(D, Sc)
|
||
for D in A.kw_defaults:
|
||
Walk(D, Sc)
|
||
if A.vararg is not None:
|
||
Walk(A.vararg.annotation, Sc)
|
||
if A.kwarg is not None:
|
||
Walk(A.kwarg.annotation, Sc)
|
||
Walk(Node.returns, Sc)
|
||
FS = Scope('func', Sc)
|
||
for P in Formals:
|
||
FS.Add(P.arg, Node.lineno, 0, AnnClass(P.annotation))
|
||
if A.vararg is not None:
|
||
FS.Add(A.vararg.arg, Node.lineno, 0, AnnClass(A.vararg.annotation))
|
||
if A.kwarg is not None:
|
||
FS.Add(A.kwarg.arg, Node.lineno, 0, AnnClass(A.kwarg.annotation))
|
||
for S in Node.body:
|
||
Walk(S, FS)
|
||
return
|
||
if isinstance(Node, ast.Lambda):
|
||
A = Node.args
|
||
Formals = list(A.posonlyargs) + list(A.args) + list(A.kwonlyargs)
|
||
for P in Formals:
|
||
Walk(P.annotation, Sc)
|
||
for D in A.defaults:
|
||
Walk(D, Sc)
|
||
for D in A.kw_defaults:
|
||
Walk(D, Sc)
|
||
LS = Scope('func', Sc)
|
||
for P in Formals:
|
||
LS.Add(P.arg, Node.lineno, 0, AnnClass(P.annotation))
|
||
Walk(Node.body, LS)
|
||
return
|
||
if isinstance(Node, ast.ClassDef):
|
||
for D in Node.decorator_list:
|
||
Walk(D, Sc)
|
||
for B in Node.bases:
|
||
Walk(B, Sc)
|
||
for K in Node.keywords:
|
||
Walk(K.value, Sc)
|
||
CS = Scope('class', Sc)
|
||
for S in Node.body:
|
||
Walk(S, CS)
|
||
return
|
||
if isinstance(Node, ast.Attribute):
|
||
# 只收源码里真实存在的属性访问: vast 脱糖 `X*` 会合成 `t.CPtr` Attribute,
|
||
# 该节点**连 lineno 属性都没有**（全树 555 个, 全为 ('t','CPtr')）, 不是源码
|
||
# 访问、也不能参与位置比较 => 用 getattr 显式排除。
|
||
if isinstance(Node.value, ast.Name) and getattr(Node, 'lineno', None) is not None:
|
||
Uses.append((Node, Sc))
|
||
Walk(Node.value, Sc)
|
||
return
|
||
if isinstance(Node, ast.AnnAssign):
|
||
if isinstance(Node.target, ast.Name):
|
||
Sc.Add(Node.target.id, Node.target.lineno, Node.target.col_offset,
|
||
AnnClass(Node.annotation))
|
||
elif isinstance(Node.target, ast.Attribute):
|
||
Walk(Node.target, Sc) # `x.y: T = ...` 的属性访问
|
||
Walk(Node.annotation, Sc)
|
||
Walk(Node.value, Sc)
|
||
return
|
||
if isinstance(Node, ast.Assign):
|
||
Cls = None
|
||
V = Node.value
|
||
if isinstance(V, ast.Call):
|
||
Cls = CastClass(V)
|
||
if Cls is None and isinstance(V.func, ast.Name):
|
||
Cls = FuncRet.get(V.func.id)
|
||
elif isinstance(V, ast.Name):
|
||
Cls, _ = Resolve(Sc, V.id, V.lineno, V.col_offset)
|
||
for T in Node.targets:
|
||
if isinstance(T, ast.Name):
|
||
Sc.Add(T.id, T.lineno, T.col_offset, Cls)
|
||
# 赋值**目标**位置的属性访问也要计入（如 `vtable_node.Ty = ...`）
|
||
Walk(T, Sc)
|
||
Walk(Node.value, Sc)
|
||
return
|
||
if isinstance(Node, ast.AugAssign):
|
||
if isinstance(Node.target, ast.Name):
|
||
Sc.Add(Node.target.id, Node.target.lineno, Node.target.col_offset, None)
|
||
Walk(Node.target, Sc)
|
||
Walk(Node.value, Sc)
|
||
return
|
||
if isinstance(Node, (ast.For, ast.AsyncFor)):
|
||
for T in ast.walk(Node.target):
|
||
if isinstance(T, ast.Name):
|
||
Sc.Add(T.id, Node.lineno, 0, None)
|
||
Walk(Node.iter, Sc)
|
||
for S in Node.body:
|
||
Walk(S, Sc)
|
||
for S in Node.orelse:
|
||
Walk(S, Sc)
|
||
return
|
||
if isinstance(Node, ast.NamedExpr):
|
||
if isinstance(Node.target, ast.Name):
|
||
Sc.Add(Node.target.id, Node.target.lineno, Node.target.col_offset, None)
|
||
Walk(Node.value, Sc)
|
||
return
|
||
for Ch in ast.iter_child_nodes(Node):
|
||
Walk(Ch, Sc)
|
||
|
||
|
||
# ============================================================
|
||
# 4. 扫描（一轮 = ResetTables + BuildFamilies + 遍历 AppRoot）
|
||
# ============================================================
|
||
def BucketOf(Fam):
|
||
return Fam if Fam in ('ast', 'llvmlite') else 'OTHER:' + (Fam or 'support')
|
||
|
||
|
||
def Bump(D, Key):
|
||
D[Key] = D.get(Key, 0) + 1
|
||
|
||
|
||
def RunScan(AppRoot, IncludesDir, OutFile, Verbose):
|
||
"""跑一轮扫描: 写报告文件 + 返回统计/命中集合（不打印）。"""
|
||
ResetTables()
|
||
BuildFamilies(AppRoot, IncludesDir)
|
||
ComputeFamilyFields()
|
||
|
||
Nowhere = []
|
||
Other = []
|
||
ResolvedCount = 0
|
||
UnresolvedCount = 0
|
||
Files = sorted(glob.glob(os.path.join(AppRoot, '**', '*.vp'), recursive=True))
|
||
ParseFail = []
|
||
|
||
for Path in Files:
|
||
try:
|
||
Src = open(Path, encoding='utf-8').read()
|
||
except Exception:
|
||
continue
|
||
try:
|
||
Tree = ParseSrc(Src, Path)
|
||
except Exception as E: # noqa: BLE001 （vast 失败可能是 SyntaxError/ValueError）
|
||
ParseFail.append((os.path.relpath(Path, AppRoot), ErrLine(E), str(E)[:80]))
|
||
continue
|
||
RawLines = Src.splitlines()
|
||
Uses.clear()
|
||
CollectFuncRets(Tree)
|
||
MS = Scope('module', None)
|
||
for S in Tree.body:
|
||
Walk(S, MS)
|
||
|
||
for AttrNode, Sc in Uses:
|
||
Var = AttrNode.value.id
|
||
Field = AttrNode.attr
|
||
Cls, DeclLine = Resolve(Sc, Var, AttrNode.lineno, AttrNode.col_offset)
|
||
if Cls is None:
|
||
UnresolvedCount += 1
|
||
continue
|
||
Fam, CName = Cls
|
||
if (Fam, CName) not in ClassFields:
|
||
UnresolvedCount += 1
|
||
continue
|
||
ResolvedCount += 1
|
||
if Field in Member(Fam, CName):
|
||
continue
|
||
Rel = os.path.relpath(Path, AppRoot)
|
||
SrcLine = RawLines[AttrNode.lineno - 1].strip()[:96] if AttrNode.lineno <= len(RawLines) else ''
|
||
Row = (Rel, AttrNode.lineno, Var, CName, Field, DeclLine, SrcLine, Fam)
|
||
if Field in FamilyFields.get(Fam, set()):
|
||
Other.append(Row)
|
||
else:
|
||
Nowhere.append(Row)
|
||
|
||
# 稳定排序（便于 diff / 门禁）
|
||
SortKey = lambda R: (R[0], R[1], R[2], R[4], R[7], R[3], R[5])
|
||
Nowhere.sort(key=SortKey)
|
||
Other.sort(key=SortKey)
|
||
|
||
BucketsN = {}
|
||
BucketsO = {}
|
||
for R in Nowhere:
|
||
Bump(BucketsN, BucketOf(R[7]))
|
||
for R in Other:
|
||
Bump(BucketsO, BucketOf(R[7]))
|
||
|
||
ParsedN = len(Files) - len(ParseFail)
|
||
UsesN = ResolvedCount + UnresolvedCount
|
||
ClassLine = ' '.join('%s=%d' % (f, sum(1 for (x, _n) in ClassFields if x == f))
|
||
for f in sorted(set(x for x, _n in ClassFields)))
|
||
|
||
def W(Fh, Rows, Tag):
|
||
for R in Rows:
|
||
Fh.write('%-46s:%-5d var=%s cls=%s.%s field=%s decl@%d 判定=%s %s\n' %
|
||
(R[0], R[1], R[2], R[7], R[3], R[4], R[5], Tag, R[6]))
|
||
|
||
OutDir = os.path.dirname(os.path.abspath(OutFile))
|
||
if OutDir:
|
||
os.makedirs(OutDir, exist_ok=True)
|
||
with open(OutFile, 'w', encoding='utf-8') as Fh:
|
||
Fh.write('files=%d parsed=%d parsefail=%d uses=%d NOWHERE=%d OTHER=%d RESOLVED=%d UNRESOLVED=%d\n' %
|
||
(len(Files), ParsedN, len(ParseFail), UsesN,
|
||
len(Nowhere), len(Other), ResolvedCount, UnresolvedCount))
|
||
Fh.write('classes: %s\n' % ClassLine)
|
||
Fh.write('NOWHERE buckets: %s\n' % sorted(BucketsN.items()))
|
||
Fh.write('OTHER buckets: %s\n' % sorted(BucketsO.items()))
|
||
if AmbiguousBases:
|
||
Fh.write('AMBIGUOUS BASES (跨族重名, 已不解析): %s\n' % sorted(AmbiguousBases))
|
||
Fh.write('\n=== NOWHERE (本族任何类都没有该字段) ===\n')
|
||
W(Fh, Nowhere, 'NOWHERE')
|
||
Fh.write('\n=== OTHER (字段只存在于本族别的类) ===\n')
|
||
W(Fh, Other, 'OTHER')
|
||
if ParseFail:
|
||
Fh.write('\n=== PARSE FAIL (vast 无法解析, 已跳过) ===\n')
|
||
for P in ParseFail:
|
||
Fh.write('%s:%s %s\n' % P)
|
||
|
||
return {
|
||
'Files': len(Files), 'Parsed': ParsedN, 'ParseFail': ParseFail,
|
||
'Uses': UsesN, 'Nowhere': Nowhere, 'Other': Other,
|
||
'Resolved': ResolvedCount, 'Unresolved': UnresolvedCount,
|
||
'BucketsN': BucketsN, 'BucketsO': BucketsO,
|
||
'Ambiguous': sorted(AmbiguousBases), 'Classes': ClassLine,
|
||
'OutFile': OutFile,
|
||
}
|
||
|
||
|
||
def PrintSummary(Stats, Verbose):
|
||
print('files=%d parsed=%d parsefail=%d uses=%d NOWHERE=%d OTHER=%d RESOLVED=%d UNRESOLVED=%d -> %s' %
|
||
(Stats['Files'], Stats['Parsed'], len(Stats['ParseFail']), Stats['Uses'],
|
||
len(Stats['Nowhere']), len(Stats['Other']),
|
||
Stats['Resolved'], Stats['Unresolved'], Stats['OutFile']))
|
||
print('NOWHERE buckets: %s' % sorted(Stats['BucketsN'].items()))
|
||
print('OTHER buckets: %s' % sorted(Stats['BucketsO'].items()))
|
||
if Stats['Ambiguous']:
|
||
print('AMBIGUOUS BASES: %s' % Stats['Ambiguous'])
|
||
if Stats['ParseFail']:
|
||
print('PARSE FAIL: %s' % Stats['ParseFail'])
|
||
Limit = 25 if not Verbose else 100000
|
||
for R in Stats['Nowhere'][:Limit]:
|
||
print('N %-40s:%-5d %s.%s.%s [decl@%d]' % (R[0], R[1], R[7], R[3], R[4], R[5]))
|
||
for R in Stats['Other'][:Limit]:
|
||
print('O %-40s:%-5d %s.%s.%s [decl@%d]' % (R[0], R[1], R[7], R[3], R[4], R[5]))
|
||
|
||
|
||
# ============================================================
|
||
# 5. --selftest: 内置合成用例（临时目录生成, 跑完即删）
|
||
# 断言"报出集合 == 预期集合"；正例 6 条（含 vast 专有写法）、负例 4 个文件。
|
||
# ============================================================
|
||
SelftestCases = [
|
||
('CaseP1AstNameOther.vp', """import ast
|
||
|
||
|
||
def ProbeP1(x_r: rawptr[ast.AST]) -> int:
|
||
return x_r.name
|
||
"""),
|
||
('CaseP2LlvFunctionTyOther.vp', """import llvmlite
|
||
|
||
|
||
def ProbeP2(x_r: rawptr[llvmlite.Function]) -> int:
|
||
return x_r.Ty
|
||
"""),
|
||
('CaseP3AstNowhere.vp', """import ast
|
||
|
||
|
||
def ProbeP3(x_r: rawptr[ast.AST]) -> int:
|
||
return x_r.zzz_not_a_field
|
||
"""),
|
||
('CaseP4AstStarOther.vp', """import ast
|
||
|
||
|
||
def ProbeP4(x_r: rawptr[ast.AST]*) -> int:
|
||
return x_r.name
|
||
"""),
|
||
('CaseP5AstStarStarOther.vp', """import ast
|
||
|
||
|
||
def ProbeP5(x_r: rawptr[ast.AST]**) -> int:
|
||
return x_r.name
|
||
"""),
|
||
('CaseP6ReturnInferOther.vp', """import llvmlite
|
||
|
||
|
||
def MkP6(x_r: rawptr[llvmlite.Function]) -> rawptr[llvmlite.Function]:
|
||
return x_r
|
||
|
||
|
||
def ProbeP6(x_r: rawptr[llvmlite.Function]) -> int:
|
||
y_r = MkP6(x_r)
|
||
return y_r.Ty
|
||
"""),
|
||
('CaseN1ParamNodeTy.vp', """import llvmlite
|
||
|
||
|
||
def ProbeN1(x_r: rawptr[llvmlite.ParamNode]) -> int:
|
||
return x_r.Ty
|
||
"""),
|
||
('CaseN2SameNameScopes.vp', """import llvmlite
|
||
|
||
|
||
def ProbeN2A(x_r: rawptr[llvmlite.ParamNode]) -> int:
|
||
return x_r.Ty
|
||
|
||
|
||
def ProbeN2B(x_r: rawptr[llvmlite.Function]) -> int:
|
||
return x_r.Ty
|
||
"""),
|
||
('CaseN3DocComment.vp', """import ast
|
||
|
||
|
||
def ProbeN3(x_r: rawptr[ast.AST]) -> int:
|
||
\"\"\"docstring: x_r.name / x_r.zzz_not_a_field 只是说明文字, 不是代码。\"\"\"
|
||
# 注释: x_r.name / x_r.zzz_not_a_field
|
||
return 0
|
||
"""),
|
||
('CaseN4ReturnInferOk.vp', """import llvmlite
|
||
|
||
|
||
def MkN4(x_r: rawptr[llvmlite.ParamNode]) -> rawptr[llvmlite.ParamNode]:
|
||
return x_r
|
||
|
||
|
||
def ProbeN4(x_r: rawptr[llvmlite.ParamNode]) -> int:
|
||
y_r = MkN4(x_r)
|
||
return y_r.Ty
|
||
"""),
|
||
]
|
||
|
||
# (文件, 变量, 族.类, 字段) -> 判定
|
||
SelftestExpected = {
|
||
('CaseP1AstNameOther.vp', 'x_r', 'ast.AST', 'name'): 'OTHER',
|
||
('CaseP2LlvFunctionTyOther.vp', 'x_r', 'llvmlite.Function', 'Ty'): 'OTHER',
|
||
('CaseP3AstNowhere.vp', 'x_r', 'ast.AST', 'zzz_not_a_field'): 'NOWHERE',
|
||
('CaseP4AstStarOther.vp', 'x_r', 'ast.AST', 'name'): 'OTHER',
|
||
('CaseP5AstStarStarOther.vp', 'x_r', 'ast.AST', 'name'): 'OTHER',
|
||
('CaseP6ReturnInferOther.vp', 'y_r', 'llvmlite.Function', 'Ty'): 'OTHER',
|
||
('CaseN2SameNameScopes.vp', 'x_r', 'llvmlite.Function', 'Ty'): 'OTHER',
|
||
}
|
||
|
||
|
||
def SelfTest(IncludesDir, OutFile):
|
||
AppRoot = tempfile.mkdtemp(prefix='ScanAstFieldMissSelftest.')
|
||
try:
|
||
for Name, Src in SelftestCases:
|
||
with open(os.path.join(AppRoot, Name), 'w', encoding='utf-8', newline='\n') as Fh:
|
||
Fh.write(Src)
|
||
Report = OutFile or os.path.join(AppRoot, 'Selftest.txt')
|
||
Stats = RunScan(AppRoot, IncludesDir, Report, False)
|
||
|
||
Reported = {}
|
||
for R in Stats['Nowhere']:
|
||
Reported[(R[0], R[2], '%s.%s' % (R[7], R[3]), R[4])] = 'NOWHERE'
|
||
for R in Stats['Other']:
|
||
Reported[(R[0], R[2], '%s.%s' % (R[7], R[3]), R[4])] = 'OTHER'
|
||
|
||
Ok = True
|
||
if Stats['ParseFail']:
|
||
Ok = False
|
||
print('[selftest] 解析失败（预期 parsefail=0）:')
|
||
for P in Stats['ParseFail']:
|
||
print(' %s:%s %s' % P)
|
||
Missing = sorted(set(SelftestExpected) - set(Reported))
|
||
Extra = sorted(set(Reported) - set(SelftestExpected))
|
||
TagDiff = sorted(K for K in set(Reported) & set(SelftestExpected)
|
||
if Reported[K] != SelftestExpected[K])
|
||
for K in Missing:
|
||
Ok = False
|
||
print('[selftest] 漏报: %s -> 预期 %s' % (K, SelftestExpected[K]))
|
||
for K in Extra:
|
||
Ok = False
|
||
print('[selftest] 误报: %s -> %s' % (K, Reported[K]))
|
||
for K in TagDiff:
|
||
Ok = False
|
||
print('[selftest] 判定不符: %s -> %s（预期 %s）' % (K, Reported[K], SelftestExpected[K]))
|
||
|
||
print('[selftest] files=%d parsed=%d parsefail=%d NOWHERE=%d OTHER=%d 报出=%d 预期=%d' %
|
||
(Stats['Files'], Stats['Parsed'], len(Stats['ParseFail']),
|
||
len(Stats['Nowhere']), len(Stats['Other']), len(Reported), len(SelftestExpected)))
|
||
print('[selftest] %s' % ('PASS' if Ok else 'FAIL'))
|
||
return 0 if Ok else 1
|
||
finally:
|
||
shutil.rmtree(AppRoot, ignore_errors=True)
|
||
|
||
|
||
# ============================================================
|
||
# 6. CLI
|
||
# ============================================================
|
||
def ParseArgs(Argv):
|
||
Opt = {'AppRoot': DefaultAppRoot, 'Includes': DefaultIncludesDir, 'Out': None,
|
||
'Strict': False, 'Verbose': False, 'SelfTest': False}
|
||
Need = {'--app-root': 'AppRoot', '--includes': 'Includes', '--out': 'Out'}
|
||
Flags = {'--strict': 'Strict', '--verbose': 'Verbose', '--selftest': 'SelfTest'}
|
||
I = 0
|
||
while I < len(Argv):
|
||
A = Argv[I]
|
||
if A in Need:
|
||
if I + 1 >= len(Argv):
|
||
sys.stderr.write('参数缺值: %s\n' % A)
|
||
return None
|
||
Opt[Need[A]] = Argv[I + 1]
|
||
I += 2
|
||
continue
|
||
if A in Flags:
|
||
Opt[Flags[A]] = True
|
||
I += 1
|
||
continue
|
||
sys.stderr.write('未知参数: %s（用法见本脚本 docstring）\n' % A)
|
||
return None
|
||
return Opt
|
||
|
||
|
||
def main():
|
||
Opt = ParseArgs(sys.argv[1:])
|
||
if Opt is None:
|
||
return 2
|
||
if Opt['SelfTest']:
|
||
return SelfTest(Opt['Includes'], Opt['Out'])
|
||
Stats = RunScan(Opt['AppRoot'], Opt['Includes'], Opt['Out'] or DefaultOutFile, Opt['Verbose'])
|
||
PrintSummary(Stats, Opt['Verbose'])
|
||
if Opt['Strict'] and (Stats['Nowhere'] or Stats['Other'] or Stats['ParseFail']):
|
||
sys.stderr.write('[strict] 门禁失败: NOWHERE=%d OTHER=%d parsefail=%d\n' %
|
||
(len(Stats['Nowhere']), len(Stats['Other']), len(Stats['ParseFail'])))
|
||
return 1
|
||
return 0
|
||
|
||
|
||
if __name__ == '__main__':
|
||
sys.exit(main())
|