perf(compile): 批次加权分桶消除长尾 + 共享符号表去重复序列化 + typedef 收集走 AST 缓存
- Phase2 批次切分:原按文件数均分(batch_size=ceil(N/n)),大文件集中批次拖长尾、worker 空等;改为按源文件大小 LPT 贪心分桶,各批次总成本均匀(任务颗粒度不均匀) - 共享符号表序列化:shared_data 同时含 symbol_table 对象与 symbol_table_bytes 双重序列化(_shared_sym.json 4.65MB 近半冗余);统一为仅传 symbol_table_bytes,worker/批次端从字节反序列化(includes 与 App 两阶段同步修改) - Phase1 _collect_typedef_map / _process_file_pyi:直读+ast.parse 改为 parse_python_file mtime AST 缓存,Phase1 内重复收集与 Phase2 主进程解析共享同一缓存
This commit is contained in:
1 parent
7917be0fa1
commit
e3886a244d
2 files changed
+34
-13
No files matched your search
@@ -357,9 +357,9 @@ class Phase1Generator:
|
||||
continue
|
||||
pyi_path: str = os.path.join(self.temp_dir, fname)
|
||||
try:
|
||||
with open(pyi_path, 'r', encoding='utf-8') as f:
|
||||
content: str = f.read()
|
||||
tree: ast.Module = ast.parse(content)
|
||||
content, tree = parse_python_file(pyi_path)
|
||||
if tree is None:
|
||||
continue
|
||||
for node in ast.iter_child_nodes(tree):
|
||||
if isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name):
|
||||
var_name: str = node.target.id
|
||||
@@ -676,9 +676,9 @@ class Phase1Generator:
|
||||
continue
|
||||
pyi_path: str = os.path.join(self.temp_dir, fname)
|
||||
try:
|
||||
with open(pyi_path, 'r', encoding='utf-8') as f:
|
||||
content: str = f.read()
|
||||
tree: ast.Module = ast.parse(content)
|
||||
content, tree = parse_python_file(pyi_path)
|
||||
if tree is None:
|
||||
continue
|
||||
for node in ast.iter_child_nodes(tree):
|
||||
if isinstance(node, ast.ClassDef):
|
||||
is_enum: bool = False
|
||||
|
||||
Reference in new issue
Block a user