perf(compile): TPC 编译提速——并行 worker 满核 + vast 解析器热点优化
cProfile 分析(206s 全量重编译 TPV):~163s 为 ProcessPool 等待(16 核机器仅用 8 worker),vast.parse 35.9s(1231 次)为最大 Python 计算热点。 - Phase2Translator:4 处 n_workers 上限 8 → os.cpu_count()(16 核满负荷),并行翻译/llc 编译利用率翻倍 - vast/Tokenizer._PosInfo:isascii 快速路径 + 行索引增量查找(~300 万次调用,免每次 bisect+encode) - vast/Tokenizer._EmitOp:运算符按首字符分组,58 万次调用从 42 项全表 startswith 降到 2-3 项 - vast/Tokenizer._ScanName:内联 _IsIdentPart(70 万次循环免函数调用) - Projectrans/Utils:find_reachable 用 deque + 后缀索引替代 pop(0)/全表扫描;topological_sort 用反向邻接表 + heapq 替代每轮 sort+全图扫描
This commit is contained in:
1 parent
5d93692baa
commit
58ce673f01
4 files changed
+101
-44
No files matched your search
@@ -417,7 +417,7 @@ class Phase2Translator:
|
||||
_vlog().info(f"\n需要翻译 {len(need_translate)} 个文件")
|
||||
t_start = time.time()
|
||||
|
||||
n_workers = min(8, len(need_translate))
|
||||
n_workers = min(os.cpu_count() or 4, len(need_translate))
|
||||
|
||||
if n_workers > 1 and len(need_translate) > 1:
|
||||
_vlog().info(f" 并行翻译 (workers={n_workers})")
|
||||
@@ -1643,7 +1643,7 @@ class Phase2Translator:
|
||||
need_compile.append(ll_path)
|
||||
|
||||
if need_compile:
|
||||
n_workers = min(os.cpu_count() or 4, len(need_compile), 8)
|
||||
n_workers = min(os.cpu_count() or 4, len(need_compile))
|
||||
_vlog().info(f" 并行编译 {len(need_compile)} 个文件 (workers={n_workers})")
|
||||
errors = []
|
||||
with ThreadPoolExecutor(max_workers=n_workers) as executor:
|
||||
@@ -2167,7 +2167,7 @@ class Phase2Translator:
|
||||
shared_pickle_path = ''
|
||||
|
||||
if shared_pickle_path and os.path.exists(shared_pickle_path):
|
||||
n_workers = min(8, len(need_translate))
|
||||
n_workers = min(os.cpu_count() or 4, len(need_translate))
|
||||
errors = []
|
||||
# 波次并行翻译:按依赖关系分批提交,确保被依赖的文件先翻译完成。
|
||||
# 根因:_TryLoadStructFromStub 扫描 output 目录加载跨模块 struct 定义,
|
||||
@@ -2392,7 +2392,7 @@ class Phase2Translator:
|
||||
return (task, result)
|
||||
|
||||
if len(compile_tasks) > 1:
|
||||
n_workers = min(8, len(compile_tasks))
|
||||
n_workers = min(os.cpu_count() or 4, len(compile_tasks))
|
||||
with ThreadPoolExecutor(max_workers=n_workers) as executor:
|
||||
futures = {executor.submit(_compile_one_include, task): task for task in compile_tasks}
|
||||
done_count = 0
|
||||
|
||||
Reference in new issue
Block a user