From bd46b47b22b4e9539d70430c47a2fe2bd508e419 Mon Sep 17 00:00:00 2001 From: TermiNexus Date: Thu, 10 Sep 2026 22:33:23 +0800 Subject: [PATCH] =?UTF-8?q?docs(readme):=20=E6=8C=89=E4=BA=8B=E5=AE=9E?= =?UTF-8?q?=E9=87=8D=E5=86=99=20README,=20=E5=8F=8D=E6=98=A0=E8=87=AA?= =?UTF-8?q?=E4=B8=BE=E5=B7=B2=E5=AE=8C=E6=88=90=E4=B8=8E=E7=9C=9F=E5=AE=9E?= =?UTF-8?q?=E6=BA=90=E7=A0=81=E7=BB=93=E6=9E=84?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 自举状态: 重写中 -> 已完成自主编译, TransPyC 明确为自举种子 - 目录结构: 全文 .py 名称纠正为真实 .vp 布局 (Phase/ Stub/ core/Gen/ 树, Gen 52 文件) - 工程配置: project.json -> project.vpj; 产物路径 output/ -> .tpv_build/output/ - CLI 表补齐 --asan/--phase2command/--nowarn-register/--no-llvmpeephole/--detection/--show-args/--build-dir/--version - 内存模型: 补常驻池 512 MiB 与 CompilerFactory 单例, 说明自举后 _mbuddy 需逐模块注入 - 规模数据改为实测值 (App 76 文件 64656 行 / Gen 52 文件 / Test 39 用例) - 修复失效链接 wiki/01-overview.md --- README.md | 543 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 313 insertions(+), 230 deletions(-) diff --git a/README.md b/README.md index 4b6bbf8..e840b0c 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,9 @@ [English](#english) | [中文](#中文) -The **Viper language compiler, written in Viper itself**. TransPyV is the self-hosted (bootstrapped) successor of [TransPyC](https://git.gvsds.com/GVSDS/TransPyC). It compiles Viper sources to LLVM IR, then drives `llc` + `clang++` to produce a native Windows executable (`TransPyV.exe`). +The **Viper language compiler, written in Viper itself** — and it is **self-hosted**: TransPyV compiles its own source and produces `TransPyV.exe`. It translates Viper sources to LLVM IR, then drives `llc` + `clang++` to link a native Windows executable. The first-generation CPython compiler ([TransPyC](https://git.gvsds.com/GVSDS/TransPyC)) is now only kept as the bootstrapping **seed**. -**用 Viper 语言编写的 Viper 编译器**。TransPyV 是 [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) 的自举（self-hosted）后继版本。它把 Viper 源码编译为 LLVM IR，再调用 `llc` + `clang++` 产出 Windows 原生可执行文件（`TransPyV.exe`）。 +**用 Viper 语言编写的 Viper 编译器**，且**已完成自举**：TransPyV 能编译自身源码并产出 `TransPyV.exe`。它把 Viper 源码翻译为 LLVM IR，再调用 `llc` + `clang++` 链接为 Windows 原生可执行文件。第一代 CPython 编译器（[TransPyC](https://git.gvsds.com/GVSDS/TransPyC)）现在只作为自举**种子**保留。 --- @@ -14,167 +14,208 @@ The **Viper language compiler, written in Viper itself**. TransPyV is the self-h ### 这是什么 -`TransPyV` 是 Viper 项目**自举（bootstrapping）目标**：用 Viper 自身重写编译器。TransPyC 是第一代（用 CPython 写的），TransPyV 是第二代（用 Viper 写，编译后用 LLVM 工具链链接）。 +`TransPyV` 是 Viper 的第二代编译器：**用 Viper 自己写、也被 Viper 自己编译**。整条链路上没有任何 Python 运行时参与编译 —— 词法/语法分析走 `includes/ast`（自研解析器，9 个文件），IR 生成走 `App/lib/core/Gen/`，目标代码由 `llc` 与 `clang++` 产出。 -虽然 TransPyV 当前只在 Windows 上能跑通（`App/main.py` 直接 import `w32.*`），但它的目标是**脱离 Python 运行时**：将来一旦 TransPyV 能完整编译自身并产出可执行文件，就完成了 Viper 编译器的完全自举。 +自举链路： + +```text +TransPyC（种子，CPython 实现） + │ 只做一次：把 TransPyV 源码编译成 TransPyV.exe + ▼ +TransPyV.exe（第一阶段产物） + │ 自举：用这个 exe 再编译 TransPyV 自身 + ▼ +TransPyV.exe（自举产物，当前仓库随包/随 exe\ 提供的那份） + │ + └─ 用它编译任意 Viper 工程（ViperApps\*、Test\* …） +``` + +现状与规模（本机实测）： + +| 项 | 值 | +| --- | --- | +| 编译器源码 | `App/` 下 **76 个 `.vp` 文件，64,656 行** | +| 其中翻译器核心 | `App/lib/core/Gen/` **52 个文件，49,978 行** | +| 阶段与合并 | `App/lib/Phase/` 6 文件 5,309 行；`App/lib/Stub/` 9 文件 6,877 行 | +| 工程与工具 | `App/lib/Projectrans/` 3 文件 1,008 行 | +| 自测用例 | `Test/` 下 **39 个 `.vp`**（`App/` 36 + `NegativeTest/` 2 + `Sha1Test/` 1） | +| 平台 | Windows x64（源码直接 `import w32.*`） | + +> 自举可行性分析与历史数据见 [`wiki/13-bootstrapping.md`](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md)（该文标注：Phase A → Phase B/C 已跑通，编译器已脱离 Python 运行时）。 ### 目录结构 -``` +```text TransPyV/ -├── .gitignore -├── project.json # 顶层项目配置（输出 TransPyV.exe） -├── App/ # 编译器源码（Viper） -│ ├── main.py # CLI 入口 +├── project.vpj # 工程配置（source_dir=./App, build_dir=./.tpv_build, 输出 TransPyV.exe） +├── App/ # 编译器源码（Viper） +│ ├── main.vp # CLI 入口：内存池初始化、单例注入、参数解析、Phase 调度 │ └── lib/ │ ├── Projectrans/ -│ │ ├── Config.py # project.vpj / project.json 加载 -│ │ └── Utils.py -│ └── core/ -│ ├── BuildPipeline.py # .ll → .obj → .exe -│ ├── IncludesScanner.py # includes 目录扫描 -│ ├── Phase1.py # stub 分离（生成 .stub.ll + .text.ll） -│ ├── Phase2.py # 多文件项目翻译 -│ ├── StubMerger.py # stub 合并 -│ ├── VLogger.py # 日志系统 -│ └── Handles/ # AST 节点翻译（22 个） -│ ├── HandlesBase.py -│ ├── HandlesBody.py -│ ├── HandlesMain.py -│ ├── HandlesTranslator.py -│ ├── HandlesVar.py / Assign / AnnAssign / AugAssign -│ ├── HandlesIf / While / For -│ ├── HandlesExpr / ExprCall / ExprOps -│ ├── HandlesFunctions -│ ├── HandlesClassDef / Struct -│ ├── HandlesEnum -│ ├── HandlesImports -│ ├── HandlesReturn -│ ├── HandlesType -│ └── HandlesNonlocal -└── Test/ # 编译器的自测用例 +│ │ ├── Config.vp # project.vpj 加载与路径解析（含 includes 缓存目录） +│ │ └── Utils.vp # SHA1 / 目录清理 / 杂项 +│ ├── core/ +│ │ ├── VLogger.vp # 日志（info/warn/error/banner/success） +│ │ ├── IncludesScanner.vp # includes 目录扫描（决定哪些需重编译） +│ │ ├── CompilerFactory.vp # 工厂：持有 mb + perm_pool 两个 MemBuddy 单例 +│ │ └── Gen/ # 翻译器（52 文件）—— 把 AST 变为 LLVM IR +│ │ ├── Generator.vp / Translator.vp / GenApi.vp / GenTypes.vp +│ │ ├── BaseGenMixin / AssignGenMixin / VarGenMixin / ImportsGenMixin +│ │ ├── MatchGenMixin / AsyncGenMixin / StructGenMixin / TranslatorGenMixin +│ │ ├── Call/ (12) 调用生成：普通调用 / C 内建 / 构造 / 虚表 / 变参 / 内联汇编 … +│ │ ├── Class/ (6) 类：定义 / 泛型 / 虚表 / 枚举 / Mixin +│ │ ├── Expr/ (9) 表达式：名字 / 常量 / 运算 / 属性 / f-string / CDef … +│ │ ├── Func/ (3) 函数定义 +│ │ ├── Stmt/ (3) 语句：声明 / C-if / Mixin +│ │ └── Type/ (6) 类型解析与 C 类型映射 +│ ├── Phase/ +│ │ ├── BuildPipeline.vp # .ll → llc → .obj → clang++ → .exe（含 ASan 插桩分支） +│ │ ├── Phase1/ # Phase 1：按需翻译 includes + stub 分离（.stub.ll / .text.ll） +│ │ └── Phase2/ # Phase 2：多文件工程翻译、IR 组合、编译 +│ └── Stub/ +│ ├── StubGen/ # .vpi 读取 + 声明转换 +│ └── StubMerger/ # 多份 .stub.ll / .text.ll 依赖排序与合并 +└── Test/ # 编译器自测（Viper） ├── project.vpj - ├── App/ # 33 个测试 .py - │ ├── simple_test.py - │ ├── func_test.py / func_vtable_test.py - │ ├── oop_test.py / vtable_test.py / virtual_dispatch_test.py - │ ├── struct_test.py / class_test - │ ├── closure_test.py / namespace_test.py - │ ├── string_test.py / string_min_test.py - │ ├── flow_test.py / for_test.py - │ ├── llvmir_test.py - │ └── ... (33 个) - ├── NegativeTest/ # 错误用法负向测试 - └── Sha1Test/ # SHA1 命名空间测试 + ├── App/ # 36 个 .vp（多为 *_test.vp），入口 test_main.vp 聚合全部用例 + ├── NegativeTest/ # 负向测试（故意编译失败） + └── Sha1Test/ # SHA1 命名空间测试 ``` ### 前提条件 | 工具 | 说明 | -|------|------| -| `python` ≥ 3.10 | **仅当用 TransPyC 编译 TransPyV 时**需要 | -| `llc` | LLVM 静态编译器 | -| `clang++` | mingw-w64 工具链链接器 | -| Windows 10/11 x64 | TransPyV 当前仅在 Windows 上工作（import w32.*） | -| 至少 1GB 可用内存 | TransPyV 启动时 `stdlib.malloc(POOL_SIZE=1<<30)` | -| [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) | 用于把 TransPyV 自身编译为 `.exe` | +| --- | --- | +| Windows 10/11 x64 | 编译器源码直接 `import w32.*`，当前仅 Windows | +| `llc` | LLVM 静态编译器（本机 22.1.8），由 `project.vpj` 的 `compiler.cmd` 指定 | +| `clang++` | mingw-w64 工具链，`-fuse-ld=lld` 链接（本机 `D:/msys64/mingw64/bin/clang++.exe`） | +| ≥ 1.5 GiB 空闲内存 | 启动即 `malloc` 1 GiB 主池 + 512 MiB 常驻池 | +| [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) | **仅当需要重建自举种子时**才需要 | -目录布局要求（编译 TransPyV 时）： +目录布局（编译时需要能找到 `includes/`，`project.vpj` 里写的是相对路径 `../includes`）： -``` -D:\Users\TermiNexus\Desktop\TransPyC\ -├── TransPyC\ # 编译器仓库 -└── TransPyV\ # 本仓库 +```text +TransPyC\ # 工作区根 +├── TransPyC\ # 第一代编译器（种子） +├── TransPyV\ # 本仓库 +└── includes\ # Viper 标准库 + includes/ast 解析器 ``` ### 快速开始 -```powershell -# 1. 用 TransPyC 编译 TransPyV（自举的"第一阶段"） -cd D:\Users\TermiNexus\Desktop\TransPyC -python Projectrans.py --project TransPyV\project.json --clean +```bat +:: 日常：直接用自举产物编译任意工程 +exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean -# 2. 产物在 TransPyV\output\TransPyV.exe -.\TransPyV\output\TransPyV.exe --help +:: 编译并立即运行 +exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean --run + +:: 只有在需要重建编译器本体（自举种子）时，才动用 TransPyC +python TransPyC\Projectrans.py --project TransPyV\project.vpj --clean ``` -TransPyV CLI 参数（与 TransPyC `Projectrans.py` 一致）： +产物位置 = `<工程目录>\\output\`，即默认 `<工程>\.tpv_build\output\app.exe`； +TransPyV 自身因 `linker.output` 为 `TransPyV.exe`，产物是 `TransPyV\.tpv_build\output\TransPyV.exe`。 + +跑编译器自测套件： + +```bat +exe\TransPyV.exe --project TransPyV\Test\project.vpj --clean --run +``` + +### 命令行参数 | 参数 | 含义 | -|------|------| -| `--project ` | `project.json` 路径（默认查找当前目录） | -| `--src ` | 源文件目录（覆盖 `project.json`） | -| `--temp ` | 声明接口临时目录（覆盖 `project.json`） | -| `--output ` | 输出目录（覆盖 `project.json`） | -| `--phase 1\|2\|all` | 阶段：1=生成声明，2=翻译+编译，all=全部 | -| `--cc ` | LLVM 编译器命令（覆盖 `project.json`） | -| `--clean` | 清理 `output/` 和 `temp/` | -| `--run` | 编译成功后立即执行生成的可执行文件 | -| `--rebuild-includes` | 删除 `includes.binary` 预编译缓存 | -| `--clear-cache` | 清除 `.transpyc_cache` 全局缓存 | +| --- | --- | +| `--project ` | `project.vpj` 路径（不给则在当前目录、上级目录找） | +| `--src ` | 单文件模式源目录（覆盖配置；入口取 `/main.vp`） | +| `--temp ` | 声明接口临时目录（覆盖配置，默认 `/temp`） | +| `--output ` | 输出目录（覆盖配置，默认 `/output`） | +| `--phase 1\|2\|all` | 1=只做 stub 分离；2=只做 IR 组合+编译；all=全部（默认） | +| `--cc ` | LLVM 编译器命令（覆盖配置） | +| `--build-dir ` | 覆盖 `project.vpj` 的 `build_dir`（相对路径仍基于工程目录解析） | +| `--clean` | 清理 temp / output / includes 预编译缓存 | +| `--rebuild-includes` | 只删 includes 预编译缓存（`includes.binary`），强制重编 includes | +| `--clear-cache` | 只清 temp 缓存（`.vpi` / `.stub.ll` / `.text.ll`），强制重生成 stub | +| `--run` | 编译成功后立即执行产物 | +| `--asan` | ASan 模式：先 `opt -passes=asan` 插桩；运行库须由 `project.vpj` 的 `linker.static_libs` 声明，否则 fail-fast | +| `--phase2command ` | 不执行 `llc`/链接，只把命令写进指定 `.bat` | +| `--nowarn-register` | 抑制 Phase A 依赖类 WARN（跨模块 `text.ll` 未生成，下遍收敛） | +| `--no-llvmpeephole` | 关闭 FunctionPrint 的本地窥孔（T2 store/load 转发、T3 空 bitcast） | +| `--detection` | 开启 MemBuddy 分配检测（`0xCDCD` 标记 + 记录表探针，查越界写） | +| `--show-args` | 打印全部解析结果 | +| `--version`, `-v` | 打印 Viper / VRTE / TransPyV 版本、提交号、目标三元组与 LLVM 版本 | +| `--help`, `-h` | 打印帮助 | ### 架构 -TransPyV 与 TransPyC 共享两阶段编译模型，但实现细节不同： - -``` -源文件 (.py) +```text +source_dir 下全部 .vp/.py（多文件模式：递归扫描所有源文件） │ - ├─ 阶段一：AST 解析 → 翻译 → stub 分离 - │ Phase1.py / Handles/* → .stub.ll + .text.ll + ├─ Phase 1（stub 分离） + │ IncludesScanner 按需翻译 includes → Phase1/* + │ Translator + Gen/* 生成 IR，再分别落盘： + │ .stub.ll 声明 + │ .text.ll 实现 │ - ├─ 阶段二：stub 合并 → 完整 IR → 编译 - │ StubMerger.BuildCombinedIR - │ (本地 stub + 依赖 stubs + 本地 text) - │ BuildPipeline.run_pipeline - │ (.ll → llc → .obj → clang++ → .exe) + ├─ Phase 2（组合 + 编译） + │ StubMerger* 按拓扑序合并 本地 stub + 依赖 stub + 本地 text → 完整 IR + │ BuildPipeline：完整 IR → llc → .obj → clang++（附加 includes 预编译 .obj）→ .exe │ - └─ 输出可执行文件 + └─ /output/ ``` **核心模块分工：** | 模块 | 职责 | -|------|------| -| `App/main.py` | CLI 入口：参数解析、内存池初始化、Phase 调度 | -| `lib/Projectrans/Config.py` | `project.json` / `project.vpj` 加载与路径解析 | -| `lib/Projectrans/Utils.py` | SHA1 计算、目录清理、杂项工具 | -| `lib/core/Phase1.py` | 阶段一入口：扫描 includes、生成 stub | -| `lib/core/Phase2.py` | 阶段二入口：多文件项目翻译 | -| `lib/core/IncludesScanner.py` | 扫描 `includes/` 目录、决定哪些文件需要重编译 | -| `lib/core/StubMerger.py` | 把多个 `.stub.ll` 合并为完整 IR | -| `lib/core/BuildPipeline.py` | `.ll` → `.obj` → `.exe`（llc + clang++） | -| `lib/core/VLogger.py` | 日志系统（info/warn/error/banner/success） | -| `lib/core/Handles/HandlesTranslator.py` | `Translator` 类：状态管理 + `translate()` 入口 | -| `lib/core/Handles/HandlesBody.py` | AST Body 分派（按节点类型路由到具体 Handler） | -| `lib/core/Handles/Handles*.py` (22 个) | 每种 AST 节点的 LLVM IR 生成 | +| --- | --- | +| `App/main.vp` | CLI 入口：内存池与单例初始化、参数解析、Phase 调度、`--run` | +| `lib/Projectrans/Config.vp` | `project.vpj` 加载、路径解析、includes 缓存目录 | +| `lib/core/CompilerFactory.vp` | 持有 `mb`（主池）与 `perm_pool`（常驻池）两个 `MemBuddy` | +| `lib/core/Gen/Generator.vp` | 项目级 `Generator` 单例，Phase1/Phase2/BuildPipeline 全程复用 | +| `lib/core/Gen/Translator.vp` | 翻译器主体，`translate()` 与 `dump_ir()`（`.stub.ll` / `.text.ll` / 完整 IR） | +| `lib/core/Gen/GenApi.vp` | 翻译器对外接口与全局状态（如 temp 目录） | +| `lib/Phase/Phase1/` | 扫描 includes、拓扑排序、按需生成声明 | +| `lib/Phase/Phase2/` | 多文件工程：递归扫描源文件、翻译、IR 组合、调度编译 | +| `lib/Phase/BuildPipeline.vp` | `.ll` → `llc` → `.obj` → `clang++` → `.exe`（ASan 走插桩分支） | +| `lib/Stub/StubGen/` | 读取 `.vpi`，转换为 stub 声明 | +| `lib/Stub/StubMerger/` | `stub.ll` / `text.ll` 的收集、依赖拓扑、IR 拼接与落盘 | +| `lib/core/IncludesScanner.vp` | 扫描 `includes/`，决定哪些文件需要重编译 | +| `lib/core/VLogger.vp` | 日志系统（含 `--nowarn-register` 开关） | + +> 自举相关注意点：自举后模块级全局变量会带 **SHA1 前缀**（模块隔离），因此像 `_g_gen` +> 这类单例必须**逐模块赋值**（`GenApi._g_gen = gen` …），不能只在 `main.vp` 里赋裸名 —— +> 否则部分模块读到自己的 `None`，表现为"全局名判定恒为 0"之类的误报。 ### 内存模型 -TransPyV 用 `memhub.MemBuddy` 作为全局内存池： +全程无系统 `malloc` 抖动，只有两个 `MemBuddy` 内存池（`lib/memhub` 的伙伴分配器）： -- **POOL_SIZE = 1 GiB**（`1073741824`） -- 注释（`App/main.py` 第 33 行）说明：Phase1 翻译 87 个 includes + Phase2 翻译 30 个测试文件时，512 MiB 不足 -- `_mbuddy` 通过 `sys._mbuddy = mb` 之类的全局指针传递（参见 `App/main.py` 第 61-75 行） +- `POOL_SIZE = 1 GiB`（`1073741824`）—— 主池，`mb`，翻译过程的常规分配。 +- `PERM_POOL_SIZE = 536871176`（≈512 MiB 可用）—— 常驻池，`perm_pool`，跨阶段存活的对象。 +- 两者由 `CompilerFactory.factory` 单例持有；`mb` 另以 `sys._mbuddy` / `argparse._mbuddy` / + `ast._mbuddy` / `Config._mbuddy` … 的形式逐模块注入。 + +`--detection` 会把两个池都切到检测模式（`0xCDCD` 填充 + 记录表探针），用于排查越界写。 ### 与 TransPyC 的关系 | 维度 | TransPyC | TransPyV | -|------|----------|----------| +| --- | --- | --- | | 实现语言 | CPython | Viper | -| 自举状态 | ❌ 依赖 Python 运行时 | ✅ 目标完全自举 | -| 入口 | `Projectrans.py` | `App/main.py` | -| AST 解析 | `ast` 标准库 | `ast`（includes/ast 自研） | -| 当前状态 | 主路径，跑通所有测试 | 自举重写中 | -| 平台 | Windows / Linux | Windows（依赖 w32.*） | - -当前 TransPyC 是"主路径"（所有 Test/ 在用），TransPyV 是"目标"——一旦 TransPyV 能编译自身产出可执行文件，Viper 就完成了完全自举。 +| 定位 | 自举**种子**（只用来生成稳定的 TransPyV） | **现行编译器** | +| 自举状态 | — | ✅ 已完成，能编译自身 | +| 入口 | `TransPyC/Projectrans.py` | `App/main.vp` | +| AST 解析 | Python `ast` / 自研 vast | `includes/ast` 自研（9 文件） | +| 工程配置 | `project.json` | `project.vpj` | +| 平台 | Windows / Linux | Windows（`w32.*`） | ### 进一步阅读 -- [TransPyC 编译器仓库](https://git.gvsds.com/GVSDS/TransPyC) -- [Viper 语言概览](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/01-overview.md) -- [Viper 自举路线图](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md) +- [TransPyC 编译器仓库（种子）](https://git.gvsds.com/GVSDS/TransPyC) +- [Viper 语言文档（wiki 目录）](https://git.gvsds.com/GVSDS/TransPyC/tree/main/wiki) +- [自举可行性分析与进展](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md) +- [工程配置 `project.vpj` 说明](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/12-project.md) - [ViperOS 操作系统（用 Viper 写的 OS）](https://git.gvsds.com/GVSDS/ViperOS) --- @@ -185,165 +226,207 @@ TransPyV 用 `memhub.MemBuddy` 作为全局内存池： ### What is this -`TransPyV` is the **Viper compiler, written in Viper itself** — the bootstrapping target. TransPyC is the first generation (implemented in CPython); TransPyV is the second generation (implemented in Viper, linked with the LLVM toolchain). +`TransPyV` is the second-generation Viper compiler: **written in Viper and compiled by Viper**. No Python runtime takes part in the compilation path — lexing/parsing goes through `includes/ast` (a self-implemented parser, 9 files), IR generation through `App/lib/core/Gen/`, and machine code through `llc` + `clang++`. -While TransPyV currently only runs on Windows (its `App/main.py` directly imports `w32.*`), its goal is to **escape the Python runtime**: once TransPyV can compile itself end-to-end and produce a self-contained executable, Viper will be fully self-hosted. +The bootstrap chain: + +```text +TransPyC (seed, implemented in CPython) + │ one-time only: compile TransPyV sources into TransPyV.exe + ▼ +TransPyV.exe (stage-one artifact) + │ bootstrap: this exe compiles TransPyV itself + ▼ +TransPyV.exe (self-hosted artifact — the one shipped in exe\) + │ + └─ used to build any Viper project (ViperApps\*, Test\* …) +``` + +Current state and size (measured on the dev machine): + +| Item | Value | +| --- | --- | +| Compiler sources | **76 `.vp` files, 64,656 lines** under `App/` | +| Translator core | `App/lib/core/Gen/` — **52 files, 49,978 lines** | +| Phases & merging | `App/lib/Phase/` 6 files 5,309 lines; `App/lib/Stub/` 9 files 6,877 lines | +| Project & utils | `App/lib/Projectrans/` 3 files 1,008 lines | +| Self-tests | **39 `.vp`** under `Test/` (`App/` 36 + `NegativeTest/` 2 + `Sha1Test/` 1) | +| Platform | Windows x64 (sources `import w32.*` directly) | ### Directory layout -``` +```text TransPyV/ -├── .gitignore -├── project.json # top-level project config (outputs TransPyV.exe) -├── App/ # compiler source (Viper) -│ ├── main.py # CLI entry +├── project.vpj # project config (source_dir=./App, build_dir=./.tpv_build, output TransPyV.exe) +├── App/ # compiler sources (Viper) +│ ├── main.vp # CLI entry: pools, singleton injection, arg parsing, phase dispatch │ └── lib/ │ ├── Projectrans/ -│ │ ├── Config.py # project.vpj / project.json loader -│ │ └── Utils.py -│ └── core/ -│ ├── BuildPipeline.py # .ll -> .obj -> .exe -│ ├── IncludesScanner.py # includes directory scanner -│ ├── Phase1.py # stub split (generates .stub.ll + .text.ll) -│ ├── Phase2.py # multi-file project translation -│ ├── StubMerger.py # stub merger -│ ├── VLogger.py # logging -│ └── Handles/ # AST node translators (22 files) -│ ├── HandlesBase.py -│ ├── HandlesBody.py -│ ├── HandlesMain.py -│ ├── HandlesTranslator.py -│ ├── HandlesVar.py / Assign / AnnAssign / AugAssign -│ ├── HandlesIf / While / For -│ ├── HandlesExpr / ExprCall / ExprOps -│ ├── HandlesFunctions -│ ├── HandlesClassDef / Struct -│ ├── HandlesEnum -│ ├── HandlesImports -│ ├── HandlesReturn -│ ├── HandlesType -│ └── HandlesNonlocal -└── Test/ # compiler self-test cases +│ │ ├── Config.vp # project.vpj loader + path resolution (incl. includes cache dir) +│ │ └── Utils.vp # SHA1 / dir cleanup / misc +│ ├── core/ +│ │ ├── VLogger.vp # logging (info/warn/error/banner/success) +│ │ ├── IncludesScanner.vp # scan includes/, decide what must be rebuilt +│ │ ├── CompilerFactory.vp # holds the two MemBuddy singletons (mb + perm_pool) +│ │ └── Gen/ # translator (52 files) — AST to LLVM IR +│ │ ├── Generator.vp / Translator.vp / GenApi.vp / GenTypes.vp +│ │ ├── BaseGenMixin / AssignGenMixin / VarGenMixin / ImportsGenMixin +│ │ ├── MatchGenMixin / AsyncGenMixin / StructGenMixin / TranslatorGenMixin +│ │ ├── Call/ (12) calls: plain / C builtin / ctor / vtable / varargs / inline asm … +│ │ ├── Class/ (6) classes: def / generic / vtable / enum / mixin +│ │ ├── Expr/ (9) expressions: name / const / ops / attr / f-string / CDef … +│ │ ├── Func/ (3) function definitions +│ │ ├── Stmt/ (3) statements: decl / C-if / mixin +│ │ └── Type/ (6) type resolution and C type mapping +│ ├── Phase/ +│ │ ├── BuildPipeline.vp # .ll -> llc -> .obj -> clang++ -> .exe (plus ASan branch) +│ │ ├── Phase1/ # Phase 1: on-demand includes translation + stub split (.stub.ll / .text.ll) +│ │ └── Phase2/ # Phase 2: multi-file translation, IR assembly, compile +│ └── Stub/ +│ ├── StubGen/ # .vpi reader + declaration converter +│ └── StubMerger/ # dependency-ordered merge of .stub.ll / .text.ll +└── Test/ # compiler self-tests (Viper) ├── project.vpj - ├── App/ # 33 test .py - │ ├── simple_test.py - │ ├── func_test.py / func_vtable_test.py - │ ├── oop_test.py / vtable_test.py / virtual_dispatch_test.py - │ ├── struct_test.py - │ ├── closure_test.py / namespace_test.py - │ ├── string_test.py / string_min_test.py - │ ├── flow_test.py / for_test.py - │ ├── llvmir_test.py - │ └── ... (33 in total) - ├── NegativeTest/ # negative tests (error cases) - └── Sha1Test/ # SHA1 namespace tests + ├── App/ # 36 .vp (mostly *_test.vp); entry point test_main.vp aggregates them all + ├── NegativeTest/ # negative tests (must fail to compile) + └── Sha1Test/ # SHA1 namespace test ``` ### Prerequisites | Tool | Notes | -|------|-------| -| `python` ≥ 3.10 | only needed when compiling TransPyV with TransPyC | -| `llc` | LLVM static compiler | -| `clang++` | mingw-w64 toolchain linker | -| Windows 10/11 x64 | TransPyV currently Windows-only (imports `w32.*`) | -| ≥ 1 GiB free RAM | TransPyV allocates `POOL_SIZE=1<<30` at startup | -| [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) | needed to compile TransPyV itself into an `.exe` | +| --- | --- | +| Windows 10/11 x64 | sources `import w32.*`; Windows-only for now | +| `llc` | LLVM static compiler (22.1.8 here), set by `compiler.cmd` in `project.vpj` | +| `clang++` | mingw-w64 toolchain, `-fuse-ld=lld` (here `D:/msys64/mingw64/bin/clang++.exe`) | +| ≥ 1.5 GiB free RAM | allocates a 1 GiB main pool + 512 MiB permanent pool at startup | +| [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) | **only needed to rebuild the bootstrap seed** | -Required layout (to compile TransPyV): +Required layout (`project.vpj` refers to `../includes`): -``` -D:\Users\TermiNexus\Desktop\TransPyC\ -├── TransPyC\ # compiler repo -└── TransPyV\ # this repo +```text +TransPyC\ # workspace root +├── TransPyC\ # first-generation compiler (seed) +├── TransPyV\ # this repo +└── includes\ # Viper standard library + includes/ast parser ``` ### Quick start -```powershell -# 1. Use TransPyC to compile TransPyV (the "first stage" of bootstrapping) -cd D:\Users\TermiNexus\Desktop\TransPyC -python Projectrans.py --project TransPyV\project.json --clean +```bat +:: everyday use: build any project with the self-hosted compiler +exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean -# 2. Output is at TransPyV\output\TransPyV.exe -.\TransPyV\output\TransPyV.exe --help +:: build and run immediately +exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean --run + +:: only when rebuilding the compiler itself (bootstrap seed), use TransPyC +python TransPyC\Projectrans.py --project TransPyV\project.vpj --clean ``` -TransPyV CLI flags (mirrors `Projectrans.py`): +Artifacts land in `\\output\` — by default +`\.tpv_build\output\app.exe`. TransPyV itself sets `linker.output` to `TransPyV.exe`, +so its own product is `TransPyV\.tpv_build\output\TransPyV.exe`. + +Run the compiler self-test suite: + +```bat +exe\TransPyV.exe --project TransPyV\Test\project.vpj --clean --run +``` + +### CLI flags | Flag | Meaning | -|------|---------| -| `--project ` | `project.json` path (default: search current dir) | -| `--src ` | source dir (overrides `project.json`) | -| `--temp ` | declaration temp dir (overrides `project.json`) | -| `--output ` | output dir (overrides `project.json`) | -| `--phase 1\|2\|all` | phase: 1=decl only, 2=translate+compile, all=both | -| `--cc ` | LLVM compiler command (overrides `project.json`) | -| `--clean` | clean `output/` and `temp/` | -| `--run` | execute the produced executable on success | -| `--rebuild-includes` | delete `includes.binary` precompiled cache | -| `--clear-cache` | clear `.transpyc_cache` global cache | +| --- | --- | +| `--project ` | `project.vpj` path (default: current dir, then parent) | +| `--src ` | single-file mode source dir (overrides config; entry `/main.vp`) | +| `--temp ` | declaration temp dir (default `/temp`) | +| `--output ` | output dir (default `/output`) | +| `--phase 1\|2\|all` | 1 = stub split only; 2 = IR assembly + compile only; all = both (default) | +| `--cc ` | LLVM compiler command (overrides config) | +| `--build-dir ` | override `build_dir` from `project.vpj` (relative paths still resolve against the project dir) | +| `--clean` | clean temp / output / includes precompiled cache | +| `--rebuild-includes` | delete only the includes cache (`includes.binary`), forcing a full includes rebuild | +| `--clear-cache` | clear only temp (`.vpi` / `.stub.ll` / `.text.ll`), forcing stub regeneration | +| `--run` | execute the produced binary on success | +| `--asan` | ASan mode: `opt -passes=asan` first; runtimes must be declared in `project.vpj` `linker.static_libs`, else fail-fast | +| `--phase2command ` | do not run `llc`/link; write the commands into the given `.bat` | +| `--nowarn-register` | suppress Phase A dependency WARNs (cross-module `text.ll` not yet emitted, converges next pass) | +| `--no-llvmpeephole` | disable FunctionPrint local peepholes (T2 store/load forwarding, T3 no-op bitcast) | +| `--detection` | enable MemBuddy allocation detection (`0xCDCD` fill + record-table probe for out-of-bounds writes) | +| `--show-args` | dump all parsed arguments | +| `--version`, `-v` | print Viper / VRTE / TransPyV versions, commit ids, target triple and LLVM version | +| `--help`, `-h` | print help | ### Architecture -TransPyV shares the two-phase model with TransPyC, with these differences: - -``` -Source files (.py) +```text +all .vp/.py under source_dir (multi-file mode: recursively scanned) │ - ├─ Phase 1: AST parse -> translate -> stub split - │ Phase1.py / Handles/* -> .stub.ll + .text.ll + ├─ Phase 1 (stub split) + │ IncludesScanner translates includes on demand -> Phase1/* + │ Translator + Gen/* emit IR, then write two files: + │ .stub.ll declarations + │ .text.ll implementations │ - ├─ Phase 2: stub merge -> full IR -> compile - │ StubMerger.BuildCombinedIR - │ (local stub + dep stubs + local text) - │ BuildPipeline.run_pipeline - │ (.ll -> llc -> .obj -> clang++ -> .exe) + ├─ Phase 2 (assemble + compile) + │ StubMerger* topologically merge local stub + dependency stubs + local text -> full IR + │ BuildPipeline: full IR -> llc -> .obj -> clang++ (+ precompiled includes .obj) -> .exe │ - └─ Native executable + └─ /output/ ``` **Module responsibilities:** | Module | Responsibility | -|--------|---------------| -| `App/main.py` | CLI entry: arg parse, memory pool init, phase dispatch | -| `lib/Projectrans/Config.py` | `project.json` / `project.vpj` loader + path resolution | -| `lib/Projectrans/Utils.py` | SHA1, dir cleanup, misc utilities | -| `lib/core/Phase1.py` | Phase 1 entry: scan includes, generate stubs | -| `lib/core/Phase2.py` | Phase 2 entry: multi-file project translation | -| `lib/core/IncludesScanner.py` | scan `includes/`, decide what to rebuild | -| `lib/core/StubMerger.py` | merge multiple `.stub.ll` into a single IR | -| `lib/core/BuildPipeline.py` | `.ll` -> `.obj` -> `.exe` (llc + clang++) | -| `lib/core/VLogger.py` | logging (info/warn/error/banner/success) | -| `lib/core/Handles/HandlesTranslator.py` | `Translator` class: state + `translate()` | -| `lib/core/Handles/HandlesBody.py` | AST body dispatcher (route by node type) | -| `lib/core/Handles/Handles*.py` (22) | one LLVM-IR generator per AST node type | +| --- | --- | +| `App/main.vp` | CLI entry: pools and singletons, arg parsing, phase dispatch, `--run` | +| `lib/Projectrans/Config.vp` | `project.vpj` loading, path resolution, includes cache dir | +| `lib/core/CompilerFactory.vp` | owns `mb` (main pool) and `perm_pool` (permanent pool) | +| `lib/core/Gen/Generator.vp` | project-level `Generator` singleton shared by Phase1/Phase2/BuildPipeline | +| `lib/core/Gen/Translator.vp` | translator body: `translate()` and `dump_ir()` (`.stub.ll` / `.text.ll` / full IR) | +| `lib/core/Gen/GenApi.vp` | translator public API and global state (e.g. temp dir) | +| `lib/Phase/Phase1/` | scan includes, topological order, emit declarations on demand | +| `lib/Phase/Phase2/` | multi-file projects: scan sources, translate, assemble IR, drive the build | +| `lib/Phase/BuildPipeline.vp` | `.ll` -> `llc` -> `.obj` -> `clang++` -> `.exe` (ASan branch included) | +| `lib/Stub/StubGen/` | read `.vpi`, convert to stub declarations | +| `lib/Stub/StubMerger/` | collect / topologically sort / concatenate / store `stub.ll` and `text.ll` | +| `lib/core/IncludesScanner.vp` | scan `includes/`, decide what needs rebuilding | +| `lib/core/VLogger.vp` | logging (with the `--nowarn-register` switch) | + +> Bootstrap note: after self-hosting, module-level globals carry a **SHA1 prefix** (module +> isolation), so singletons such as `_g_gen` must be assigned **per module** +> (`GenApi._g_gen = gen` …), not once by bare name in `main.vp` — otherwise some modules read +> their own `None`, showing up as bogus results like "global-name test always 0". ### Memory model -TransPyV uses `memhub.MemBuddy` as a global arena: +There is no per-object system `malloc` churn — only two `MemBuddy` buddy-allocator pools: -- **POOL_SIZE = 1 GiB** (`1073741824`) -- The comment on `App/main.py` line 33 explains: Phase1 translates 87 includes files + Phase2 translates 30 test files, 512 MiB is not enough -- `_mbuddy` is propagated via globals like `sys._mbuddy = mb` (see `App/main.py` lines 61–75) +- `POOL_SIZE = 1 GiB` (`1073741824`) — main pool `mb`, ordinary translation allocations. +- `PERM_POOL_SIZE = 536871176` (≈512 MiB usable) — permanent pool `perm_pool`, objects that outlive a phase. +- Both are held by the `CompilerFactory.factory` singleton; `mb` is additionally injected per module + as `sys._mbuddy` / `argparse._mbuddy` / `ast._mbuddy` / `Config._mbuddy` … + +`--detection` switches both pools into detection mode (`0xCDCD` fill + record-table probe) to hunt +out-of-bounds writes. ### Relation to TransPyC | Dimension | TransPyC | TransPyV | -|-----------|----------|----------| +| --- | --- | --- | | Implementation | CPython | Viper | -| Self-hosted? | No (Python runtime required) | Target: yes | -| Entry | `Projectrans.py` | `App/main.py` | -| AST parsing | standard `ast` | `ast` (self-implemented in `includes/ast`) | -| Current status | Main path, all tests pass | Bootstrapping rewrite in progress | -| Platform | Windows / Linux | Windows only (depends on `w32.*`) | - -Currently TransPyC is the main path (all `Test/` use it); TransPyV is the target. Once TransPyV can compile itself into an executable, Viper will be fully self-hosted. +| Role | bootstrap **seed** (only used to produce a stable TransPyV) | **the current compiler** | +| Self-hosted? | — | ✅ yes, it compiles itself | +| Entry | `TransPyC/Projectrans.py` | `App/main.vp` | +| AST parsing | Python `ast` / in-house vast | `includes/ast` (9 files, in-house) | +| Project config | `project.json` | `project.vpj` | +| Platform | Windows / Linux | Windows (`w32.*`) | ### Further reading -- [TransPyC compiler repo](https://git.gvsds.com/GVSDS/TransPyC) -- [Viper language overview](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/01-overview.md) -- [Viper bootstrapping roadmap](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md) +- [TransPyC compiler repo (the seed)](https://git.gvsds.com/GVSDS/TransPyC) +- [Viper language docs (wiki)](https://git.gvsds.com/GVSDS/TransPyC/tree/main/wiki) +- [Bootstrapping analysis and progress](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md) +- [Project config `project.vpj`](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/12-project.md) - [ViperOS (an OS written in Viper)](https://git.gvsds.com/GVSDS/ViperOS)