# TransPyV [English](#english) | [中文](#中文) The **Viper language compiler, written in Viper itself** — and it is **self-hosted**: TransPyV compiles its own source and produces `TransPyV.exe`. It translates Viper sources to LLVM IR, then drives `llc` + `clang++` to link a native Windows executable. The first-generation CPython compiler ([TransPyC](https://git.gvsds.com/GVSDS/TransPyC)) is now only kept as the bootstrapping **seed**. **用 Viper 语言编写的 Viper 编译器**,且**已完成自举**:TransPyV 能编译自身源码并产出 `TransPyV.exe`。它把 Viper 源码翻译为 LLVM IR,再调用 `llc` + `clang++` 链接为 Windows 原生可执行文件。第一代 CPython 编译器([TransPyC](https://git.gvsds.com/GVSDS/TransPyC))现在只作为自举**种子**保留。 --- ## 中文 ### 这是什么 `TransPyV` 是 Viper 的第二代编译器:**用 Viper 自己写、也被 Viper 自己编译**。整条链路上没有任何 Python 运行时参与编译 —— 词法/语法分析走 `includes/ast`(自研解析器,9 个文件),IR 生成走 `App/lib/core/Gen/`,目标代码由 `llc` 与 `clang++` 产出。 自举链路: ```text TransPyC(种子,CPython 实现) │ 只做一次:把 TransPyV 源码编译成 TransPyV.exe ▼ TransPyV.exe(第一阶段产物) │ 自举:用这个 exe 再编译 TransPyV 自身 ▼ TransPyV.exe(自举产物,当前仓库随包/随 exe\ 提供的那份) │ └─ 用它编译任意 Viper 工程(ViperApps\*、Test\* …) ``` 现状与规模(本机实测): | 项 | 值 | | --- | --- | | 编译器源码 | `App/` 下 **76 个 `.vp` 文件,64,656 行** | | 其中翻译器核心 | `App/lib/core/Gen/` **52 个文件,49,978 行** | | 阶段与合并 | `App/lib/Phase/` 6 文件 5,309 行;`App/lib/Stub/` 9 文件 6,877 行 | | 工程与工具 | `App/lib/Projectrans/` 3 文件 1,008 行 | | 自测用例 | `Test/` 下 **39 个 `.vp`**(`App/` 36 + `NegativeTest/` 2 + `Sha1Test/` 1) | | 平台 | Windows x64(源码直接 `import w32.*`) | > 自举可行性分析与历史数据见 [`wiki/13-bootstrapping.md`](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md)(该文标注:Phase A → Phase B/C 已跑通,编译器已脱离 Python 运行时)。 ### 目录结构 ```text TransPyV/ ├── project.vpj # 工程配置(source_dir=./App, build_dir=./.tpv_build, 输出 TransPyV.exe) ├── App/ # 编译器源码(Viper) │ ├── main.vp # CLI 入口:内存池初始化、单例注入、参数解析、Phase 调度 │ └── lib/ │ ├── Projectrans/ │ │ ├── Config.vp # project.vpj 加载与路径解析(含 includes 缓存目录) │ │ └── Utils.vp # SHA1 / 目录清理 / 杂项 │ ├── core/ │ │ ├── VLogger.vp # 日志(info/warn/error/banner/success) │ │ ├── IncludesScanner.vp # includes 目录扫描(决定哪些需重编译) │ │ ├── CompilerFactory.vp # 工厂:持有 mb + perm_pool 两个 MemBuddy 单例 │ │ └── Gen/ # 翻译器(52 文件)—— 把 AST 变为 LLVM IR │ │ ├── Generator.vp / Translator.vp / GenApi.vp / GenTypes.vp │ │ ├── BaseGenMixin / AssignGenMixin / VarGenMixin / ImportsGenMixin │ │ ├── MatchGenMixin / AsyncGenMixin / StructGenMixin / TranslatorGenMixin │ │ ├── Call/ (12) 调用生成:普通调用 / C 内建 / 构造 / 虚表 / 变参 / 内联汇编 … │ │ ├── Class/ (6) 类:定义 / 泛型 / 虚表 / 枚举 / Mixin │ │ ├── Expr/ (9) 表达式:名字 / 常量 / 运算 / 属性 / f-string / CDef … │ │ ├── Func/ (3) 函数定义 │ │ ├── Stmt/ (3) 语句:声明 / C-if / Mixin │ │ └── Type/ (6) 类型解析与 C 类型映射 │ ├── Phase/ │ │ ├── BuildPipeline.vp # .ll → llc → .obj → clang++ → .exe(含 ASan 插桩分支) │ │ ├── Phase1/ # Phase 1:按需翻译 includes + stub 分离(.stub.ll / .text.ll) │ │ └── Phase2/ # Phase 2:多文件工程翻译、IR 组合、编译 │ └── Stub/ │ ├── StubGen/ # .vpi 读取 + 声明转换 │ └── StubMerger/ # 多份 .stub.ll / .text.ll 依赖排序与合并 └── Test/ # 编译器自测(Viper) ├── project.vpj ├── App/ # 36 个 .vp(多为 *_test.vp),入口 test_main.vp 聚合全部用例 ├── NegativeTest/ # 负向测试(故意编译失败) └── Sha1Test/ # SHA1 命名空间测试 ``` ### 前提条件 | 工具 | 说明 | | --- | --- | | Windows 10/11 x64 | 编译器源码直接 `import w32.*`,当前仅 Windows | | `llc` | LLVM 静态编译器(本机 22.1.8),由 `project.vpj` 的 `compiler.cmd` 指定 | | `clang++` | mingw-w64 工具链,`-fuse-ld=lld` 链接(本机 `D:/msys64/mingw64/bin/clang++.exe`) | | ≥ 1.5 GiB 空闲内存 | 启动即 `malloc` 1 GiB 主池 + 512 MiB 常驻池 | | [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) | **仅当需要重建自举种子时**才需要 | 目录布局(编译时需要能找到 `includes/`,`project.vpj` 里写的是相对路径 `../includes`): ```text TransPyC\ # 工作区根 ├── TransPyC\ # 第一代编译器(种子) ├── TransPyV\ # 本仓库 └── includes\ # Viper 标准库 + includes/ast 解析器 ``` ### 快速开始 ```bat :: 日常:直接用自举产物编译任意工程 exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean :: 编译并立即运行 exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean --run :: 只有在需要重建编译器本体(自举种子)时,才动用 TransPyC python TransPyC\Projectrans.py --project TransPyV\project.vpj --clean ``` 产物位置 = `<工程目录>\\output\`,即默认 `<工程>\.tpv_build\output\app.exe`; TransPyV 自身因 `linker.output` 为 `TransPyV.exe`,产物是 `TransPyV\.tpv_build\output\TransPyV.exe`。 跑编译器自测套件: ```bat exe\TransPyV.exe --project TransPyV\Test\project.vpj --clean --run ``` ### 命令行参数 | 参数 | 含义 | | --- | --- | | `--project ` | `project.vpj` 路径(不给则在当前目录、上级目录找) | | `--src ` | 单文件模式源目录(覆盖配置;入口取 `/main.vp`) | | `--temp ` | 声明接口临时目录(覆盖配置,默认 `/temp`) | | `--output ` | 输出目录(覆盖配置,默认 `/output`) | | `--phase 1\|2\|all` | 1=只做 stub 分离;2=只做 IR 组合+编译;all=全部(默认) | | `--cc ` | LLVM 编译器命令(覆盖配置) | | `--build-dir ` | 覆盖 `project.vpj` 的 `build_dir`(相对路径仍基于工程目录解析) | | `--clean` | 清理 temp / output / includes 预编译缓存 | | `--rebuild-includes` | 只删 includes 预编译缓存(`includes.binary`),强制重编 includes | | `--clear-cache` | 只清 temp 缓存(`.vpi` / `.stub.ll` / `.text.ll`),强制重生成 stub | | `--run` | 编译成功后立即执行产物 | | `--asan` | ASan 模式:先 `opt -passes=asan` 插桩;运行库须由 `project.vpj` 的 `linker.static_libs` 声明,否则 fail-fast | | `--phase2command ` | 不执行 `llc`/链接,只把命令写进指定 `.bat` | | `--nowarn-register` | 抑制 Phase A 依赖类 WARN(跨模块 `text.ll` 未生成,下遍收敛) | | `--no-llvmpeephole` | 关闭 FunctionPrint 的本地窥孔(T2 store/load 转发、T3 空 bitcast) | | `--detection` | 开启 MemBuddy 分配检测(`0xCDCD` 标记 + 记录表探针,查越界写) | | `--show-args` | 打印全部解析结果 | | `--version`, `-v` | 打印 Viper / VRTE / TransPyV 版本、提交号、目标三元组与 LLVM 版本 | | `--help`, `-h` | 打印帮助 | ### 架构 ```text source_dir 下全部 .vp/.py(多文件模式:递归扫描所有源文件) │ ├─ Phase 1(stub 分离) │ IncludesScanner 按需翻译 includes → Phase1/* │ Translator + Gen/* 生成 IR,再分别落盘: │ .stub.ll 声明 │ .text.ll 实现 │ ├─ Phase 2(组合 + 编译) │ StubMerger* 按拓扑序合并 本地 stub + 依赖 stub + 本地 text → 完整 IR │ BuildPipeline:完整 IR → llc → .obj → clang++(附加 includes 预编译 .obj)→ .exe │ └─ /output/ ``` **核心模块分工:** | 模块 | 职责 | | --- | --- | | `App/main.vp` | CLI 入口:内存池与单例初始化、参数解析、Phase 调度、`--run` | | `lib/Projectrans/Config.vp` | `project.vpj` 加载、路径解析、includes 缓存目录 | | `lib/core/CompilerFactory.vp` | 持有 `mb`(主池)与 `perm_pool`(常驻池)两个 `MemBuddy` | | `lib/core/Gen/Generator.vp` | 项目级 `Generator` 单例,Phase1/Phase2/BuildPipeline 全程复用 | | `lib/core/Gen/Translator.vp` | 翻译器主体,`translate()` 与 `dump_ir()`(`.stub.ll` / `.text.ll` / 完整 IR) | | `lib/core/Gen/GenApi.vp` | 翻译器对外接口与全局状态(如 temp 目录) | | `lib/Phase/Phase1/` | 扫描 includes、拓扑排序、按需生成声明 | | `lib/Phase/Phase2/` | 多文件工程:递归扫描源文件、翻译、IR 组合、调度编译 | | `lib/Phase/BuildPipeline.vp` | `.ll` → `llc` → `.obj` → `clang++` → `.exe`(ASan 走插桩分支) | | `lib/Stub/StubGen/` | 读取 `.vpi`,转换为 stub 声明 | | `lib/Stub/StubMerger/` | `stub.ll` / `text.ll` 的收集、依赖拓扑、IR 拼接与落盘 | | `lib/core/IncludesScanner.vp` | 扫描 `includes/`,决定哪些文件需要重编译 | | `lib/core/VLogger.vp` | 日志系统(含 `--nowarn-register` 开关) | > 自举相关注意点:自举后模块级全局变量会带 **SHA1 前缀**(模块隔离),因此像 `_g_gen` > 这类单例必须**逐模块赋值**(`GenApi._g_gen = gen` …),不能只在 `main.vp` 里赋裸名 —— > 否则部分模块读到自己的 `None`,表现为"全局名判定恒为 0"之类的误报。 ### 内存模型 全程无系统 `malloc` 抖动,只有两个 `MemBuddy` 内存池(`lib/memhub` 的伙伴分配器): - `POOL_SIZE = 1 GiB`(`1073741824`)—— 主池,`mb`,翻译过程的常规分配。 - `PERM_POOL_SIZE = 536871176`(≈512 MiB 可用)—— 常驻池,`perm_pool`,跨阶段存活的对象。 - 两者由 `CompilerFactory.factory` 单例持有;`mb` 另以 `sys._mbuddy` / `argparse._mbuddy` / `ast._mbuddy` / `Config._mbuddy` … 的形式逐模块注入。 `--detection` 会把两个池都切到检测模式(`0xCDCD` 填充 + 记录表探针),用于排查越界写。 ### 与 TransPyC 的关系 | 维度 | TransPyC | TransPyV | | --- | --- | --- | | 实现语言 | CPython | Viper | | 定位 | 自举**种子**(只用来生成稳定的 TransPyV) | **现行编译器** | | 自举状态 | — | ✅ 已完成,能编译自身 | | 入口 | `TransPyC/Projectrans.py` | `App/main.vp` | | AST 解析 | Python `ast` / 自研 vast | `includes/ast` 自研(9 文件) | | 工程配置 | `project.json` | `project.vpj` | | 平台 | Windows / Linux | Windows(`w32.*`) | ### 进一步阅读 - [TransPyC 编译器仓库(种子)](https://git.gvsds.com/GVSDS/TransPyC) - [Viper 语言文档(wiki 目录)](https://git.gvsds.com/GVSDS/TransPyC/tree/main/wiki) - [自举可行性分析与进展](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md) - [工程配置 `project.vpj` 说明](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/12-project.md) - [ViperOS 操作系统(用 Viper 写的 OS)](https://git.gvsds.com/GVSDS/ViperOS) --- ## English ### What is this `TransPyV` is the second-generation Viper compiler: **written in Viper and compiled by Viper**. No Python runtime takes part in the compilation path — lexing/parsing goes through `includes/ast` (a self-implemented parser, 9 files), IR generation through `App/lib/core/Gen/`, and machine code through `llc` + `clang++`. The bootstrap chain: ```text TransPyC (seed, implemented in CPython) │ one-time only: compile TransPyV sources into TransPyV.exe ▼ TransPyV.exe (stage-one artifact) │ bootstrap: this exe compiles TransPyV itself ▼ TransPyV.exe (self-hosted artifact — the one shipped in exe\) │ └─ used to build any Viper project (ViperApps\*, Test\* …) ``` Current state and size (measured on the dev machine): | Item | Value | | --- | --- | | Compiler sources | **76 `.vp` files, 64,656 lines** under `App/` | | Translator core | `App/lib/core/Gen/` — **52 files, 49,978 lines** | | Phases & merging | `App/lib/Phase/` 6 files 5,309 lines; `App/lib/Stub/` 9 files 6,877 lines | | Project & utils | `App/lib/Projectrans/` 3 files 1,008 lines | | Self-tests | **39 `.vp`** under `Test/` (`App/` 36 + `NegativeTest/` 2 + `Sha1Test/` 1) | | Platform | Windows x64 (sources `import w32.*` directly) | ### Directory layout ```text TransPyV/ ├── project.vpj # project config (source_dir=./App, build_dir=./.tpv_build, output TransPyV.exe) ├── App/ # compiler sources (Viper) │ ├── main.vp # CLI entry: pools, singleton injection, arg parsing, phase dispatch │ └── lib/ │ ├── Projectrans/ │ │ ├── Config.vp # project.vpj loader + path resolution (incl. includes cache dir) │ │ └── Utils.vp # SHA1 / dir cleanup / misc │ ├── core/ │ │ ├── VLogger.vp # logging (info/warn/error/banner/success) │ │ ├── IncludesScanner.vp # scan includes/, decide what must be rebuilt │ │ ├── CompilerFactory.vp # holds the two MemBuddy singletons (mb + perm_pool) │ │ └── Gen/ # translator (52 files) — AST to LLVM IR │ │ ├── Generator.vp / Translator.vp / GenApi.vp / GenTypes.vp │ │ ├── BaseGenMixin / AssignGenMixin / VarGenMixin / ImportsGenMixin │ │ ├── MatchGenMixin / AsyncGenMixin / StructGenMixin / TranslatorGenMixin │ │ ├── Call/ (12) calls: plain / C builtin / ctor / vtable / varargs / inline asm … │ │ ├── Class/ (6) classes: def / generic / vtable / enum / mixin │ │ ├── Expr/ (9) expressions: name / const / ops / attr / f-string / CDef … │ │ ├── Func/ (3) function definitions │ │ ├── Stmt/ (3) statements: decl / C-if / mixin │ │ └── Type/ (6) type resolution and C type mapping │ ├── Phase/ │ │ ├── BuildPipeline.vp # .ll -> llc -> .obj -> clang++ -> .exe (plus ASan branch) │ │ ├── Phase1/ # Phase 1: on-demand includes translation + stub split (.stub.ll / .text.ll) │ │ └── Phase2/ # Phase 2: multi-file translation, IR assembly, compile │ └── Stub/ │ ├── StubGen/ # .vpi reader + declaration converter │ └── StubMerger/ # dependency-ordered merge of .stub.ll / .text.ll └── Test/ # compiler self-tests (Viper) ├── project.vpj ├── App/ # 36 .vp (mostly *_test.vp); entry point test_main.vp aggregates them all ├── NegativeTest/ # negative tests (must fail to compile) └── Sha1Test/ # SHA1 namespace test ``` ### Prerequisites | Tool | Notes | | --- | --- | | Windows 10/11 x64 | sources `import w32.*`; Windows-only for now | | `llc` | LLVM static compiler (22.1.8 here), set by `compiler.cmd` in `project.vpj` | | `clang++` | mingw-w64 toolchain, `-fuse-ld=lld` (here `D:/msys64/mingw64/bin/clang++.exe`) | | ≥ 1.5 GiB free RAM | allocates a 1 GiB main pool + 512 MiB permanent pool at startup | | [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) | **only needed to rebuild the bootstrap seed** | Required layout (`project.vpj` refers to `../includes`): ```text TransPyC\ # workspace root ├── TransPyC\ # first-generation compiler (seed) ├── TransPyV\ # this repo └── includes\ # Viper standard library + includes/ast parser ``` ### Quick start ```bat :: everyday use: build any project with the self-hosted compiler exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean :: build and run immediately exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean --run :: only when rebuilding the compiler itself (bootstrap seed), use TransPyC python TransPyC\Projectrans.py --project TransPyV\project.vpj --clean ``` Artifacts land in `\\output\` — by default `\.tpv_build\output\app.exe`. TransPyV itself sets `linker.output` to `TransPyV.exe`, so its own product is `TransPyV\.tpv_build\output\TransPyV.exe`. Run the compiler self-test suite: ```bat exe\TransPyV.exe --project TransPyV\Test\project.vpj --clean --run ``` ### CLI flags | Flag | Meaning | | --- | --- | | `--project ` | `project.vpj` path (default: current dir, then parent) | | `--src ` | single-file mode source dir (overrides config; entry `/main.vp`) | | `--temp ` | declaration temp dir (default `/temp`) | | `--output ` | output dir (default `/output`) | | `--phase 1\|2\|all` | 1 = stub split only; 2 = IR assembly + compile only; all = both (default) | | `--cc ` | LLVM compiler command (overrides config) | | `--build-dir ` | override `build_dir` from `project.vpj` (relative paths still resolve against the project dir) | | `--clean` | clean temp / output / includes precompiled cache | | `--rebuild-includes` | delete only the includes cache (`includes.binary`), forcing a full includes rebuild | | `--clear-cache` | clear only temp (`.vpi` / `.stub.ll` / `.text.ll`), forcing stub regeneration | | `--run` | execute the produced binary on success | | `--asan` | ASan mode: `opt -passes=asan` first; runtimes must be declared in `project.vpj` `linker.static_libs`, else fail-fast | | `--phase2command ` | do not run `llc`/link; write the commands into the given `.bat` | | `--nowarn-register` | suppress Phase A dependency WARNs (cross-module `text.ll` not yet emitted, converges next pass) | | `--no-llvmpeephole` | disable FunctionPrint local peepholes (T2 store/load forwarding, T3 no-op bitcast) | | `--detection` | enable MemBuddy allocation detection (`0xCDCD` fill + record-table probe for out-of-bounds writes) | | `--show-args` | dump all parsed arguments | | `--version`, `-v` | print Viper / VRTE / TransPyV versions, commit ids, target triple and LLVM version | | `--help`, `-h` | print help | ### Architecture ```text all .vp/.py under source_dir (multi-file mode: recursively scanned) │ ├─ Phase 1 (stub split) │ IncludesScanner translates includes on demand -> Phase1/* │ Translator + Gen/* emit IR, then write two files: │ .stub.ll declarations │ .text.ll implementations │ ├─ Phase 2 (assemble + compile) │ StubMerger* topologically merge local stub + dependency stubs + local text -> full IR │ BuildPipeline: full IR -> llc -> .obj -> clang++ (+ precompiled includes .obj) -> .exe │ └─ /output/ ``` **Module responsibilities:** | Module | Responsibility | | --- | --- | | `App/main.vp` | CLI entry: pools and singletons, arg parsing, phase dispatch, `--run` | | `lib/Projectrans/Config.vp` | `project.vpj` loading, path resolution, includes cache dir | | `lib/core/CompilerFactory.vp` | owns `mb` (main pool) and `perm_pool` (permanent pool) | | `lib/core/Gen/Generator.vp` | project-level `Generator` singleton shared by Phase1/Phase2/BuildPipeline | | `lib/core/Gen/Translator.vp` | translator body: `translate()` and `dump_ir()` (`.stub.ll` / `.text.ll` / full IR) | | `lib/core/Gen/GenApi.vp` | translator public API and global state (e.g. temp dir) | | `lib/Phase/Phase1/` | scan includes, topological order, emit declarations on demand | | `lib/Phase/Phase2/` | multi-file projects: scan sources, translate, assemble IR, drive the build | | `lib/Phase/BuildPipeline.vp` | `.ll` -> `llc` -> `.obj` -> `clang++` -> `.exe` (ASan branch included) | | `lib/Stub/StubGen/` | read `.vpi`, convert to stub declarations | | `lib/Stub/StubMerger/` | collect / topologically sort / concatenate / store `stub.ll` and `text.ll` | | `lib/core/IncludesScanner.vp` | scan `includes/`, decide what needs rebuilding | | `lib/core/VLogger.vp` | logging (with the `--nowarn-register` switch) | > Bootstrap note: after self-hosting, module-level globals carry a **SHA1 prefix** (module > isolation), so singletons such as `_g_gen` must be assigned **per module** > (`GenApi._g_gen = gen` …), not once by bare name in `main.vp` — otherwise some modules read > their own `None`, showing up as bogus results like "global-name test always 0". ### Memory model There is no per-object system `malloc` churn — only two `MemBuddy` buddy-allocator pools: - `POOL_SIZE = 1 GiB` (`1073741824`) — main pool `mb`, ordinary translation allocations. - `PERM_POOL_SIZE = 536871176` (≈512 MiB usable) — permanent pool `perm_pool`, objects that outlive a phase. - Both are held by the `CompilerFactory.factory` singleton; `mb` is additionally injected per module as `sys._mbuddy` / `argparse._mbuddy` / `ast._mbuddy` / `Config._mbuddy` … `--detection` switches both pools into detection mode (`0xCDCD` fill + record-table probe) to hunt out-of-bounds writes. ### Relation to TransPyC | Dimension | TransPyC | TransPyV | | --- | --- | --- | | Implementation | CPython | Viper | | Role | bootstrap **seed** (only used to produce a stable TransPyV) | **the current compiler** | | Self-hosted? | — | ✅ yes, it compiles itself | | Entry | `TransPyC/Projectrans.py` | `App/main.vp` | | AST parsing | Python `ast` / in-house vast | `includes/ast` (9 files, in-house) | | Project config | `project.json` | `project.vpj` | | Platform | Windows / Linux | Windows (`w32.*`) | ### Further reading - [TransPyC compiler repo (the seed)](https://git.gvsds.com/GVSDS/TransPyC) - [Viper language docs (wiki)](https://git.gvsds.com/GVSDS/TransPyC/tree/main/wiki) - [Bootstrapping analysis and progress](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md) - [Project config `project.vpj`](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/12-project.md) - [ViperOS (an OS written in Viper)](https://git.gvsds.com/GVSDS/ViperOS)