23 KiB
23 KiB
TransPyV
The Viper language compiler, written in Viper itself — and it is self-hosted: TransPyV compiles its own source and produces TransPyV.exe. It translates Viper sources to LLVM IR, then drives llc + clang++ to link a native Windows executable. The first-generation CPython compiler (TransPyC) is now only kept as the bootstrapping seed.
用 Viper 语言编写的 Viper 编译器，且已完成自举：TransPyV 能编译自身源码并产出 TransPyV.exe。它把 Viper 源码翻译为 LLVM IR，再调用 llc + clang++ 链接为 Windows 原生可执行文件。第一代 CPython 编译器（TransPyC）现在只作为自举种子保留。
中文
这是什么
TransPyV 是 Viper 的第二代编译器：用 Viper 自己写、也被 Viper 自己编译。整条链路上没有任何 Python 运行时参与编译 —— 词法/语法分析走 includes/ast（自研解析器，9 个文件），IR 生成走 App/lib/core/Gen/，目标代码由 llc 与 clang++ 产出。
自举链路：
TransPyC（种子，CPython 实现）
│ 只做一次：把 TransPyV 源码编译成 TransPyV.exe
▼
TransPyV.exe（第一阶段产物）
│ 自举：用这个 exe 再编译 TransPyV 自身
▼
TransPyV.exe（自举产物，当前仓库随包/随 exe\ 提供的那份）
│
└─ 用它编译任意 Viper 工程（ViperApps\*、Test\* …）
现状与规模（本机实测）：
| 项 | 值 |
|---|---|
| 编译器源码 | App/ 下 76 个 .vp 文件，64,656 行 |
| 其中翻译器核心 | App/lib/core/Gen/ 52 个文件，49,978 行 |
| 阶段与合并 | App/lib/Phase/ 6 文件 5,309 行；App/lib/Stub/ 9 文件 6,877 行 |
| 工程与工具 | App/lib/Projectrans/ 3 文件 1,008 行 |
| 自测用例 | Test/ 下 39 个 .vp（App/ 36 + NegativeTest/ 2 + Sha1Test/ 1） |
| 平台 | Windows x64（源码直接 import w32.*） |
自举可行性分析与历史数据见
wiki/13-bootstrapping.md（该文标注：Phase A → Phase B/C 已跑通，编译器已脱离 Python 运行时）。
目录结构
TransPyV/
├── project.vpj # 工程配置（source_dir=./App, build_dir=./.tpv_build, 输出 TransPyV.exe）
├── App/ # 编译器源码（Viper）
│ ├── main.vp # CLI 入口：内存池初始化、单例注入、参数解析、Phase 调度
│ └── lib/
│ ├── Projectrans/
│ │ ├── Config.vp # project.vpj 加载与路径解析（含 includes 缓存目录）
│ │ └── Utils.vp # SHA1 / 目录清理 / 杂项
│ ├── core/
│ │ ├── VLogger.vp # 日志（info/warn/error/banner/success）
│ │ ├── IncludesScanner.vp # includes 目录扫描（决定哪些需重编译）
│ │ ├── CompilerFactory.vp # 工厂：持有 mb + perm_pool 两个 MemBuddy 单例
│ │ └── Gen/ # 翻译器（52 文件）—— 把 AST 变为 LLVM IR
│ │ ├── Generator.vp / Translator.vp / GenApi.vp / GenTypes.vp
│ │ ├── BaseGenMixin / AssignGenMixin / VarGenMixin / ImportsGenMixin
│ │ ├── MatchGenMixin / AsyncGenMixin / StructGenMixin / TranslatorGenMixin
│ │ ├── Call/ (12) 调用生成：普通调用 / C 内建 / 构造 / 虚表 / 变参 / 内联汇编 …
│ │ ├── Class/ (6) 类：定义 / 泛型 / 虚表 / 枚举 / Mixin
│ │ ├── Expr/ (9) 表达式：名字 / 常量 / 运算 / 属性 / f-string / CDef …
│ │ ├── Func/ (3) 函数定义
│ │ ├── Stmt/ (3) 语句：声明 / C-if / Mixin
│ │ └── Type/ (6) 类型解析与 C 类型映射
│ ├── Phase/
│ │ ├── BuildPipeline.vp # .ll → llc → .obj → clang++ → .exe（含 ASan 插桩分支）
│ │ ├── Phase1/ # Phase 1：按需翻译 includes + stub 分离（.stub.ll / .text.ll）
│ │ └── Phase2/ # Phase 2：多文件工程翻译、IR 组合、编译
│ └── Stub/
│ ├── StubGen/ # .vpi 读取 + 声明转换
│ └── StubMerger/ # 多份 .stub.ll / .text.ll 依赖排序与合并
└── Test/ # 编译器自测（Viper）
├── project.vpj
├── App/ # 36 个 .vp（多为 *_test.vp），入口 test_main.vp 聚合全部用例
├── NegativeTest/ # 负向测试（故意编译失败）
└── Sha1Test/ # SHA1 命名空间测试
前提条件
| 工具 | 说明 |
|---|---|
| Windows 10/11 x64 | 编译器源码直接 import w32.*，当前仅 Windows |
llc |
LLVM 静态编译器（本机 22.1.8），由 project.vpj 的 compiler.cmd 指定 |
clang++ |
mingw-w64 工具链，-fuse-ld=lld 链接（本机 D:/msys64/mingw64/bin/clang++.exe） |
| ≥ 1.5 GiB 空闲内存 | 启动即 malloc 1 GiB 主池 + 512 MiB 常驻池 |
| TransPyC | 仅当需要重建自举种子时才需要 |
目录布局（编译时需要能找到 includes/，project.vpj 里写的是相对路径 ../includes）：
TransPyC\ # 工作区根
├── TransPyC\ # 第一代编译器（种子）
├── TransPyV\ # 本仓库
└── includes\ # Viper 标准库 + includes/ast 解析器
快速开始
:: 日常：直接用自举产物编译任意工程
exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean
:: 编译并立即运行
exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean --run
:: 只有在需要重建编译器本体（自举种子）时，才动用 TransPyC
python TransPyC\Projectrans.py --project TransPyV\project.vpj --clean
产物位置 = <工程目录>\<build_dir>\output\<linker.output>，即默认 <工程>\.tpv_build\output\app.exe；
TransPyV 自身因 linker.output 为 TransPyV.exe，产物是 TransPyV\.tpv_build\output\TransPyV.exe。
跑编译器自测套件：
exe\TransPyV.exe --project TransPyV\Test\project.vpj --clean --run
命令行参数
| 参数 | 含义 |
|---|---|
--project <path> |
project.vpj 路径（不给则在当前目录、上级目录找） |
--src <dir> |
单文件模式源目录（覆盖配置；入口取 <dir>/main.vp） |
--temp <dir> |
声明接口临时目录（覆盖配置，默认 <build_dir>/temp） |
--output <dir> |
输出目录（覆盖配置，默认 <build_dir>/output） |
--phase 1|2|all |
1=只做 stub 分离；2=只做 IR 组合+编译；all=全部（默认） |
--cc <cmd> |
LLVM 编译器命令（覆盖配置） |
--build-dir <dir> |
覆盖 project.vpj 的 build_dir（相对路径仍基于工程目录解析） |
--clean |
清理 temp / output / includes 预编译缓存 |
--rebuild-includes |
只删 includes 预编译缓存（includes.binary），强制重编 includes |
--clear-cache |
只清 temp 缓存（.vpi / .stub.ll / .text.ll），强制重生成 stub |
--run |
编译成功后立即执行产物 |
--repl |
交互式 REPL（M1：表达式片段 → DLL → 调用回显）；未给 --project 时按 cwd → TransPyV.exe 同目录查找 repl.vpj 模板工程（找不到则报错并提示 --project）；与 --asan / --phase2command 互斥 |
--asan |
ASan 模式：先 opt -passes=asan 插桩；运行库须由 project.vpj 的 linker.static_libs 声明，否则 fail-fast |
--phase2command <bat> |
不执行 llc/链接，只把命令写进指定 .bat |
--nowarn-register |
抑制 Phase A 依赖类 WARN（跨模块 text.ll 未生成，下遍收敛） |
--no-llvmpeephole |
关闭 FunctionPrint 的本地窥孔（T2 store/load 转发、T3 空 bitcast） |
--detection |
开启 MemBuddy 分配检测（0xCDCD 标记 + 记录表探针，查越界写） |
--show-args |
打印全部解析结果 |
--version, -v |
打印 Viper / VRTE / TransPyV 版本、提交号、目标三元组与 LLVM 版本 |
--help, -h |
打印帮助 |
架构
source_dir 下全部 .vp/.py（多文件模式：递归扫描所有源文件）
│
├─ Phase 1（stub 分离）
│ IncludesScanner 按需翻译 includes → Phase1/*
│ Translator + Gen/* 生成 IR，再分别落盘：
│ <sha1>.stub.ll 声明
│ <sha1>.text.ll 实现
│
├─ Phase 2（组合 + 编译）
│ StubMerger* 按拓扑序合并 本地 stub + 依赖 stub + 本地 text → 完整 IR
│ BuildPipeline：完整 IR → llc → .obj → clang++（附加 includes 预编译 .obj）→ .exe
│
└─ <build_dir>/output/<linker.output>
核心模块分工：
| 模块 | 职责 |
|---|---|
App/main.vp |
CLI 入口：内存池与单例初始化、参数解析、Phase 调度、--run |
lib/Projectrans/Config.vp |
project.vpj 加载、路径解析、includes 缓存目录 |
lib/core/CompilerFactory.vp |
持有 mb（主池）与 perm_pool（常驻池）两个 MemBuddy |
lib/core/Gen/Generator.vp |
项目级 Generator 单例，Phase1/Phase2/BuildPipeline 全程复用 |
lib/core/Gen/Translator.vp |
翻译器主体，translate() 与 dump_ir()（.stub.ll / .text.ll / 完整 IR） |
lib/core/Gen/GenApi.vp |
翻译器对外接口与全局状态（如 temp 目录） |
lib/Phase/Phase1/ |
扫描 includes、拓扑排序、按需生成声明 |
lib/Phase/Phase2/ |
多文件工程：递归扫描源文件、翻译、IR 组合、调度编译 |
lib/Phase/BuildPipeline.vp |
.ll → llc → .obj → clang++ → .exe（ASan 走插桩分支） |
lib/Stub/StubGen/ |
读取 .vpi，转换为 stub 声明 |
lib/Stub/StubMerger/ |
stub.ll / text.ll 的收集、依赖拓扑、IR 拼接与落盘 |
lib/core/IncludesScanner.vp |
扫描 includes/，决定哪些文件需要重编译 |
lib/core/VLogger.vp |
日志系统（含 --nowarn-register 开关） |
自举相关注意点：自举后模块级全局变量会带 SHA1 前缀（模块隔离），因此像
_g_gen这类单例必须逐模块赋值（GenApi._g_gen = gen…），不能只在main.vp里赋裸名 —— 否则部分模块读到自己的None，表现为"全局名判定恒为 0"之类的误报。
内存模型
全程无系统 malloc 抖动，只有两个 MemBuddy 内存池（lib/memhub 的伙伴分配器）：
POOL_SIZE = 1 GiB（1073741824）—— 主池，mb，翻译过程的常规分配。PERM_POOL_SIZE = 536871176（≈512 MiB 可用）—— 常驻池，perm_pool，跨阶段存活的对象。- 两者由
CompilerFactory.factory单例持有；mb另以sys._mbuddy/argparse._mbuddy/ast._mbuddy/Config._mbuddy… 的形式逐模块注入。
--detection 会把两个池都切到检测模式（0xCDCD 填充 + 记录表探针），用于排查越界写。
与 TransPyC 的关系
| 维度 | TransPyC | TransPyV |
|---|---|---|
| 实现语言 | CPython | Viper |
| 定位 | 自举种子（只用来生成稳定的 TransPyV） | 现行编译器 |
| 自举状态 | — | ✅ 已完成，能编译自身 |
| 入口 | TransPyC/Projectrans.py |
App/main.vp |
| AST 解析 | Python ast / 自研 vast |
includes/ast 自研（9 文件） |
| 工程配置 | project.json |
project.vpj |
| 平台 | Windows / Linux | Windows（w32.*） |
进一步阅读
English
What is this
TransPyV is the second-generation Viper compiler: written in Viper and compiled by Viper. No Python runtime takes part in the compilation path — lexing/parsing goes through includes/ast (a self-implemented parser, 9 files), IR generation through App/lib/core/Gen/, and machine code through llc + clang++.
The bootstrap chain:
TransPyC (seed, implemented in CPython)
│ one-time only: compile TransPyV sources into TransPyV.exe
▼
TransPyV.exe (stage-one artifact)
│ bootstrap: this exe compiles TransPyV itself
▼
TransPyV.exe (self-hosted artifact — the one shipped in exe\)
│
└─ used to build any Viper project (ViperApps\*, Test\* …)
Current state and size (measured on the dev machine):
| Item | Value |
|---|---|
| Compiler sources | 76 .vp files, 64,656 lines under App/ |
| Translator core | App/lib/core/Gen/ — 52 files, 49,978 lines |
| Phases & merging | App/lib/Phase/ 6 files 5,309 lines; App/lib/Stub/ 9 files 6,877 lines |
| Project & utils | App/lib/Projectrans/ 3 files 1,008 lines |
| Self-tests | 39 .vp under Test/ (App/ 36 + NegativeTest/ 2 + Sha1Test/ 1) |
| Platform | Windows x64 (sources import w32.* directly) |
Directory layout
TransPyV/
├── project.vpj # project config (source_dir=./App, build_dir=./.tpv_build, output TransPyV.exe)
├── App/ # compiler sources (Viper)
│ ├── main.vp # CLI entry: pools, singleton injection, arg parsing, phase dispatch
│ └── lib/
│ ├── Projectrans/
│ │ ├── Config.vp # project.vpj loader + path resolution (incl. includes cache dir)
│ │ └── Utils.vp # SHA1 / dir cleanup / misc
│ ├── core/
│ │ ├── VLogger.vp # logging (info/warn/error/banner/success)
│ │ ├── IncludesScanner.vp # scan includes/, decide what must be rebuilt
│ │ ├── CompilerFactory.vp # holds the two MemBuddy singletons (mb + perm_pool)
│ │ └── Gen/ # translator (52 files) — AST to LLVM IR
│ │ ├── Generator.vp / Translator.vp / GenApi.vp / GenTypes.vp
│ │ ├── BaseGenMixin / AssignGenMixin / VarGenMixin / ImportsGenMixin
│ │ ├── MatchGenMixin / AsyncGenMixin / StructGenMixin / TranslatorGenMixin
│ │ ├── Call/ (12) calls: plain / C builtin / ctor / vtable / varargs / inline asm …
│ │ ├── Class/ (6) classes: def / generic / vtable / enum / mixin
│ │ ├── Expr/ (9) expressions: name / const / ops / attr / f-string / CDef …
│ │ ├── Func/ (3) function definitions
│ │ ├── Stmt/ (3) statements: decl / C-if / mixin
│ │ └── Type/ (6) type resolution and C type mapping
│ ├── Phase/
│ │ ├── BuildPipeline.vp # .ll -> llc -> .obj -> clang++ -> .exe (plus ASan branch)
│ │ ├── Phase1/ # Phase 1: on-demand includes translation + stub split (.stub.ll / .text.ll)
│ │ └── Phase2/ # Phase 2: multi-file translation, IR assembly, compile
│ └── Stub/
│ ├── StubGen/ # .vpi reader + declaration converter
│ └── StubMerger/ # dependency-ordered merge of .stub.ll / .text.ll
└── Test/ # compiler self-tests (Viper)
├── project.vpj
├── App/ # 36 .vp (mostly *_test.vp); entry point test_main.vp aggregates them all
├── NegativeTest/ # negative tests (must fail to compile)
└── Sha1Test/ # SHA1 namespace test
Prerequisites
| Tool | Notes |
|---|---|
| Windows 10/11 x64 | sources import w32.*; Windows-only for now |
llc |
LLVM static compiler (22.1.8 here), set by compiler.cmd in project.vpj |
clang++ |
mingw-w64 toolchain, -fuse-ld=lld (here D:/msys64/mingw64/bin/clang++.exe) |
| ≥ 1.5 GiB free RAM | allocates a 1 GiB main pool + 512 MiB permanent pool at startup |
| TransPyC | only needed to rebuild the bootstrap seed |
Required layout (project.vpj refers to ../includes):
TransPyC\ # workspace root
├── TransPyC\ # first-generation compiler (seed)
├── TransPyV\ # this repo
└── includes\ # Viper standard library + includes/ast parser
Quick start
:: everyday use: build any project with the self-hosted compiler
exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean
:: build and run immediately
exe\TransPyV.exe --project ViperApps\ViperMNIST\project.vpj --clean --run
:: only when rebuilding the compiler itself (bootstrap seed), use TransPyC
python TransPyC\Projectrans.py --project TransPyV\project.vpj --clean
Artifacts land in <project dir>\<build_dir>\output\<linker.output> — by default
<project>\.tpv_build\output\app.exe. TransPyV itself sets linker.output to TransPyV.exe,
so its own product is TransPyV\.tpv_build\output\TransPyV.exe.
Run the compiler self-test suite:
exe\TransPyV.exe --project TransPyV\Test\project.vpj --clean --run
CLI flags
| Flag | Meaning |
|---|---|
--project <path> |
project.vpj path (default: current dir, then parent) |
--src <dir> |
single-file mode source dir (overrides config; entry <dir>/main.vp) |
--temp <dir> |
declaration temp dir (default <build_dir>/temp) |
--output <dir> |
output dir (default <build_dir>/output) |
--phase 1|2|all |
1 = stub split only; 2 = IR assembly + compile only; all = both (default) |
--cc <cmd> |
LLVM compiler command (overrides config) |
--build-dir <dir> |
override build_dir from project.vpj (relative paths still resolve against the project dir) |
--clean |
clean temp / output / includes precompiled cache |
--rebuild-includes |
delete only the includes cache (includes.binary), forcing a full includes rebuild |
--clear-cache |
clear only temp (.vpi / .stub.ll / .text.ll), forcing stub regeneration |
--run |
execute the produced binary on success |
--repl |
interactive REPL (M1: expression fragment → DLL → call and echo); without --project it looks for the repl.vpj template project in cwd, then next to TransPyV.exe (errors out with a --project hint if neither exists); mutually exclusive with --asan / --phase2command |
--asan |
ASan mode: opt -passes=asan first; runtimes must be declared in project.vpj linker.static_libs, else fail-fast |
--phase2command <bat> |
do not run llc/link; write the commands into the given .bat |
--nowarn-register |
suppress Phase A dependency WARNs (cross-module text.ll not yet emitted, converges next pass) |
--no-llvmpeephole |
disable FunctionPrint local peepholes (T2 store/load forwarding, T3 no-op bitcast) |
--detection |
enable MemBuddy allocation detection (0xCDCD fill + record-table probe for out-of-bounds writes) |
--show-args |
dump all parsed arguments |
--version, -v |
print Viper / VRTE / TransPyV versions, commit ids, target triple and LLVM version |
--help, -h |
print help |
Architecture
all .vp/.py under source_dir (multi-file mode: recursively scanned)
│
├─ Phase 1 (stub split)
│ IncludesScanner translates includes on demand -> Phase1/*
│ Translator + Gen/* emit IR, then write two files:
│ <sha1>.stub.ll declarations
│ <sha1>.text.ll implementations
│
├─ Phase 2 (assemble + compile)
│ StubMerger* topologically merge local stub + dependency stubs + local text -> full IR
│ BuildPipeline: full IR -> llc -> .obj -> clang++ (+ precompiled includes .obj) -> .exe
│
└─ <build_dir>/output/<linker.output>
Module responsibilities:
| Module | Responsibility |
|---|---|
App/main.vp |
CLI entry: pools and singletons, arg parsing, phase dispatch, --run |
lib/Projectrans/Config.vp |
project.vpj loading, path resolution, includes cache dir |
lib/core/CompilerFactory.vp |
owns mb (main pool) and perm_pool (permanent pool) |
lib/core/Gen/Generator.vp |
project-level Generator singleton shared by Phase1/Phase2/BuildPipeline |
lib/core/Gen/Translator.vp |
translator body: translate() and dump_ir() (.stub.ll / .text.ll / full IR) |
lib/core/Gen/GenApi.vp |
translator public API and global state (e.g. temp dir) |
lib/Phase/Phase1/ |
scan includes, topological order, emit declarations on demand |
lib/Phase/Phase2/ |
multi-file projects: scan sources, translate, assemble IR, drive the build |
lib/Phase/BuildPipeline.vp |
.ll -> llc -> .obj -> clang++ -> .exe (ASan branch included) |
lib/Stub/StubGen/ |
read .vpi, convert to stub declarations |
lib/Stub/StubMerger/ |
collect / topologically sort / concatenate / store stub.ll and text.ll |
lib/core/IncludesScanner.vp |
scan includes/, decide what needs rebuilding |
lib/core/VLogger.vp |
logging (with the --nowarn-register switch) |
Bootstrap note: after self-hosting, module-level globals carry a SHA1 prefix (module isolation), so singletons such as
_g_genmust be assigned per module (GenApi._g_gen = gen…), not once by bare name inmain.vp— otherwise some modules read their ownNone, showing up as bogus results like "global-name test always 0".
Memory model
There is no per-object system malloc churn — only two MemBuddy buddy-allocator pools:
POOL_SIZE = 1 GiB(1073741824) — main poolmb, ordinary translation allocations.PERM_POOL_SIZE = 536871176(≈512 MiB usable) — permanent poolperm_pool, objects that outlive a phase.- Both are held by the
CompilerFactory.factorysingleton;mbis additionally injected per module assys._mbuddy/argparse._mbuddy/ast._mbuddy/Config._mbuddy…
--detection switches both pools into detection mode (0xCDCD fill + record-table probe) to hunt
out-of-bounds writes.
Relation to TransPyC
| Dimension | TransPyC | TransPyV |
|---|---|---|
| Implementation | CPython | Viper |
| Role | bootstrap seed (only used to produce a stable TransPyV) | the current compiler |
| Self-hosted? | — | ✅ yes, it compiles itself |
| Entry | TransPyC/Projectrans.py |
App/main.vp |
| AST parsing | Python ast / in-house vast |
includes/ast (9 files, in-house) |
| Project config | project.json |
project.vpj |
| Platform | Windows / Linux | Windows (w32.*) |