15 KiB
15 KiB
TransPyV
The Viper language compiler, written in Viper itself. TransPyV is the self-hosted (bootstrapped) successor of TransPyC. It compiles Viper sources to LLVM IR, then drives llc + clang++ to produce a native Windows executable (TransPyV.exe).
用 Viper 语言编写的 Viper 编译器。TransPyV 是 TransPyC 的自举（self-hosted）后继版本。它把 Viper 源码编译为 LLVM IR，再调用 llc + clang++ 产出 Windows 原生可执行文件（TransPyV.exe）。
中文
这是什么
TransPyV 是 Viper 项目自举（bootstrapping）目标：用 Viper 自身重写编译器。TransPyC 是第一代（用 CPython 写的），TransPyV 是第二代（用 Viper 写，编译后用 LLVM 工具链链接）。
虽然 TransPyV 当前只在 Windows 上能跑通（App/main.py 直接 import w32.*），但它的目标是脱离 Python 运行时：将来一旦 TransPyV 能完整编译自身并产出可执行文件，就完成了 Viper 编译器的完全自举。
目录结构
TransPyV/
├── .gitignore
├── project.json # 顶层项目配置（输出 TransPyV.exe）
├── App/ # 编译器源码（Viper）
│ ├── main.py # CLI 入口
│ └── lib/
│ ├── Projectrans/
│ │ ├── Config.py # project.vpj / project.json 加载
│ │ └── Utils.py
│ └── core/
│ ├── BuildPipeline.py # .ll → .obj → .exe
│ ├── IncludesScanner.py # includes 目录扫描
│ ├── Phase1.py # stub 分离（生成 .stub.ll + .text.ll）
│ ├── Phase2.py # 多文件项目翻译
│ ├── StubMerger.py # stub 合并
│ ├── VLogger.py # 日志系统
│ └── Handles/ # AST 节点翻译（22 个）
│ ├── HandlesBase.py
│ ├── HandlesBody.py
│ ├── HandlesMain.py
│ ├── HandlesTranslator.py
│ ├── HandlesVar.py / Assign / AnnAssign / AugAssign
│ ├── HandlesIf / While / For
│ ├── HandlesExpr / ExprCall / ExprOps
│ ├── HandlesFunctions
│ ├── HandlesClassDef / Struct
│ ├── HandlesEnum
│ ├── HandlesImports
│ ├── HandlesReturn
│ ├── HandlesType
│ └── HandlesNonlocal
└── Test/ # 编译器的自测用例
├── project.vpj
├── App/ # 33 个测试 .py
│ ├── simple_test.py
│ ├── func_test.py / func_vtable_test.py
│ ├── oop_test.py / vtable_test.py / virtual_dispatch_test.py
│ ├── struct_test.py / class_test
│ ├── closure_test.py / namespace_test.py
│ ├── string_test.py / string_min_test.py
│ ├── flow_test.py / for_test.py
│ ├── llvmir_test.py
│ └── ... (33 个)
├── NegativeTest/ # 错误用法负向测试
└── Sha1Test/ # SHA1 命名空间测试
前提条件
| 工具 | 说明 |
|---|---|
python ≥ 3.10 |
仅当用 TransPyC 编译 TransPyV 时需要 |
llc |
LLVM 静态编译器 |
clang++ |
mingw-w64 工具链链接器 |
| Windows 10/11 x64 | TransPyV 当前仅在 Windows 上工作（import w32.*） |
| 至少 1GB 可用内存 | TransPyV 启动时 stdlib.malloc(POOL_SIZE=1<<30) |
| TransPyC | 用于把 TransPyV 自身编译为 .exe |
目录布局要求（编译 TransPyV 时）：
D:\Users\TermiNexus\Desktop\TransPyC\
├── TransPyC\ # 编译器仓库
└── TransPyV\ # 本仓库
快速开始
# 1. 用 TransPyC 编译 TransPyV（自举的"第一阶段"）
cd D:\Users\TermiNexus\Desktop\TransPyC
python Projectrans.py --project TransPyV\project.json --clean
# 2. 产物在 TransPyV\output\TransPyV.exe
.\TransPyV\output\TransPyV.exe --help
TransPyV CLI 参数（与 TransPyC Projectrans.py 一致）：
| 参数 | 含义 |
|---|---|
--project <path> |
project.json 路径（默认查找当前目录） |
--src <dir> |
源文件目录（覆盖 project.json） |
--temp <dir> |
声明接口临时目录（覆盖 project.json） |
--output <dir> |
输出目录（覆盖 project.json） |
--phase 1|2|all |
阶段：1=生成声明，2=翻译+编译，all=全部 |
--cc <cmd> |
LLVM 编译器命令（覆盖 project.json） |
--clean |
清理 output/ 和 temp/ |
--run |
编译成功后立即执行生成的可执行文件 |
--rebuild-includes |
删除 includes.binary 预编译缓存 |
--clear-cache |
清除 .transpyc_cache 全局缓存 |
架构
TransPyV 与 TransPyC 共享两阶段编译模型，但实现细节不同：
源文件 (.py)
│
├─ 阶段一：AST 解析 → 翻译 → stub 分离
│ Phase1.py / Handles/* → .stub.ll + .text.ll
│
├─ 阶段二：stub 合并 → 完整 IR → 编译
│ StubMerger.BuildCombinedIR
│ (本地 stub + 依赖 stubs + 本地 text)
│ BuildPipeline.run_pipeline
│ (.ll → llc → .obj → clang++ → .exe)
│
└─ 输出可执行文件
核心模块分工：
| 模块 | 职责 |
|---|---|
App/main.py |
CLI 入口：参数解析、内存池初始化、Phase 调度 |
lib/Projectrans/Config.py |
project.json / project.vpj 加载与路径解析 |
lib/Projectrans/Utils.py |
SHA1 计算、目录清理、杂项工具 |
lib/core/Phase1.py |
阶段一入口：扫描 includes、生成 stub |
lib/core/Phase2.py |
阶段二入口：多文件项目翻译 |
lib/core/IncludesScanner.py |
扫描 includes/ 目录、决定哪些文件需要重编译 |
lib/core/StubMerger.py |
把多个 .stub.ll 合并为完整 IR |
lib/core/BuildPipeline.py |
.ll → .obj → .exe（llc + clang++） |
lib/core/VLogger.py |
日志系统（info/warn/error/banner/success） |
lib/core/Handles/HandlesTranslator.py |
Translator 类：状态管理 + translate() 入口 |
lib/core/Handles/HandlesBody.py |
AST Body 分派（按节点类型路由到具体 Handler） |
lib/core/Handles/Handles*.py (22 个) |
每种 AST 节点的 LLVM IR 生成 |
内存模型
TransPyV 用 memhub.MemBuddy 作为全局内存池：
- POOL_SIZE = 1 GiB（
1073741824） - 注释（
App/main.py第 33 行）说明：Phase1 翻译 87 个 includes + Phase2 翻译 30 个测试文件时，512 MiB 不足 _mbuddy通过sys._mbuddy = mb之类的全局指针传递（参见App/main.py第 61-75 行）
与 TransPyC 的关系
| 维度 | TransPyC | TransPyV |
|---|---|---|
| 实现语言 | CPython | Viper |
| 自举状态 | ❌ 依赖 Python 运行时 | ✅ 目标完全自举 |
| 入口 | Projectrans.py |
App/main.py |
| AST 解析 | ast 标准库 |
ast（includes/ast 自研） |
| 当前状态 | 主路径，跑通所有测试 | 自举重写中 |
| 平台 | Windows / Linux | Windows（依赖 w32.*） |
当前 TransPyC 是"主路径"（所有 Test/ 在用），TransPyV 是"目标"——一旦 TransPyV 能编译自身产出可执行文件，Viper 就完成了完全自举。
进一步阅读
English
What is this
TransPyV is the Viper compiler, written in Viper itself — the bootstrapping target. TransPyC is the first generation (implemented in CPython); TransPyV is the second generation (implemented in Viper, linked with the LLVM toolchain).
While TransPyV currently only runs on Windows (its App/main.py directly imports w32.*), its goal is to escape the Python runtime: once TransPyV can compile itself end-to-end and produce a self-contained executable, Viper will be fully self-hosted.
Directory layout
TransPyV/
├── .gitignore
├── project.json # top-level project config (outputs TransPyV.exe)
├── App/ # compiler source (Viper)
│ ├── main.py # CLI entry
│ └── lib/
│ ├── Projectrans/
│ │ ├── Config.py # project.vpj / project.json loader
│ │ └── Utils.py
│ └── core/
│ ├── BuildPipeline.py # .ll -> .obj -> .exe
│ ├── IncludesScanner.py # includes directory scanner
│ ├── Phase1.py # stub split (generates .stub.ll + .text.ll)
│ ├── Phase2.py # multi-file project translation
│ ├── StubMerger.py # stub merger
│ ├── VLogger.py # logging
│ └── Handles/ # AST node translators (22 files)
│ ├── HandlesBase.py
│ ├── HandlesBody.py
│ ├── HandlesMain.py
│ ├── HandlesTranslator.py
│ ├── HandlesVar.py / Assign / AnnAssign / AugAssign
│ ├── HandlesIf / While / For
│ ├── HandlesExpr / ExprCall / ExprOps
│ ├── HandlesFunctions
│ ├── HandlesClassDef / Struct
│ ├── HandlesEnum
│ ├── HandlesImports
│ ├── HandlesReturn
│ ├── HandlesType
│ └── HandlesNonlocal
└── Test/ # compiler self-test cases
├── project.vpj
├── App/ # 33 test .py
│ ├── simple_test.py
│ ├── func_test.py / func_vtable_test.py
│ ├── oop_test.py / vtable_test.py / virtual_dispatch_test.py
│ ├── struct_test.py
│ ├── closure_test.py / namespace_test.py
│ ├── string_test.py / string_min_test.py
│ ├── flow_test.py / for_test.py
│ ├── llvmir_test.py
│ └── ... (33 in total)
├── NegativeTest/ # negative tests (error cases)
└── Sha1Test/ # SHA1 namespace tests
Prerequisites
| Tool | Notes |
|---|---|
python ≥ 3.10 |
only needed when compiling TransPyV with TransPyC |
llc |
LLVM static compiler |
clang++ |
mingw-w64 toolchain linker |
| Windows 10/11 x64 | TransPyV currently Windows-only (imports w32.*) |
| ≥ 1 GiB free RAM | TransPyV allocates POOL_SIZE=1<<30 at startup |
| TransPyC | needed to compile TransPyV itself into an .exe |
Required layout (to compile TransPyV):
D:\Users\TermiNexus\Desktop\TransPyC\
├── TransPyC\ # compiler repo
└── TransPyV\ # this repo
Quick start
# 1. Use TransPyC to compile TransPyV (the "first stage" of bootstrapping)
cd D:\Users\TermiNexus\Desktop\TransPyC
python Projectrans.py --project TransPyV\project.json --clean
# 2. Output is at TransPyV\output\TransPyV.exe
.\TransPyV\output\TransPyV.exe --help
TransPyV CLI flags (mirrors Projectrans.py):
| Flag | Meaning |
|---|---|
--project <path> |
project.json path (default: search current dir) |
--src <dir> |
source dir (overrides project.json) |
--temp <dir> |
declaration temp dir (overrides project.json) |
--output <dir> |
output dir (overrides project.json) |
--phase 1|2|all |
phase: 1=decl only, 2=translate+compile, all=both |
--cc <cmd> |
LLVM compiler command (overrides project.json) |
--clean |
clean output/ and temp/ |
--run |
execute the produced executable on success |
--rebuild-includes |
delete includes.binary precompiled cache |
--clear-cache |
clear .transpyc_cache global cache |
Architecture
TransPyV shares the two-phase model with TransPyC, with these differences:
Source files (.py)
│
├─ Phase 1: AST parse -> translate -> stub split
│ Phase1.py / Handles/* -> .stub.ll + .text.ll
│
├─ Phase 2: stub merge -> full IR -> compile
│ StubMerger.BuildCombinedIR
│ (local stub + dep stubs + local text)
│ BuildPipeline.run_pipeline
│ (.ll -> llc -> .obj -> clang++ -> .exe)
│
└─ Native executable
Module responsibilities:
| Module | Responsibility |
|---|---|
App/main.py |
CLI entry: arg parse, memory pool init, phase dispatch |
lib/Projectrans/Config.py |
project.json / project.vpj loader + path resolution |
lib/Projectrans/Utils.py |
SHA1, dir cleanup, misc utilities |
lib/core/Phase1.py |
Phase 1 entry: scan includes, generate stubs |
lib/core/Phase2.py |
Phase 2 entry: multi-file project translation |
lib/core/IncludesScanner.py |
scan includes/, decide what to rebuild |
lib/core/StubMerger.py |
merge multiple .stub.ll into a single IR |
lib/core/BuildPipeline.py |
.ll -> .obj -> .exe (llc + clang++) |
lib/core/VLogger.py |
logging (info/warn/error/banner/success) |
lib/core/Handles/HandlesTranslator.py |
Translator class: state + translate() |
lib/core/Handles/HandlesBody.py |
AST body dispatcher (route by node type) |
lib/core/Handles/Handles*.py (22) |
one LLVM-IR generator per AST node type |
Memory model
TransPyV uses memhub.MemBuddy as a global arena:
- POOL_SIZE = 1 GiB (
1073741824) - The comment on
App/main.pyline 33 explains: Phase1 translates 87 includes files + Phase2 translates 30 test files, 512 MiB is not enough _mbuddyis propagated via globals likesys._mbuddy = mb(seeApp/main.pylines 61–75)
Relation to TransPyC
| Dimension | TransPyC | TransPyV |
|---|---|---|
| Implementation | CPython | Viper |
| Self-hosted? | No (Python runtime required) | Target: yes |
| Entry | Projectrans.py |
App/main.py |
| AST parsing | standard ast |
ast (self-implemented in includes/ast) |
| Current status | Main path, all tests pass | Bootstrapping rewrite in progress |
| Platform | Windows / Linux | Windows only (depends on w32.*) |
Currently TransPyC is the main path (all Test/ use it); TransPyV is the target. Once TransPyV can compile itself into an executable, Viper will be fully self-hosted.