350 lines
15 KiB
Markdown
350 lines
15 KiB
Markdown
# TransPyV
|
||
|
||
[English](#english) | [中文](#中文)
|
||
|
||
The **Viper language compiler, written in Viper itself**. TransPyV is the self-hosted (bootstrapped) successor of [TransPyC](https://git.gvsds.com/GVSDS/TransPyC). It compiles Viper sources to LLVM IR, then drives `llc` + `clang++` to produce a native Windows executable (`TransPyV.exe`).
|
||
|
||
**用 Viper 语言编写的 Viper 编译器**。TransPyV 是 [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) 的自举（self-hosted）后继版本。它把 Viper 源码编译为 LLVM IR，再调用 `llc` + `clang++` 产出 Windows 原生可执行文件（`TransPyV.exe`）。
|
||
|
||
---
|
||
|
||
<a id="中文"></a>
|
||
|
||
## 中文
|
||
|
||
### 这是什么
|
||
|
||
`TransPyV` 是 Viper 项目**自举（bootstrapping）目标**：用 Viper 自身重写编译器。TransPyC 是第一代（用 CPython 写的），TransPyV 是第二代（用 Viper 写，编译后用 LLVM 工具链链接）。
|
||
|
||
虽然 TransPyV 当前只在 Windows 上能跑通（`App/main.py` 直接 import `w32.*`），但它的目标是**脱离 Python 运行时**：将来一旦 TransPyV 能完整编译自身并产出可执行文件，就完成了 Viper 编译器的完全自举。
|
||
|
||
### 目录结构
|
||
|
||
```
|
||
TransPyV/
|
||
├── .gitignore
|
||
├── project.json # 顶层项目配置（输出 TransPyV.exe）
|
||
├── App/ # 编译器源码（Viper）
|
||
│ ├── main.py # CLI 入口
|
||
│ └── lib/
|
||
│ ├── Projectrans/
|
||
│ │ ├── Config.py # project.vpj / project.json 加载
|
||
│ │ └── Utils.py
|
||
│ └── core/
|
||
│ ├── BuildPipeline.py # .ll → .obj → .exe
|
||
│ ├── IncludesScanner.py # includes 目录扫描
|
||
│ ├── Phase1.py # stub 分离（生成 .stub.ll + .text.ll）
|
||
│ ├── Phase2.py # 多文件项目翻译
|
||
│ ├── StubMerger.py # stub 合并
|
||
│ ├── VLogger.py # 日志系统
|
||
│ └── Handles/ # AST 节点翻译（22 个）
|
||
│ ├── HandlesBase.py
|
||
│ ├── HandlesBody.py
|
||
│ ├── HandlesMain.py
|
||
│ ├── HandlesTranslator.py
|
||
│ ├── HandlesVar.py / Assign / AnnAssign / AugAssign
|
||
│ ├── HandlesIf / While / For
|
||
│ ├── HandlesExpr / ExprCall / ExprOps
|
||
│ ├── HandlesFunctions
|
||
│ ├── HandlesClassDef / Struct
|
||
│ ├── HandlesEnum
|
||
│ ├── HandlesImports
|
||
│ ├── HandlesReturn
|
||
│ ├── HandlesType
|
||
│ └── HandlesNonlocal
|
||
└── Test/ # 编译器的自测用例
|
||
├── project.vpj
|
||
├── App/ # 33 个测试 .py
|
||
│ ├── simple_test.py
|
||
│ ├── func_test.py / func_vtable_test.py
|
||
│ ├── oop_test.py / vtable_test.py / virtual_dispatch_test.py
|
||
│ ├── struct_test.py / class_test
|
||
│ ├── closure_test.py / namespace_test.py
|
||
│ ├── string_test.py / string_min_test.py
|
||
│ ├── flow_test.py / for_test.py
|
||
│ ├── llvmir_test.py
|
||
│ └── ... (33 个)
|
||
├── NegativeTest/ # 错误用法负向测试
|
||
└── Sha1Test/ # SHA1 命名空间测试
|
||
```
|
||
|
||
### 前提条件
|
||
|
||
| 工具 | 说明 |
|
||
|------|------|
|
||
| `python` ≥ 3.10 | **仅当用 TransPyC 编译 TransPyV 时**需要 |
|
||
| `llc` | LLVM 静态编译器 |
|
||
| `clang++` | mingw-w64 工具链链接器 |
|
||
| Windows 10/11 x64 | TransPyV 当前仅在 Windows 上工作（import w32.*） |
|
||
| 至少 1GB 可用内存 | TransPyV 启动时 `stdlib.malloc(POOL_SIZE=1<<30)` |
|
||
| [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) | 用于把 TransPyV 自身编译为 `.exe` |
|
||
|
||
目录布局要求（编译 TransPyV 时）：
|
||
|
||
```
|
||
D:\Users\TermiNexus\Desktop\TransPyC\
|
||
├── TransPyC\ # 编译器仓库
|
||
└── TransPyV\ # 本仓库
|
||
```
|
||
|
||
### 快速开始
|
||
|
||
```powershell
|
||
# 1. 用 TransPyC 编译 TransPyV（自举的"第一阶段"）
|
||
cd D:\Users\TermiNexus\Desktop\TransPyC
|
||
python Projectrans.py --project TransPyV\project.json --clean
|
||
|
||
# 2. 产物在 TransPyV\output\TransPyV.exe
|
||
.\TransPyV\output\TransPyV.exe --help
|
||
```
|
||
|
||
TransPyV CLI 参数（与 TransPyC `Projectrans.py` 一致）：
|
||
|
||
| 参数 | 含义 |
|
||
|------|------|
|
||
| `--project <path>` | `project.json` 路径（默认查找当前目录） |
|
||
| `--src <dir>` | 源文件目录（覆盖 `project.json`） |
|
||
| `--temp <dir>` | 声明接口临时目录（覆盖 `project.json`） |
|
||
| `--output <dir>` | 输出目录（覆盖 `project.json`） |
|
||
| `--phase 1\|2\|all` | 阶段：1=生成声明，2=翻译+编译，all=全部 |
|
||
| `--cc <cmd>` | LLVM 编译器命令（覆盖 `project.json`） |
|
||
| `--clean` | 清理 `output/` 和 `temp/` |
|
||
| `--run` | 编译成功后立即执行生成的可执行文件 |
|
||
| `--rebuild-includes` | 删除 `includes.binary` 预编译缓存 |
|
||
| `--clear-cache` | 清除 `.transpyc_cache` 全局缓存 |
|
||
|
||
### 架构
|
||
|
||
TransPyV 与 TransPyC 共享两阶段编译模型，但实现细节不同：
|
||
|
||
```
|
||
源文件 (.py)
|
||
│
|
||
├─ 阶段一：AST 解析 → 翻译 → stub 分离
|
||
│ Phase1.py / Handles/* → .stub.ll + .text.ll
|
||
│
|
||
├─ 阶段二：stub 合并 → 完整 IR → 编译
|
||
│ StubMerger.BuildCombinedIR
|
||
│ (本地 stub + 依赖 stubs + 本地 text)
|
||
│ BuildPipeline.run_pipeline
|
||
│ (.ll → llc → .obj → clang++ → .exe)
|
||
│
|
||
└─ 输出可执行文件
|
||
```
|
||
|
||
**核心模块分工：**
|
||
|
||
| 模块 | 职责 |
|
||
|------|------|
|
||
| `App/main.py` | CLI 入口：参数解析、内存池初始化、Phase 调度 |
|
||
| `lib/Projectrans/Config.py` | `project.json` / `project.vpj` 加载与路径解析 |
|
||
| `lib/Projectrans/Utils.py` | SHA1 计算、目录清理、杂项工具 |
|
||
| `lib/core/Phase1.py` | 阶段一入口：扫描 includes、生成 stub |
|
||
| `lib/core/Phase2.py` | 阶段二入口：多文件项目翻译 |
|
||
| `lib/core/IncludesScanner.py` | 扫描 `includes/` 目录、决定哪些文件需要重编译 |
|
||
| `lib/core/StubMerger.py` | 把多个 `.stub.ll` 合并为完整 IR |
|
||
| `lib/core/BuildPipeline.py` | `.ll` → `.obj` → `.exe`（llc + clang++） |
|
||
| `lib/core/VLogger.py` | 日志系统（info/warn/error/banner/success） |
|
||
| `lib/core/Handles/HandlesTranslator.py` | `Translator` 类：状态管理 + `translate()` 入口 |
|
||
| `lib/core/Handles/HandlesBody.py` | AST Body 分派（按节点类型路由到具体 Handler） |
|
||
| `lib/core/Handles/Handles*.py` (22 个) | 每种 AST 节点的 LLVM IR 生成 |
|
||
|
||
### 内存模型
|
||
|
||
TransPyV 用 `memhub.MemBuddy` 作为全局内存池：
|
||
|
||
- **POOL_SIZE = 1 GiB**（`1073741824`）
|
||
- 注释（`App/main.py` 第 33 行）说明：Phase1 翻译 87 个 includes + Phase2 翻译 30 个测试文件时，512 MiB 不足
|
||
- `_mbuddy` 通过 `sys._mbuddy = mb` 之类的全局指针传递（参见 `App/main.py` 第 61-75 行）
|
||
|
||
### 与 TransPyC 的关系
|
||
|
||
| 维度 | TransPyC | TransPyV |
|
||
|------|----------|----------|
|
||
| 实现语言 | CPython | Viper |
|
||
| 自举状态 | ❌ 依赖 Python 运行时 | ✅ 目标完全自举 |
|
||
| 入口 | `Projectrans.py` | `App/main.py` |
|
||
| AST 解析 | `ast` 标准库 | `ast`（includes/ast 自研） |
|
||
| 当前状态 | 主路径，跑通所有测试 | 自举重写中 |
|
||
| 平台 | Windows / Linux | Windows（依赖 w32.*） |
|
||
|
||
当前 TransPyC 是"主路径"（所有 Test/ 在用），TransPyV 是"目标"——一旦 TransPyV 能编译自身产出可执行文件，Viper 就完成了完全自举。
|
||
|
||
### 进一步阅读
|
||
|
||
- [TransPyC 编译器仓库](https://git.gvsds.com/GVSDS/TransPyC)
|
||
- [Viper 语言概览](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/01-overview.md)
|
||
- [Viper 自举路线图](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md)
|
||
- [ViperOS 操作系统（用 Viper 写的 OS）](https://git.gvsds.com/GVSDS/ViperOS)
|
||
|
||
---
|
||
|
||
<a id="english"></a>
|
||
|
||
## English
|
||
|
||
### What is this
|
||
|
||
`TransPyV` is the **Viper compiler, written in Viper itself** — the bootstrapping target. TransPyC is the first generation (implemented in CPython); TransPyV is the second generation (implemented in Viper, linked with the LLVM toolchain).
|
||
|
||
While TransPyV currently only runs on Windows (its `App/main.py` directly imports `w32.*`), its goal is to **escape the Python runtime**: once TransPyV can compile itself end-to-end and produce a self-contained executable, Viper will be fully self-hosted.
|
||
|
||
### Directory layout
|
||
|
||
```
|
||
TransPyV/
|
||
├── .gitignore
|
||
├── project.json # top-level project config (outputs TransPyV.exe)
|
||
├── App/ # compiler source (Viper)
|
||
│ ├── main.py # CLI entry
|
||
│ └── lib/
|
||
│ ├── Projectrans/
|
||
│ │ ├── Config.py # project.vpj / project.json loader
|
||
│ │ └── Utils.py
|
||
│ └── core/
|
||
│ ├── BuildPipeline.py # .ll -> .obj -> .exe
|
||
│ ├── IncludesScanner.py # includes directory scanner
|
||
│ ├── Phase1.py # stub split (generates .stub.ll + .text.ll)
|
||
│ ├── Phase2.py # multi-file project translation
|
||
│ ├── StubMerger.py # stub merger
|
||
│ ├── VLogger.py # logging
|
||
│ └── Handles/ # AST node translators (22 files)
|
||
│ ├── HandlesBase.py
|
||
│ ├── HandlesBody.py
|
||
│ ├── HandlesMain.py
|
||
│ ├── HandlesTranslator.py
|
||
│ ├── HandlesVar.py / Assign / AnnAssign / AugAssign
|
||
│ ├── HandlesIf / While / For
|
||
│ ├── HandlesExpr / ExprCall / ExprOps
|
||
│ ├── HandlesFunctions
|
||
│ ├── HandlesClassDef / Struct
|
||
│ ├── HandlesEnum
|
||
│ ├── HandlesImports
|
||
│ ├── HandlesReturn
|
||
│ ├── HandlesType
|
||
│ └── HandlesNonlocal
|
||
└── Test/ # compiler self-test cases
|
||
├── project.vpj
|
||
├── App/ # 33 test .py
|
||
│ ├── simple_test.py
|
||
│ ├── func_test.py / func_vtable_test.py
|
||
│ ├── oop_test.py / vtable_test.py / virtual_dispatch_test.py
|
||
│ ├── struct_test.py
|
||
│ ├── closure_test.py / namespace_test.py
|
||
│ ├── string_test.py / string_min_test.py
|
||
│ ├── flow_test.py / for_test.py
|
||
│ ├── llvmir_test.py
|
||
│ └── ... (33 in total)
|
||
├── NegativeTest/ # negative tests (error cases)
|
||
└── Sha1Test/ # SHA1 namespace tests
|
||
```
|
||
|
||
### Prerequisites
|
||
|
||
| Tool | Notes |
|
||
|------|-------|
|
||
| `python` ≥ 3.10 | only needed when compiling TransPyV with TransPyC |
|
||
| `llc` | LLVM static compiler |
|
||
| `clang++` | mingw-w64 toolchain linker |
|
||
| Windows 10/11 x64 | TransPyV currently Windows-only (imports `w32.*`) |
|
||
| ≥ 1 GiB free RAM | TransPyV allocates `POOL_SIZE=1<<30` at startup |
|
||
| [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) | needed to compile TransPyV itself into an `.exe` |
|
||
|
||
Required layout (to compile TransPyV):
|
||
|
||
```
|
||
D:\Users\TermiNexus\Desktop\TransPyC\
|
||
├── TransPyC\ # compiler repo
|
||
└── TransPyV\ # this repo
|
||
```
|
||
|
||
### Quick start
|
||
|
||
```powershell
|
||
# 1. Use TransPyC to compile TransPyV (the "first stage" of bootstrapping)
|
||
cd D:\Users\TermiNexus\Desktop\TransPyC
|
||
python Projectrans.py --project TransPyV\project.json --clean
|
||
|
||
# 2. Output is at TransPyV\output\TransPyV.exe
|
||
.\TransPyV\output\TransPyV.exe --help
|
||
```
|
||
|
||
TransPyV CLI flags (mirrors `Projectrans.py`):
|
||
|
||
| Flag | Meaning |
|
||
|------|---------|
|
||
| `--project <path>` | `project.json` path (default: search current dir) |
|
||
| `--src <dir>` | source dir (overrides `project.json`) |
|
||
| `--temp <dir>` | declaration temp dir (overrides `project.json`) |
|
||
| `--output <dir>` | output dir (overrides `project.json`) |
|
||
| `--phase 1\|2\|all` | phase: 1=decl only, 2=translate+compile, all=both |
|
||
| `--cc <cmd>` | LLVM compiler command (overrides `project.json`) |
|
||
| `--clean` | clean `output/` and `temp/` |
|
||
| `--run` | execute the produced executable on success |
|
||
| `--rebuild-includes` | delete `includes.binary` precompiled cache |
|
||
| `--clear-cache` | clear `.transpyc_cache` global cache |
|
||
|
||
### Architecture
|
||
|
||
TransPyV shares the two-phase model with TransPyC, with these differences:
|
||
|
||
```
|
||
Source files (.py)
|
||
│
|
||
├─ Phase 1: AST parse -> translate -> stub split
|
||
│ Phase1.py / Handles/* -> .stub.ll + .text.ll
|
||
│
|
||
├─ Phase 2: stub merge -> full IR -> compile
|
||
│ StubMerger.BuildCombinedIR
|
||
│ (local stub + dep stubs + local text)
|
||
│ BuildPipeline.run_pipeline
|
||
│ (.ll -> llc -> .obj -> clang++ -> .exe)
|
||
│
|
||
└─ Native executable
|
||
```
|
||
|
||
**Module responsibilities:**
|
||
|
||
| Module | Responsibility |
|
||
|--------|---------------|
|
||
| `App/main.py` | CLI entry: arg parse, memory pool init, phase dispatch |
|
||
| `lib/Projectrans/Config.py` | `project.json` / `project.vpj` loader + path resolution |
|
||
| `lib/Projectrans/Utils.py` | SHA1, dir cleanup, misc utilities |
|
||
| `lib/core/Phase1.py` | Phase 1 entry: scan includes, generate stubs |
|
||
| `lib/core/Phase2.py` | Phase 2 entry: multi-file project translation |
|
||
| `lib/core/IncludesScanner.py` | scan `includes/`, decide what to rebuild |
|
||
| `lib/core/StubMerger.py` | merge multiple `.stub.ll` into a single IR |
|
||
| `lib/core/BuildPipeline.py` | `.ll` -> `.obj` -> `.exe` (llc + clang++) |
|
||
| `lib/core/VLogger.py` | logging (info/warn/error/banner/success) |
|
||
| `lib/core/Handles/HandlesTranslator.py` | `Translator` class: state + `translate()` |
|
||
| `lib/core/Handles/HandlesBody.py` | AST body dispatcher (route by node type) |
|
||
| `lib/core/Handles/Handles*.py` (22) | one LLVM-IR generator per AST node type |
|
||
|
||
### Memory model
|
||
|
||
TransPyV uses `memhub.MemBuddy` as a global arena:
|
||
|
||
- **POOL_SIZE = 1 GiB** (`1073741824`)
|
||
- The comment on `App/main.py` line 33 explains: Phase1 translates 87 includes files + Phase2 translates 30 test files, 512 MiB is not enough
|
||
- `_mbuddy` is propagated via globals like `sys._mbuddy = mb` (see `App/main.py` lines 61–75)
|
||
|
||
### Relation to TransPyC
|
||
|
||
| Dimension | TransPyC | TransPyV |
|
||
|-----------|----------|----------|
|
||
| Implementation | CPython | Viper |
|
||
| Self-hosted? | No (Python runtime required) | Target: yes |
|
||
| Entry | `Projectrans.py` | `App/main.py` |
|
||
| AST parsing | standard `ast` | `ast` (self-implemented in `includes/ast`) |
|
||
| Current status | Main path, all tests pass | Bootstrapping rewrite in progress |
|
||
| Platform | Windows / Linux | Windows only (depends on `w32.*`) |
|
||
|
||
Currently TransPyC is the main path (all `Test/` use it); TransPyV is the target. Once TransPyV can compile itself into an executable, Viper will be fully self-hosted.
|
||
|
||
### Further reading
|
||
|
||
- [TransPyC compiler repo](https://git.gvsds.com/GVSDS/TransPyC)
|
||
- [Viper language overview](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/01-overview.md)
|
||
- [Viper bootstrapping roadmap](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md)
|
||
- [ViperOS (an OS written in Viper)](https://git.gvsds.com/GVSDS/ViperOS)
|