From 3cf67e262f84a26d7dc2a7347546def7421268d5 Mon Sep 17 00:00:00 2001 From: Viper Date: Sun, 19 Jul 2026 13:33:50 +0800 Subject: [PATCH] Add bilingual README --- README.md | 349 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 349 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..4b6bbf8 --- /dev/null +++ b/README.md @@ -0,0 +1,349 @@ +# TransPyV + +[English](#english) | [中文](#中文) + +The **Viper language compiler, written in Viper itself**. TransPyV is the self-hosted (bootstrapped) successor of [TransPyC](https://git.gvsds.com/GVSDS/TransPyC). It compiles Viper sources to LLVM IR, then drives `llc` + `clang++` to produce a native Windows executable (`TransPyV.exe`). + +**用 Viper 语言编写的 Viper 编译器**。TransPyV 是 [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) 的自举（self-hosted）后继版本。它把 Viper 源码编译为 LLVM IR，再调用 `llc` + `clang++` 产出 Windows 原生可执行文件（`TransPyV.exe`）。 + +--- + + + +## 中文 + +### 这是什么 + +`TransPyV` 是 Viper 项目**自举（bootstrapping）目标**：用 Viper 自身重写编译器。TransPyC 是第一代（用 CPython 写的），TransPyV 是第二代（用 Viper 写，编译后用 LLVM 工具链链接）。 + +虽然 TransPyV 当前只在 Windows 上能跑通（`App/main.py` 直接 import `w32.*`），但它的目标是**脱离 Python 运行时**：将来一旦 TransPyV 能完整编译自身并产出可执行文件，就完成了 Viper 编译器的完全自举。 + +### 目录结构 + +``` +TransPyV/ +├── .gitignore +├── project.json # 顶层项目配置（输出 TransPyV.exe） +├── App/ # 编译器源码（Viper） +│ ├── main.py # CLI 入口 +│ └── lib/ +│ ├── Projectrans/ +│ │ ├── Config.py # project.vpj / project.json 加载 +│ │ └── Utils.py +│ └── core/ +│ ├── BuildPipeline.py # .ll → .obj → .exe +│ ├── IncludesScanner.py # includes 目录扫描 +│ ├── Phase1.py # stub 分离（生成 .stub.ll + .text.ll） +│ ├── Phase2.py # 多文件项目翻译 +│ ├── StubMerger.py # stub 合并 +│ ├── VLogger.py # 日志系统 +│ └── Handles/ # AST 节点翻译（22 个） +│ ├── HandlesBase.py +│ ├── HandlesBody.py +│ ├── HandlesMain.py +│ ├── HandlesTranslator.py +│ ├── HandlesVar.py / Assign / AnnAssign / AugAssign +│ ├── HandlesIf / While / For +│ ├── HandlesExpr / ExprCall / ExprOps +│ ├── HandlesFunctions +│ ├── HandlesClassDef / Struct +│ ├── HandlesEnum +│ ├── HandlesImports +│ ├── HandlesReturn +│ ├── HandlesType +│ └── HandlesNonlocal +└── Test/ # 编译器的自测用例 + ├── project.vpj + ├── App/ # 33 个测试 .py + │ ├── simple_test.py + │ ├── func_test.py / func_vtable_test.py + │ ├── oop_test.py / vtable_test.py / virtual_dispatch_test.py + │ ├── struct_test.py / class_test + │ ├── closure_test.py / namespace_test.py + │ ├── string_test.py / string_min_test.py + │ ├── flow_test.py / for_test.py + │ ├── llvmir_test.py + │ └── ... (33 个) + ├── NegativeTest/ # 错误用法负向测试 + └── Sha1Test/ # SHA1 命名空间测试 +``` + +### 前提条件 + +| 工具 | 说明 | +|------|------| +| `python` ≥ 3.10 | **仅当用 TransPyC 编译 TransPyV 时**需要 | +| `llc` | LLVM 静态编译器 | +| `clang++` | mingw-w64 工具链链接器 | +| Windows 10/11 x64 | TransPyV 当前仅在 Windows 上工作（import w32.*） | +| 至少 1GB 可用内存 | TransPyV 启动时 `stdlib.malloc(POOL_SIZE=1<<30)` | +| [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) | 用于把 TransPyV 自身编译为 `.exe` | + +目录布局要求（编译 TransPyV 时）： + +``` +D:\Users\TermiNexus\Desktop\TransPyC\ +├── TransPyC\ # 编译器仓库 +└── TransPyV\ # 本仓库 +``` + +### 快速开始 + +```powershell +# 1. 用 TransPyC 编译 TransPyV（自举的"第一阶段"） +cd D:\Users\TermiNexus\Desktop\TransPyC +python Projectrans.py --project TransPyV\project.json --clean + +# 2. 产物在 TransPyV\output\TransPyV.exe +.\TransPyV\output\TransPyV.exe --help +``` + +TransPyV CLI 参数（与 TransPyC `Projectrans.py` 一致）： + +| 参数 | 含义 | +|------|------| +| `--project ` | `project.json` 路径（默认查找当前目录） | +| `--src ` | 源文件目录（覆盖 `project.json`） | +| `--temp ` | 声明接口临时目录（覆盖 `project.json`） | +| `--output ` | 输出目录（覆盖 `project.json`） | +| `--phase 1\|2\|all` | 阶段：1=生成声明，2=翻译+编译，all=全部 | +| `--cc ` | LLVM 编译器命令（覆盖 `project.json`） | +| `--clean` | 清理 `output/` 和 `temp/` | +| `--run` | 编译成功后立即执行生成的可执行文件 | +| `--rebuild-includes` | 删除 `includes.binary` 预编译缓存 | +| `--clear-cache` | 清除 `.transpyc_cache` 全局缓存 | + +### 架构 + +TransPyV 与 TransPyC 共享两阶段编译模型，但实现细节不同： + +``` +源文件 (.py) + │ + ├─ 阶段一：AST 解析 → 翻译 → stub 分离 + │ Phase1.py / Handles/* → .stub.ll + .text.ll + │ + ├─ 阶段二：stub 合并 → 完整 IR → 编译 + │ StubMerger.BuildCombinedIR + │ (本地 stub + 依赖 stubs + 本地 text) + │ BuildPipeline.run_pipeline + │ (.ll → llc → .obj → clang++ → .exe) + │ + └─ 输出可执行文件 +``` + +**核心模块分工：** + +| 模块 | 职责 | +|------|------| +| `App/main.py` | CLI 入口：参数解析、内存池初始化、Phase 调度 | +| `lib/Projectrans/Config.py` | `project.json` / `project.vpj` 加载与路径解析 | +| `lib/Projectrans/Utils.py` | SHA1 计算、目录清理、杂项工具 | +| `lib/core/Phase1.py` | 阶段一入口：扫描 includes、生成 stub | +| `lib/core/Phase2.py` | 阶段二入口：多文件项目翻译 | +| `lib/core/IncludesScanner.py` | 扫描 `includes/` 目录、决定哪些文件需要重编译 | +| `lib/core/StubMerger.py` | 把多个 `.stub.ll` 合并为完整 IR | +| `lib/core/BuildPipeline.py` | `.ll` → `.obj` → `.exe`（llc + clang++） | +| `lib/core/VLogger.py` | 日志系统（info/warn/error/banner/success） | +| `lib/core/Handles/HandlesTranslator.py` | `Translator` 类：状态管理 + `translate()` 入口 | +| `lib/core/Handles/HandlesBody.py` | AST Body 分派（按节点类型路由到具体 Handler） | +| `lib/core/Handles/Handles*.py` (22 个) | 每种 AST 节点的 LLVM IR 生成 | + +### 内存模型 + +TransPyV 用 `memhub.MemBuddy` 作为全局内存池： + +- **POOL_SIZE = 1 GiB**（`1073741824`） +- 注释（`App/main.py` 第 33 行）说明：Phase1 翻译 87 个 includes + Phase2 翻译 30 个测试文件时，512 MiB 不足 +- `_mbuddy` 通过 `sys._mbuddy = mb` 之类的全局指针传递（参见 `App/main.py` 第 61-75 行） + +### 与 TransPyC 的关系 + +| 维度 | TransPyC | TransPyV | +|------|----------|----------| +| 实现语言 | CPython | Viper | +| 自举状态 | ❌ 依赖 Python 运行时 | ✅ 目标完全自举 | +| 入口 | `Projectrans.py` | `App/main.py` | +| AST 解析 | `ast` 标准库 | `ast`（includes/ast 自研） | +| 当前状态 | 主路径，跑通所有测试 | 自举重写中 | +| 平台 | Windows / Linux | Windows（依赖 w32.*） | + +当前 TransPyC 是"主路径"（所有 Test/ 在用），TransPyV 是"目标"——一旦 TransPyV 能编译自身产出可执行文件，Viper 就完成了完全自举。 + +### 进一步阅读 + +- [TransPyC 编译器仓库](https://git.gvsds.com/GVSDS/TransPyC) +- [Viper 语言概览](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/01-overview.md) +- [Viper 自举路线图](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md) +- [ViperOS 操作系统（用 Viper 写的 OS）](https://git.gvsds.com/GVSDS/ViperOS) + +--- + + + +## English + +### What is this + +`TransPyV` is the **Viper compiler, written in Viper itself** — the bootstrapping target. TransPyC is the first generation (implemented in CPython); TransPyV is the second generation (implemented in Viper, linked with the LLVM toolchain). + +While TransPyV currently only runs on Windows (its `App/main.py` directly imports `w32.*`), its goal is to **escape the Python runtime**: once TransPyV can compile itself end-to-end and produce a self-contained executable, Viper will be fully self-hosted. + +### Directory layout + +``` +TransPyV/ +├── .gitignore +├── project.json # top-level project config (outputs TransPyV.exe) +├── App/ # compiler source (Viper) +│ ├── main.py # CLI entry +│ └── lib/ +│ ├── Projectrans/ +│ │ ├── Config.py # project.vpj / project.json loader +│ │ └── Utils.py +│ └── core/ +│ ├── BuildPipeline.py # .ll -> .obj -> .exe +│ ├── IncludesScanner.py # includes directory scanner +│ ├── Phase1.py # stub split (generates .stub.ll + .text.ll) +│ ├── Phase2.py # multi-file project translation +│ ├── StubMerger.py # stub merger +│ ├── VLogger.py # logging +│ └── Handles/ # AST node translators (22 files) +│ ├── HandlesBase.py +│ ├── HandlesBody.py +│ ├── HandlesMain.py +│ ├── HandlesTranslator.py +│ ├── HandlesVar.py / Assign / AnnAssign / AugAssign +│ ├── HandlesIf / While / For +│ ├── HandlesExpr / ExprCall / ExprOps +│ ├── HandlesFunctions +│ ├── HandlesClassDef / Struct +│ ├── HandlesEnum +│ ├── HandlesImports +│ ├── HandlesReturn +│ ├── HandlesType +│ └── HandlesNonlocal +└── Test/ # compiler self-test cases + ├── project.vpj + ├── App/ # 33 test .py + │ ├── simple_test.py + │ ├── func_test.py / func_vtable_test.py + │ ├── oop_test.py / vtable_test.py / virtual_dispatch_test.py + │ ├── struct_test.py + │ ├── closure_test.py / namespace_test.py + │ ├── string_test.py / string_min_test.py + │ ├── flow_test.py / for_test.py + │ ├── llvmir_test.py + │ └── ... (33 in total) + ├── NegativeTest/ # negative tests (error cases) + └── Sha1Test/ # SHA1 namespace tests +``` + +### Prerequisites + +| Tool | Notes | +|------|-------| +| `python` ≥ 3.10 | only needed when compiling TransPyV with TransPyC | +| `llc` | LLVM static compiler | +| `clang++` | mingw-w64 toolchain linker | +| Windows 10/11 x64 | TransPyV currently Windows-only (imports `w32.*`) | +| ≥ 1 GiB free RAM | TransPyV allocates `POOL_SIZE=1<<30` at startup | +| [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) | needed to compile TransPyV itself into an `.exe` | + +Required layout (to compile TransPyV): + +``` +D:\Users\TermiNexus\Desktop\TransPyC\ +├── TransPyC\ # compiler repo +└── TransPyV\ # this repo +``` + +### Quick start + +```powershell +# 1. Use TransPyC to compile TransPyV (the "first stage" of bootstrapping) +cd D:\Users\TermiNexus\Desktop\TransPyC +python Projectrans.py --project TransPyV\project.json --clean + +# 2. Output is at TransPyV\output\TransPyV.exe +.\TransPyV\output\TransPyV.exe --help +``` + +TransPyV CLI flags (mirrors `Projectrans.py`): + +| Flag | Meaning | +|------|---------| +| `--project ` | `project.json` path (default: search current dir) | +| `--src ` | source dir (overrides `project.json`) | +| `--temp ` | declaration temp dir (overrides `project.json`) | +| `--output ` | output dir (overrides `project.json`) | +| `--phase 1\|2\|all` | phase: 1=decl only, 2=translate+compile, all=both | +| `--cc ` | LLVM compiler command (overrides `project.json`) | +| `--clean` | clean `output/` and `temp/` | +| `--run` | execute the produced executable on success | +| `--rebuild-includes` | delete `includes.binary` precompiled cache | +| `--clear-cache` | clear `.transpyc_cache` global cache | + +### Architecture + +TransPyV shares the two-phase model with TransPyC, with these differences: + +``` +Source files (.py) + │ + ├─ Phase 1: AST parse -> translate -> stub split + │ Phase1.py / Handles/* -> .stub.ll + .text.ll + │ + ├─ Phase 2: stub merge -> full IR -> compile + │ StubMerger.BuildCombinedIR + │ (local stub + dep stubs + local text) + │ BuildPipeline.run_pipeline + │ (.ll -> llc -> .obj -> clang++ -> .exe) + │ + └─ Native executable +``` + +**Module responsibilities:** + +| Module | Responsibility | +|--------|---------------| +| `App/main.py` | CLI entry: arg parse, memory pool init, phase dispatch | +| `lib/Projectrans/Config.py` | `project.json` / `project.vpj` loader + path resolution | +| `lib/Projectrans/Utils.py` | SHA1, dir cleanup, misc utilities | +| `lib/core/Phase1.py` | Phase 1 entry: scan includes, generate stubs | +| `lib/core/Phase2.py` | Phase 2 entry: multi-file project translation | +| `lib/core/IncludesScanner.py` | scan `includes/`, decide what to rebuild | +| `lib/core/StubMerger.py` | merge multiple `.stub.ll` into a single IR | +| `lib/core/BuildPipeline.py` | `.ll` -> `.obj` -> `.exe` (llc + clang++) | +| `lib/core/VLogger.py` | logging (info/warn/error/banner/success) | +| `lib/core/Handles/HandlesTranslator.py` | `Translator` class: state + `translate()` | +| `lib/core/Handles/HandlesBody.py` | AST body dispatcher (route by node type) | +| `lib/core/Handles/Handles*.py` (22) | one LLVM-IR generator per AST node type | + +### Memory model + +TransPyV uses `memhub.MemBuddy` as a global arena: + +- **POOL_SIZE = 1 GiB** (`1073741824`) +- The comment on `App/main.py` line 33 explains: Phase1 translates 87 includes files + Phase2 translates 30 test files, 512 MiB is not enough +- `_mbuddy` is propagated via globals like `sys._mbuddy = mb` (see `App/main.py` lines 61–75) + +### Relation to TransPyC + +| Dimension | TransPyC | TransPyV | +|-----------|----------|----------| +| Implementation | CPython | Viper | +| Self-hosted? | No (Python runtime required) | Target: yes | +| Entry | `Projectrans.py` | `App/main.py` | +| AST parsing | standard `ast` | `ast` (self-implemented in `includes/ast`) | +| Current status | Main path, all tests pass | Bootstrapping rewrite in progress | +| Platform | Windows / Linux | Windows only (depends on `w32.*`) | + +Currently TransPyC is the main path (all `Test/` use it); TransPyV is the target. Once TransPyV can compile itself into an executable, Viper will be fully self-hosted. + +### Further reading + +- [TransPyC compiler repo](https://git.gvsds.com/GVSDS/TransPyC) +- [Viper language overview](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/01-overview.md) +- [Viper bootstrapping roadmap](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md) +- [ViperOS (an OS written in Viper)](https://git.gvsds.com/GVSDS/ViperOS)