226 lines
13 KiB
Markdown
226 lines
13 KiB
Markdown
This is a request to remove the Chinese portions from the ViperC README. Here is the resulting document with all Chinese sections stripped, keeping only the English content.
|
|
|
|
---
|
|
|
|
# ViperC
|
|
|
|
The **Viper language compiler, written in Viper itself** — and it is **self-hosted**: ViperC compiles its own source and produces `viperc.exe`. It translates Viper sources to LLVM IR, then drives `llc` + `clang++` to link a native Windows executable. The first-generation CPython compiler ([TransPyC](https://git.gvsds.com/GVSDS/TransPyC)) is now only kept as the bootstrapping **seed**.
|
|
|
|
---
|
|
|
|
<a id="english"></a>
|
|
|
|
## English
|
|
|
|
### What is this
|
|
|
|
`ViperC` is the second-generation Viper compiler: **written in Viper and compiled by Viper**. No Python runtime takes part in the compilation path — lexing/parsing goes through `includes/ast` (a self-implemented parser, 9 files), IR generation through `App/lib/core/Gen/`, and machine code through `llc` + `clang++`.
|
|
|
|
The bootstrap chain:
|
|
|
|
```text
|
|
TransPyC (seed, implemented in CPython)
|
|
│ one-time only: compile ViperC sources into viperc.exe
|
|
▼
|
|
viperc.exe (stage-one artifact)
|
|
│ bootstrap: this exe compiles ViperC itself
|
|
▼
|
|
viperc.exe (self-hosted artifact — the one shipped in exe\)
|
|
│
|
|
└─ used to build any Viper project (ViperApps\*, Test\* …)
|
|
```
|
|
|
|
Current state and size (measured on the dev machine):
|
|
|
|
| Item | Value |
|
|
| --- | --- |
|
|
| Compiler sources | **82 `.vp` files, 63,783 lines** under `App/` |
|
|
| Translator core | `App/lib/core/Gen/` — **54 files, 47,297 lines** |
|
|
| Phases & merging | `App/lib/Phase/` 6 files 5,339 lines; `App/lib/Stub/` 9 files 6,207 lines |
|
|
| Project & utils | `App/lib/Projectrans/` 3 files 712 lines |
|
|
| Self-tests | workspace root `Test\` (**separate repo**): **97 projects, 129 `.vp`**, driven by `Test\SequentialRegressionTest.py` |
|
|
| Platform | Windows x64 (sources `import win32.*` directly) |
|
|
|
|
### Directory layout
|
|
|
|
```text
|
|
ViperC/
|
|
├── project.vpj # project config (source_dir=./App, build_dir=./__vpcbuild__, output viperc.exe)
|
|
├── App/ # compiler sources (Viper)
|
|
│ ├── main.vp # CLI entry: pools, singleton injection, arg parsing, phase dispatch
|
|
│ └── lib/
|
|
│ ├── Projectrans/
|
|
│ │ ├── Config.vp # project.vpj loader + path resolution (incl. includes cache dir)
|
|
│ │ └── Utils.vp # SHA1 / dir cleanup / misc
|
|
│ ├── core/
|
|
│ │ ├── VLogger.vp # logging (info/warn/error/banner/success)
|
|
│ │ ├── IncludesScanner.vp # scan includes/, decide what must be rebuilt
|
|
│ │ ├── CompilerFactory.vp # holds the two MemBuddy singletons (mb + perm_pool)
|
|
│ │ └── Gen/ # translator (52 files) — AST to LLVM IR
|
|
│ │ ├── Generator.vp / Translator.vp / GenApi.vp / GenTypes.vp
|
|
│ │ ├── BaseGenMixin / AssignGenMixin / VarGenMixin / ImportsGenMixin
|
|
│ │ ├── MatchGenMixin / AsyncGenMixin / StructGenMixin / TranslatorGenMixin
|
|
│ │ ├── Call/ (12) calls: plain / C builtin / ctor / vtable / varargs / inline asm …
|
|
│ │ ├── Class/ (6) classes: def / generic / vtable / enum / mixin
|
|
│ │ ├── Expr/ (9) expressions: name / const / ops / attr / f-string / CDef …
|
|
│ │ ├── Func/ (3) function definitions
|
|
│ │ ├── Stmt/ (3) statements: decl / C-if / mixin
|
|
│ │ └── Type/ (6) type resolution and C type mapping
|
|
│ ├── Phase/
|
|
│ │ ├── BuildPipeline.vp # .ll -> llc -> .obj -> clang++ -> .exe (plus ASan branch)
|
|
│ │ ├── Phase1/ # Phase 1: on-demand includes translation + stub split (.stub.ll / .text.ll)
|
|
│ │ └── Phase2/ # Phase 2: multi-file translation, IR assembly, compile
|
|
│ └── Stub/
|
|
│ ├── StubGen/ # .vpi reader + declaration converter
|
|
│ └── StubMerger/ # dependency-ordered merge of .stub.ll / .text.ll
|
|
```
|
|
|
|
> The self-test/regression suite is not in this repo: it lives at the workspace root `Test\`
|
|
> (a separate repo, 97 projects, 129 `.vp`), driven by `Test\SequentialRegressionTest.py`
|
|
> (`--exe exe\viperc.exe`).
|
|
|
|
### Prerequisites
|
|
|
|
| Tool | Notes |
|
|
| --- | --- |
|
|
| Windows 10/11 x64 | sources `import win32.*`; Windows-only for now |
|
|
| `llc` | LLVM static compiler (22.1.8 here), set by `compiler.cmd` in `project.vpj` |
|
|
| `clang++` | mingw-w64 toolchain, `-fuse-ld=lld` (here `D:/msys64/mingw64/bin/clang++.exe`) |
|
|
| ≥ 1.5 GiB free RAM | allocates a 1 GiB main pool + 512 MiB permanent pool at startup |
|
|
| [TransPyC](https://git.gvsds.com/GVSDS/TransPyC) | **only needed to rebuild the bootstrap seed** |
|
|
|
|
Required layout (`project.vpj` refers to `../includes`):
|
|
|
|
```text
|
|
TransPyC\ # workspace root
|
|
├── TransPyC\ # first-generation compiler (seed)
|
|
├── ViperC\ # this repo
|
|
├── Test\ # regression suite (separate repo)
|
|
├── exe\ # shipped self-hosted artifact (viperc.exe)
|
|
└── includes\ # Viper standard library + includes/ast parser
|
|
```
|
|
|
|
### Quick start
|
|
|
|
```bat
|
|
:: everyday use: build any project with the self-hosted compiler
|
|
exe\viperc.exe --project ViperApps\ViperMNIST\project.vpj --clean
|
|
|
|
:: build and run immediately
|
|
exe\viperc.exe --project ViperApps\ViperMNIST\project.vpj --clean --run
|
|
|
|
:: only when rebuilding the compiler itself (bootstrap seed), use TransPyC
|
|
python TransPyC\Projectrans.py --project ViperC\project.vpj --clean
|
|
```
|
|
|
|
Artifacts land in `<project dir>\<build_dir>\output\<linker.output>` — by default
|
|
`<project>\__vpcbuild__\output\app.exe`. ViperC itself sets `linker.output` to `viperc.exe`,
|
|
so its own product is `ViperC\__vpcbuild__\output\viperc.exe`.
|
|
|
|
Run the regression suite (workspace root `Test\`, a separate repo):
|
|
|
|
```bat
|
|
python Test\SequentialRegressionTest.py --exe exe\viperc.exe
|
|
```
|
|
|
|
### CLI flags
|
|
|
|
| Flag | Meaning |
|
|
| --- | --- |
|
|
| `--project <path>` | `project.vpj` path (default: current dir, then parent) |
|
|
| `--src <dir>` | single-file mode source dir (overrides config; entry `<dir>/main.vp`) |
|
|
| `--temp <dir>` | declaration temp dir (default `<build_dir>/temp`) |
|
|
| `--output <dir>` | output dir (default `<build_dir>/output`) |
|
|
| `--phase 1\|2\|all` | 1 = stub split only; 2 = IR assembly + compile only; all = both (default) |
|
|
| `--cc <cmd>` | LLVM compiler command (overrides config) |
|
|
| `--build-dir <dir>` | override `build_dir` from `project.vpj` (relative paths still resolve against the project dir) |
|
|
| `--clean` | clean temp / output / includes precompiled cache |
|
|
| `--rebuild-includes` | delete only the includes cache (`includes.binary`), forcing a full includes rebuild |
|
|
| `--clear-cache` | clear only temp (`.vpi` / `.stub.ll` / `.text.ll`), forcing stub regeneration |
|
|
| `--run` | execute the produced binary on success |
|
|
| `--repl` | interactive REPL (M1: expression fragment → DLL → call and echo); without `--project` it looks for the `repl.vpj` template project in cwd, then next to `viperc.exe` (errors out with a `--project` hint if neither exists); mutually exclusive with `--asan` / `--phase2command` / `--spend-file` |
|
|
| `--asan` | ASan mode: `opt -passes=asan` first; runtimes must be declared in `project.vpj` `linker.static_libs`, else fail-fast |
|
|
| `--spend-file <path>` | vprofile report path for the built program (relative to the output exe dir, `../x.txt` and absolute paths allowed); enables function instrumentation and force-links `includes/vprofile.vp`. Mutually exclusive with `--repl` |
|
|
| `--vpf <0-3>` | vprofile granularity: 0=off, 1=call counts, 2=+per-function time, 3=+caller/callee edges (default with `--spend-file`) |
|
|
| `--build-self` | rebuild the compiler into `ViperC\__vpcbuild_2__` and let `copyself` replace `ViperC\__vpcbuild__`; every other argument is forwarded verbatim except `--project` / `--build-dir` (the self-build owns those) |
|
|
| `--phaseb-legacy` | use the legacy `mlir-opt` + `mlir-translate` + `llc` child-process PhaseB instead of the in-process `ircore.dll` backend |
|
|
| `--phase2command <bat>` | do not run `llc`/link; write the commands into the given `.bat` |
|
|
| `--nowarn-register` | suppress Phase A dependency WARNs (cross-module `text.ll` not yet emitted, converges next pass) |
|
|
| `--no-llvmpeephole` | disable FunctionPrint local peepholes (T2 store/load forwarding, T3 no-op bitcast) |
|
|
| `--detection` | enable MemBuddy allocation detection (`0xCDCD` fill + record-table probe for out-of-bounds writes) |
|
|
| `--show-args` | dump all parsed arguments |
|
|
| `--version`, `-v` | print Viper / VRTE / ViperC versions, commit ids, target triple and LLVM version |
|
|
| `--help`, `-h` | print help |
|
|
|
|
### Architecture
|
|
|
|
```text
|
|
all .vp/.py under source_dir (multi-file mode: recursively scanned)
|
|
│
|
|
├─ Phase 1 (stub split)
|
|
│ IncludesScanner translates includes on demand -> Phase1/*
|
|
│ Translator + Gen/* emit IR, then write two files:
|
|
│ <sha1>.stub.ll declarations
|
|
│ <sha1>.text.ll implementations
|
|
│
|
|
├─ Phase 2 (assemble + compile)
|
|
│ StubMerger* topologically merge local stub + dependency stubs + local text -> full IR
|
|
│ BuildPipeline: full IR -> llc -> .obj -> clang++ (+ precompiled includes .obj) -> .exe
|
|
│
|
|
└─ <build_dir>/output/<linker.output>
|
|
```
|
|
|
|
**Module responsibilities:**
|
|
|
|
| Module | Responsibility |
|
|
| --- | --- |
|
|
| `App/main.vp` | CLI entry: pools and singletons, arg parsing, phase dispatch, `--run` |
|
|
| `lib/Projectrans/Config.vp` | `project.vpj` loading, path resolution, includes cache dir |
|
|
| `lib/core/CompilerFactory.vp` | owns `mb` (main pool) and `perm_pool` (permanent pool) |
|
|
| `lib/core/Gen/Generator.vp` | project-level `Generator` singleton shared by Phase1/Phase2/BuildPipeline |
|
|
| `lib/core/Gen/Translator.vp` | translator body: `translate()` and `dump_ir()` (`.stub.ll` / `.text.ll` / full IR) |
|
|
| `lib/core/Gen/GenApi.vp` | translator public API and global state (e.g. temp dir) |
|
|
| `lib/Phase/Phase1/` | scan includes, topological order, emit declarations on demand |
|
|
| `lib/Phase/Phase2/` | multi-file projects: scan sources, translate, assemble IR, drive the build |
|
|
| `lib/Phase/BuildPipeline.vp` | `.ll` -> `llc` -> `.obj` -> `clang++` -> `.exe` (ASan branch included) |
|
|
| `lib/Stub/StubGen/` | read `.vpi`, convert to stub declarations |
|
|
| `lib/Stub/StubMerger/` | collect / topologically sort / concatenate / store `stub.ll` and `text.ll` |
|
|
| `lib/core/IncludesScanner.vp` | scan `includes/`, decide what needs rebuilding |
|
|
| `lib/core/VLogger.vp` | logging (with the `--nowarn-register` switch) |
|
|
|
|
> Bootstrap note: after self-hosting, module-level globals carry a **SHA1 prefix** (module
|
|
> isolation), so singletons such as `_g_gen` must be assigned **per module**
|
|
> (`GenApi._g_gen = gen` …), not once by bare name in `main.vp` — otherwise some modules read
|
|
> their own `None`, showing up as bogus results like "global-name test always 0".
|
|
|
|
### Memory model
|
|
|
|
There is no per-object system `malloc` churn — only two `MemBuddy` buddy-allocator pools:
|
|
|
|
- `POOL_SIZE = 1 GiB` (`1073741824`) — main pool `mb`, ordinary translation allocations.
|
|
- `PERM_POOL_SIZE = 536871176` (≈512 MiB usable) — permanent pool `perm_pool`, objects that outlive a phase.
|
|
- Both are held by the `CompilerFactory.factory` singleton; `mb` is additionally injected per module
|
|
as `sys._mbuddy` / `argparse._mbuddy` / `ast._mbuddy` / `Config._mbuddy` …
|
|
|
|
`--detection` switches both pools into detection mode (`0xCDCD` fill + record-table probe) to hunt
|
|
out-of-bounds writes.
|
|
|
|
### Relation to TransPyC
|
|
|
|
| Dimension | TransPyC | ViperC |
|
|
| --- | --- | --- |
|
|
| Implementation | CPython | Viper |
|
|
| Role | bootstrap **seed** (only used to produce a stable ViperC) | **the current compiler** |
|
|
| Self-hosted? | — | ✅ yes, it compiles itself |
|
|
| Entry | `TransPyC/Projectrans.py` | `App/main.vp` |
|
|
| AST parsing | Python `ast` / in-house vast | `includes/ast` (9 files, in-house) |
|
|
| Project config | `project.json` | `project.vpj` |
|
|
| Platform | Windows / Linux | Windows (`win32.*`) |
|
|
|
|
### Further reading
|
|
|
|
- [TransPyC compiler repo (the seed)](https://git.gvsds.com/GVSDS/TransPyC)
|
|
- [Viper language docs (wiki)](https://git.gvsds.com/GVSDS/TransPyC/tree/main/wiki)
|
|
- [Bootstrapping analysis and progress](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/13-bootstrapping.md)
|
|
- [Compilation system (`project.vpj` / `viperbuild` metaprogramming)](https://git.gvsds.com/GVSDS/TransPyC/blob/main/wiki/12-compilation-system.md)
|
|
- [ViperOS (an OS written in Viper)](https://git.gvsds.com/GVSDS/ViperOS) |