GVSDS/ViperC

0 Fork 0

Files

T

2026-10-19 18:25:15 +08:00

12 KiB

Raw Permalink Blame History

This is a request to remove the Chinese portions from the ViperC README. Here is the resulting document with all Chinese sections stripped, keeping only the English content.

ViperC

The Viper language compiler, written in Viper itself — and it is self-hosted: ViperC compiles its own source and produces viperc.exe. It translates Viper sources to LLVM IR, then drives llc + clang++ to link a native Windows executable. The first-generation CPython compiler (TransPyC) is now only kept as the bootstrapping seed.

English

What is this

ViperC is the second-generation Viper compiler: written in Viper and compiled by Viper. No Python runtime takes part in the compilation path — lexing/parsing goes through includes/ast (a self-implemented parser, 9 files), IR generation through App/lib/core/Gen/, and machine code through llc + clang++.

The bootstrap chain:

TransPyC (seed, implemented in CPython)
    │  one-time only: compile ViperC sources into viperc.exe
    ▼
viperc.exe (stage-one artifact)
    │  bootstrap: this exe compiles ViperC itself
    ▼
viperc.exe (self-hosted artifact — the one shipped in exe\)
    │
    └─ used to build any Viper project (ViperApps\*, Test\* …)

Current state and size (measured on the dev machine):

Item	Value
Compiler sources	82 `.vp` files, 63,783 lines under `App/`
Translator core	`App/lib/core/Gen/` — 54 files, 47,297 lines
Phases & merging	`App/lib/Phase/` 6 files 5,339 lines; `App/lib/Stub/` 9 files 6,207 lines
Project & utils	`App/lib/Projectrans/` 3 files 712 lines
Self-tests	workspace root `Test\` (separate repo): 97 projects, 129 `.vp`, driven by `Test\SequentialRegressionTest.py`
Platform	Windows x64 (sources `import win32.*` directly)

Directory layout

ViperC/
├── project.vpj                 # project config (source_dir=./App, build_dir=./__vpcbuild__, output viperc.exe)
├── App/                        # compiler sources (Viper)
│   ├── main.vp                 # CLI entry: pools, singleton injection, arg parsing, phase dispatch
│   └── lib/
│       ├── Projectrans/
│       │   ├── Config.vp        # project.vpj loader + path resolution (incl. includes cache dir)
│       │   └── Utils.vp         # SHA1 / dir cleanup / misc
│       ├── core/
│       │   ├── VLogger.vp       # logging (info/warn/error/banner/success)
│       │   ├── IncludesScanner.vp   # scan includes/, decide what must be rebuilt
│       │   ├── CompilerFactory.vp   # holds the two MemBuddy singletons (mb + perm_pool)
│       │   └── Gen/             # translator (52 files) — AST to LLVM IR
│       │       ├── Generator.vp / Translator.vp / GenApi.vp / GenTypes.vp
│       │       ├── BaseGenMixin / AssignGenMixin / VarGenMixin / ImportsGenMixin
│       │       ├── MatchGenMixin / AsyncGenMixin / StructGenMixin / TranslatorGenMixin
│       │       ├── Call/   (12) calls: plain / C builtin / ctor / vtable / varargs / inline asm …
│       │       ├── Class/  (6)  classes: def / generic / vtable / enum / mixin
│       │       ├── Expr/   (9)  expressions: name / const / ops / attr / f-string / CDef …
│       │       ├── Func/   (3)  function definitions
│       │       ├── Stmt/   (3)  statements: decl / C-if / mixin
│       │       └── Type/   (6)  type resolution and C type mapping
│       ├── Phase/
│       │   ├── BuildPipeline.vp # .ll -> llc -> .obj -> clang++ -> .exe (plus ASan branch)
│       │   ├── Phase1/          # Phase 1: on-demand includes translation + stub split (.stub.ll / .text.ll)
│       │   └── Phase2/          # Phase 2: multi-file translation, IR assembly, compile
│       └── Stub/
│           ├── StubGen/         # .vpi reader + declaration converter
│           └── StubMerger/      # dependency-ordered merge of .stub.ll / .text.ll

The self-test/regression suite is not in this repo: it lives at the workspace root Test\ (a separate repo, 97 projects, 129 .vp), driven by Test\SequentialRegressionTest.py (--exe exe\viperc.exe).

Prerequisites

Tool	Notes
Windows 10/11 x64	sources `import win32.*`; Windows-only for now
`llc`	LLVM static compiler (22.1.8 here), set by `compiler.cmd` in `project.vpj`
`clang++`	mingw-w64 toolchain, `-fuse-ld=lld` (here `D:/msys64/mingw64/bin/clang++.exe`)
≥ 1.5 GiB free RAM	allocates a 1 GiB main pool + 512 MiB permanent pool at startup
TransPyC	only needed to rebuild the bootstrap seed

Required layout (project.vpj refers to ../includes):

TransPyC\                 # workspace root
├── TransPyC\             # first-generation compiler (seed)
├── ViperC\               # this repo
├── Test\                 # regression suite (separate repo)
├── exe\                  # shipped self-hosted artifact (viperc.exe)
└── includes\             # Viper standard library + includes/ast parser

Quick start

:: everyday use: build any project with the self-hosted compiler
exe\viperc.exe --project ViperApps\ViperMNIST\project.vpj --clean

:: build and run immediately
exe\viperc.exe --project ViperApps\ViperMNIST\project.vpj --clean --run

:: only when rebuilding the compiler itself (bootstrap seed), use TransPyC
python TransPyC\Projectrans.py --project ViperC\project.vpj --clean

Artifacts land in <project dir>\<build_dir>\output\<linker.output> — by default <project>\__vpcbuild__\output\app.exe. ViperC itself sets linker.output to viperc.exe, so its own product is ViperC\__vpcbuild__\output\viperc.exe.

Run the regression suite (workspace root Test\, a separate repo):

python Test\SequentialRegressionTest.py --exe exe\viperc.exe

CLI flags

Flag	Meaning
`--project <path>`	`project.vpj` path (default: current dir, then parent)
`--src <dir>`	single-file mode source dir (overrides config; entry `<dir>/main.vp`)
`--temp <dir>`	declaration temp dir (default `<build_dir>/temp`)
`--output <dir>`	output dir (default `<build_dir>/output`)
`--phase 1\|2\|all`	1 = stub split only; 2 = IR assembly + compile only; all = both (default)
`--cc <cmd>`	LLVM compiler command (overrides config)
`--build-dir <dir>`	override `build_dir` from `project.vpj` (relative paths still resolve against the project dir)
`--clean`	clean temp / output / includes precompiled cache
`--rebuild-includes`	delete only the includes cache (`includes.binary`), forcing a full includes rebuild
`--clear-cache`	clear only temp (`.vpi` / `.stub.ll` / `.text.ll`), forcing stub regeneration
`--run`	execute the produced binary on success
`--repl`	interactive REPL (M1: expression fragment → DLL → call and echo); without `--project` it looks for the `repl.vpj` template project in cwd, then next to `viperc.exe` (errors out with a `--project` hint if neither exists); mutually exclusive with `--asan` / `--phase2command`
`--asan`	ASan mode: `opt -passes=asan` first; runtimes must be declared in `project.vpj` `linker.static_libs`, else fail-fast
`--phase2command <bat>`	do not run `llc`/link; write the commands into the given `.bat`
`--nowarn-register`	suppress Phase A dependency WARNs (cross-module `text.ll` not yet emitted, converges next pass)
`--no-llvmpeephole`	disable FunctionPrint local peepholes (T2 store/load forwarding, T3 no-op bitcast)
`--detection`	enable MemBuddy allocation detection (`0xCDCD` fill + record-table probe for out-of-bounds writes)
`--show-args`	dump all parsed arguments
`--version`, `-v`	print Viper / VRTE / ViperC versions, commit ids, target triple and LLVM version
`--help`, `-h`	print help

Architecture

all .vp/.py under source_dir (multi-file mode: recursively scanned)
    │
    ├─ Phase 1 (stub split)
    │     IncludesScanner translates includes on demand -> Phase1/*
    │     Translator + Gen/* emit IR, then write two files:
    │       <sha1>.stub.ll   declarations
    │       <sha1>.text.ll   implementations
    │
    ├─ Phase 2 (assemble + compile)
    │     StubMerger*  topologically merge local stub + dependency stubs + local text -> full IR
    │     BuildPipeline: full IR -> llc -> .obj -> clang++ (+ precompiled includes .obj) -> .exe
    │
    └─ <build_dir>/output/<linker.output>

Module responsibilities:

Module	Responsibility
`App/main.vp`	CLI entry: pools and singletons, arg parsing, phase dispatch, `--run`
`lib/Projectrans/Config.vp`	`project.vpj` loading, path resolution, includes cache dir
`lib/core/CompilerFactory.vp`	owns `mb` (main pool) and `perm_pool` (permanent pool)
`lib/core/Gen/Generator.vp`	project-level `Generator` singleton shared by Phase1/Phase2/BuildPipeline
`lib/core/Gen/Translator.vp`	translator body: `translate()` and `dump_ir()` (`.stub.ll` / `.text.ll` / full IR)
`lib/core/Gen/GenApi.vp`	translator public API and global state (e.g. temp dir)
`lib/Phase/Phase1/`	scan includes, topological order, emit declarations on demand
`lib/Phase/Phase2/`	multi-file projects: scan sources, translate, assemble IR, drive the build
`lib/Phase/BuildPipeline.vp`	`.ll` -> `llc` -> `.obj` -> `clang++` -> `.exe` (ASan branch included)
`lib/Stub/StubGen/`	read `.vpi`, convert to stub declarations
`lib/Stub/StubMerger/`	collect / topologically sort / concatenate / store `stub.ll` and `text.ll`
`lib/core/IncludesScanner.vp`	scan `includes/`, decide what needs rebuilding
`lib/core/VLogger.vp`	logging (with the `--nowarn-register` switch)

Bootstrap note: after self-hosting, module-level globals carry a SHA1 prefix (module isolation), so singletons such as _g_gen must be assigned per module (GenApi._g_gen = gen …), not once by bare name in main.vp — otherwise some modules read their own None, showing up as bogus results like "global-name test always 0".

Memory model

There is no per-object system malloc churn — only two MemBuddy buddy-allocator pools:

POOL_SIZE = 1 GiB (1073741824) — main pool mb, ordinary translation allocations.
PERM_POOL_SIZE = 536871176 (≈512 MiB usable) — permanent pool perm_pool, objects that outlive a phase.
Both are held by the CompilerFactory.factory singleton; mb is additionally injected per module as sys._mbuddy / argparse._mbuddy / ast._mbuddy / Config._mbuddy …

--detection switches both pools into detection mode (0xCDCD fill + record-table probe) to hunt out-of-bounds writes.

Relation to TransPyC

Dimension	TransPyC	ViperC
Implementation	CPython	Viper
Role	bootstrap seed (only used to produce a stable ViperC)	the current compiler
Self-hosted?	—	✅ yes, it compiles itself
Entry	`TransPyC/Projectrans.py`	`App/main.vp`
AST parsing	Python `ast` / in-house vast	`includes/ast` (9 files, in-house)
Project config	`project.json`	`project.vpj`
Platform	Windows / Linux	Windows (`win32.*`)

12 KiB Raw Permalink Blame History