Files
Viper_VRTE/memhub.vp
T
TermiNexus 0055ab7ea7 refactor: rawptr[T] -> T* 与 t.CConst 前缀迁移(R14/R16)
R14:rawptr[X] -> X*(41 文件 / 2037 行改动),X 含模块限定名与嵌套下标(rawptr[GSList[NameEntry]] -> GSList[NameEntry]*)。保留 6 处 rawptr[t.CPtr](规范例外:存纯裸指针);# 注释与字符串里的 rawptr[...] 属元语法描述,未动。 R16(保守子集):t.CConst | str/bytes -> const str/bytes,t.CConst | char | t.CPtr -> const str;typedef 右值位置豁免(const str 在该处无法注册,与 R1 同源)。 验证:TPC 自举三重重编 SUCCESS;TransPyV.exe 全量 77/77 顺序回归通过。
2026-10-05 17:38:21 +08:00

2255 lines
122 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import viperio
import stdlib
import string
import atom
import stdio
# ============================================================
# memhub - Unified memory management library
# MemManager (polymorphic base class, vtable)
# ├─ MemPool (arena bump allocator, no support for individual free)
# ├─ MemSlab (fixed-size block pool, alloc/free tracked via bitmap)
# └─ MemBuddy (buddy system, supports alloc/free/realloc with coalescing)
# All subclasses override alloc/free/reset via vtable, invokable polymorphically through MemManager pointer
# ============================================================
define MEMHUB_ALIGN = 8
# MemSlab bitmap constants.
define MEMSLAB_MIN_BLOCK = 16
define MEMSLAB_BITMAP_BYTES = 256
# MemBuddy buddy constants.
define MEMBUDDY_MIN_BLOCK = 32
define MEMBUDDY_MAX_ORDERS = 32
define MEMBUDDY_HEADER_SIZE = 8
# Block-tail canary: alloc writes 8 sentinel bytes past the user area and check_alive_blocks
# verifies them; an overflow breaks the canary and pinpoints the writing file deterministically.
define MEMBUDDY_CANARY = 8
# Diagnostic fill guard: alloc can fill the user area with 0xCD so fields a constructor never wrote
# stay 0xCDCD... and stand out in field dumps. 0 = off (default), 1 = on. Off does not affect the
# self_check/canary invariants: the tail sentinel comes from MEMBUDDY_CANARY/CANARY_PATTERN and the
# post-free fill is controlled by detection, both independent of this switch.
define MEMBUDDY_FILL_GUARD = 0
define MEMBUDDY_FILL_PATTERN = 0xCD
# [DET] Allocation detection (2-byte 0xCDCD markers around the user area + record-table probing):
# in detection mode (MemBuddy(..., detection=1)) each alloc is laid out as
# [0xCDCD][requested bytes][0xCDCD] (4 bytes more than requested)
# with the returned pointer = user area + 2; free steps back 2 bytes. Every surviving tail marker
# (at ptr+size) is checked: a broken one = out-of-bounds write, reported with ptr/size and terminated.
define MEMBUDDY_DET_MARKER = 0xCDCD
# Default record area size (32 MiB; each record {ptr u64, size u64} = 16 B).
define MEMBUDDY_DET_RECORD_SIZE = 33554432
# Sampling period (power of two): every N allocs run one full-table scan (overlap + probe). The
# record append stays O(1); corruption is persistent, so a delayed scan still finds it, while a full
# O(n) scan per alloc would be O(n^2). Runtime override: a MemBuddy with deep_detect != 0
# (--detection-deep) ignores this period and behaves as 1.
define MEMBUDDY_DET_PROBE_PERIOD = 32
# [hot] Full-table periodic-scan master switch (compile-time, default 0 = off; same pattern as
# MEMBUDDY_THREAD_SAFE/MEMBUDDY_FILL_GUARD). Every MEMBUDDY_DET_PROBE_PERIOD allocs run two O(n)
# scans inside MemBuddy.alloc (record overlap + [MEMDBG] residue), the alloc O(n) hotspot, so the
# whole block is gated here. Default 0 keeps alloc O(1) (branch not taken); set 1 (with
# _memdbg_enable for the second check) to hunt duplicate allocation / free-list residue / overflow.
# Runtime override: deep_detect != 0 forces the scan without changing this constant. The branch only
# prints and exits (no state writes), so off only drops diagnostics.
define MEMBUDDY_PERIOD_SCAN = 0
# [MEMDBG] Temporary instrumentation (locate duplicate free-list put / dangling free / duplicate
# block handout; remove once located). Points: (1) alloc - in the sampling branch, check the fresh
# block is not still on its order's free list; (2) free - before returning, check the block is not
# already on that list; (3) free - no longer silently ignore invalid/dangling pointers (in-pool but
# even header); (4) existing [DET]/[MEM] exit paths add stdio.fflush(None).
define MEMBUDDY_DBG = 1
# [MEMDBG] Runtime switch, default on (follows MEMBUDDY_DBG). A negative test (e.g. BuddyTest doing a
# deliberate double free, expecting a silent reject) can disable it at runtime:
# import memhub
# memhub._memdbg_enable = 0
_memdbg_enable: int = MEMBUDDY_DBG
# [lock] Thread-safety compile-time switch: default 0 = single-threaded (the spinlock is a no-op
# body). Bootstrapping is single-threaded and each alloc otherwise pays one atomic xchg + one atomic
# store. Set 1 = multi-threaded (keeps the atomic spinlock). Compile-time is chosen because a
# constant comparison folds the branch away entirely, truly removing the atomic cost, whereas a
# runtime switch still needs a global read + branch per alloc. Multi-threaded MemBuddy use must
# set this to 1.
define MEMBUDDY_THREAD_SAFE = 0
# Block-tail canary value (0xA5A5A5A5A5A5A5A5, 64-bit). A define makes every use an immediate
# (the old _canary_pattern() call ran ~26.58M times). Built by shifting (lo << 32) | lo to avoid
# writing the large decimal literal this repo has seen truncated.
define MEMBUDDY_CANARY_PATTERN = (0xA5A5A5A5 << 32) | 0xA5A5A5A5
# [FREEFRAME] Free-block metadata stamps (forensics; written/checked only when detection != 0).
#
# Rationale: _fl_pop's value guard turns "a free block's first word was written through" into a
# deterministic report, but its two numbers (block / next) cannot name the writer (next is often
# non-pointer data). Stamps record "which words of which free block became what, over what span" in
# the block itself, and the release point attributes the overflow to a concrete allocation.
#
# Free-block layout (occupied only when detection != 0; bytes 8..31 are otherwise unused since
# memhub only reads/writes word0 of a free block):
# word0 = next pointer (existing semantics)
# word1 = MEMBUDDY_FREEFRAME_TAG ^ (order << 32) -- order stamp
# word2 = MEMBUDDY_FREEFRAME_TAG2 -- second check word
# word3 = mirror of word0 (next copy)
# All four equal to their expected values => intact; otherwise the number of mismatched words bounds
# the write span. Both constants are < 2^63, avoiding the large-literal truncation issue.
define MEMBUDDY_FREEFRAME_TAG = 0x4652454500000000
define MEMBUDDY_FREEFRAME_TAG2 = 0x4652454600000000
# [DETSITE] Call-site tag constants (stable, readable): identify the site (file:line) of an alloc.
# A record holds only {ptr, size}, so a writer's size alone cannot pin the code point; alloc_tagged
# stores a tag at the same record index and the report prints site=file:line. Values are fixed (not
# random/time based); 0 means "no tag" (report prints site=?, never fabricated).
define DETSITE_NONE = 0 # no tag (plain alloc / not registered)
define DETSITE_STRUCTGENMIXIN_570 = 1 # StructGenMixin.vp:570 add_field field-name copy
define DETSITE_STRUCTGENMIXIN_698 = 2 # StructGenMixin.vp:698 _deep_copy_list_ast elts data
define DETSITE_STRUCTGENMIXIN_1804 = 3 # StructGenMixin.vp:1804 cross-module SHA1 buffer 32B
define DETSITE_CALLGENCTOR_1175 = 4 # CallGenCtor.vp:1175 generic ctor extra_args
define DETSITE_CALLGENCTOR_1186 = 5 # CallGenCtor.vp:1186 generic ctor arg_filled
define DETSITE_STRUCTGENMIXIN_641 = 6 # StructGenMixin.vp:641 _deep_copy_default_ast Constant node
define DETSITE_STRUCTGENMIXIN_659 = 7 # StructGenMixin.vp:659 _deep_copy_default_ast string copy
define DETSITE_STRUCTGENMIXIN_693 = 8 # StructGenMixin.vp:693 _deep_copy_list_ast elts container
define DETSITE_LIST_43 = 9 # _list.vp:43 list.__init__ element array capacity*elem_size
define DETSITE_STRUCTGENMIXIN_372 = 10 # StructGenMixin.vp:372 register_struct field array (permanent pool)
define DETSITE_STRUCTGENMIXIN_374 = 11 # StructGenMixin.vp:374 register_struct field array (main pool)
define DETSITE_LEAFKIT_1013 = 12 # CallGenLeafKit.vp:1013 global function table max_count*FuncEntry.__sizeof__()
define DETSITE_VARGEN_27 = 13 # VarGenMixin.vp:27 variable table MAX_VARS*VarEntry.__sizeof__()
define DETSITE_VPIREADER_2626 = 14 # VpiReader.vp:2626 vpi class field-name table 8*GenTypes.FIELD_MAX (per class)
define DETSITE_VPIREADER_2627 = 15 # VpiReader.vp:2627 vpi class field-type table 8*GenTypes.FIELD_MAX (per class)
define DETSITE_CLASSGENDEF_153 = 16 # ClassGenDef.vp:153 parent-name table 8*GenTypes.PARENT_MAX (per class)
define DETSITE_CLASSGENVTABLE_91 = 17 # ClassGenVTable.vp:91 vtable method-name table 8*VTBL_METHOD_MAX (per class, permanent pool)
# [DETSITE] Module-level side table for record tags (no MemBuddy field, no layout change):
# entry i = {ptr, tag} (2×u64), same index and lifetime as MemBuddy._det_records. ptr is used only
# for index validation (a shifted index or cross-instance sharing mismatches -> 0 -> site=?).
# Never written/read when detection == 0 (alloc_tagged forwards directly).
_det_site_arena: uint64* = None
_det_site_cap: sizet = 0
def _det_site_put(idx: sizet, ptr: uint64, tag: uint64):
"""Record the call-site tag for record idx (lazily doubling the table).
If stdlib.malloc/realloc fails nothing is recorded (return), so the report prints site=? --
never a silent downgrade or a fabricated site. Called only when detection != 0.
"""
global _det_site_arena, _det_site_cap
if idx >= _det_site_cap:
new_cap: sizet = _det_site_cap * 2
if new_cap <= idx:
new_cap = idx + 1
if new_cap < 64:
new_cap = 64
new_ptr: rawptr = None
if _det_site_arena is None:
new_ptr = stdlib.malloc(new_cap * 16)
else:
new_ptr = stdlib.realloc(rawptr(_det_site_arena), new_cap * 16)
if new_ptr is None:
return
_det_site_arena = (uint64*)(new_ptr)
_det_site_cap = new_cap
_det_site_arena[idx * 2] = ptr
_det_site_arena[idx * 2 + 1] = tag
def _det_site_get(idx: sizet, ptr: uint64) -> uint64:
"""Tag of record idx; out of range or ptr mismatch (shifted / cross-instance) returns 0."""
if _det_site_arena is None or idx >= _det_site_cap: return 0
if uint64(_det_site_arena[idx * 2]) != ptr: return 0
return uint64(_det_site_arena[idx * 2 + 1])
def _det_site_name(tag: uint64) -> str:
"""tag -> "file:line" registry; unregistered tags (including 0) return "?"."""
if tag == uint64(DETSITE_STRUCTGENMIXIN_570): return "StructGenMixin.vp:570"
if tag == uint64(DETSITE_STRUCTGENMIXIN_698): return "StructGenMixin.vp:698"
if tag == uint64(DETSITE_STRUCTGENMIXIN_1804): return "StructGenMixin.vp:1804"
if tag == uint64(DETSITE_CALLGENCTOR_1175): return "CallGenCtor.vp:1175"
if tag == uint64(DETSITE_CALLGENCTOR_1186): return "CallGenCtor.vp:1186"
if tag == uint64(DETSITE_STRUCTGENMIXIN_641): return "StructGenMixin.vp:641"
if tag == uint64(DETSITE_STRUCTGENMIXIN_659): return "StructGenMixin.vp:659"
if tag == uint64(DETSITE_STRUCTGENMIXIN_693): return "StructGenMixin.vp:693"
if tag == uint64(DETSITE_LIST_43): return "_list.vp:43"
if tag == uint64(DETSITE_STRUCTGENMIXIN_372): return "StructGenMixin.vp:372"
if tag == uint64(DETSITE_STRUCTGENMIXIN_374): return "StructGenMixin.vp:374"
if tag == uint64(DETSITE_LEAFKIT_1013): return "CallGenLeafKit.vp:1013"
if tag == uint64(DETSITE_VARGEN_27): return "VarGenMixin.vp:27"
if tag == uint64(DETSITE_VPIREADER_2626): return "VpiReader.vp:2626"
if tag == uint64(DETSITE_VPIREADER_2627): return "VpiReader.vp:2627"
if tag == uint64(DETSITE_CLASSGENDEF_153): return "ClassGenDef.vp:153"
if tag == uint64(DETSITE_CLASSGENVTABLE_91): return "ClassGenVTable.vp:91"
return "?"
# [DETSITE] Tag of the alloc_tagged call site currently executing (0 = not an alloc_tagged path).
# alloc itself cannot see the call site but alloc_tagged knows its tag; this global lets both the
# zero-size and the allocation-failure guards print site=file:line directly. Registration is
# independent of detection (location is not detection data).
_alloc_pending_site: uint64 = 0
# Process-wide zero-size request counter: incremented only in the failure branch (never on the
# allocation hot path), to distinguish "first ever" from "repeatedly 0 from data".
_zero_alloc_hits: sizet = 0
def _alloc_site_enter(tag: uint64) -> uint64:
"""Enter an alloc_tagged call site: store tag and return the overridden previous tag.
The caller restores the returned value after alloc returns, so nested alloc_tagged calls on the
same pool (or alternating across pools) do not overwrite each other's location.
"""
global _alloc_pending_site
prev: uint64 = _alloc_pending_site
_alloc_pending_site = tag
return prev
def _alloc_site_leave(prev: uint64):
"""Leave an alloc_tagged call site: restore the tag from before entering."""
global _alloc_pending_site
_alloc_pending_site = prev
def _alloc_site_current() -> uint64:
"""Tag of the current alloc_tagged call site; 0 = unknown (report prints site=?)."""
return _alloc_pending_site
# A rejected free (_is_valid_ptr invalid) is reported only once: this silent no-op leaks the block
# forever, so the first occurrence must leave evidence.
_rej_free_reported: int = 0
def _zero_alloc_note() -> sizet:
"""Count one zero-size request and return the running total."""
global _zero_alloc_hits
_zero_alloc_hits += 1
return _zero_alloc_hits
# [CTX] Current context (stage name / current file or module name): set by TPV via SetAllocCtx at
# key stages, used only to attribute zero-size / allocation-failure reports. The two fields are kept
# separate (no concatenation, no allocation). None = not set => report prints ? (never fabricated).
_alloc_ctx_stage: bytes = None
_alloc_ctx_detail: bytes = None
def SetAllocCtx(stage: bytes, detail: bytes = None):
"""Set the current context (TPV side); the caller keeps both strings alive for the stage."""
global _alloc_ctx_stage, _alloc_ctx_detail
_alloc_ctx_stage = stage
_alloc_ctx_detail = detail
def _alloc_ctx_stage_str() -> bytes:
"""Current stage name; "?" when unset."""
if _alloc_ctx_stage is None: return "?"
return _alloc_ctx_stage
def _alloc_ctx_detail_str() -> bytes:
"""Current file/module name; "?" when unset."""
if _alloc_ctx_detail is None: return "?"
return _alloc_ctx_detail
def _canary_pattern() -> uint64:
"""Block-tail canary value (64-bit) -- single reference to the compile-time constant."""
return uint64(MEMBUDDY_CANARY_PATTERN)
def _align_up(val: sizet, align: sizet) -> sizet:
if align == 0: return val
return (val + align - 1) & ~(align - 1)
def _largest_pow2_le(val: sizet) -> sizet:
# `p <= val / 2` equals `p * 2 <= val` (integer division) without multiplying: the original
# `p * 2` wraps to 0 for val > 2^63, making `0 <= val` always true (infinite loop).
if val == 0: return 0
p: sizet = 1
while p <= (val >> 1):
p = p << 1
return p
def _block_size_at_order(order: int) -> sizet:
"""Block size at `order` (equivalent to MEMBUDDY_MIN_BLOCK << order).
Hot-path optimisation: called ~100.8M times; the original `for i in range(order): bs <<= 1` did
`order` shifts, equivalent to one left shift, so it is composed directly (O(order) -> O(1)).
Edge: order <= 0 returns the minimum block (identical to the old empty range loop) and avoids
the undefined negative shift.
"""
return MEMBUDDY_MIN_BLOCK << (0 if order <= 0 else sizet(order))
# ============================================================
# MemManager - Polymorphic base class
# virtual explicit annotation to ensure recognition as vtable class during cross-module imports
# (HandlesImports only detects IsCVTable via decorator; automatic inference is unavailable across modules)
# ============================================================
virtual class MemManager:
__provides__: list[str] = ['__memmgr__']
base: rawptr # Starting address of memory region
size: sizet # Total size of memory region
def __init__(self, base: rawptr, size: sizet):
self.base = base
self.size = size
# === Virtual functions (overridden by subclasses) ===
def alloc(self, size: sizet) -> rawptr:
# Default: allocation not supported
return None
def free(self, ptr: rawptr) -> int:
# Default: no operation
return 0
def reset(self) -> int:
# Default: no operation
return 0
# === Concrete methods (invoke virtual functions for automatic polymorphic dispatch) ===
def calloc(self, count: sizet, size: sizet) -> rawptr:
total: sizet = count * size
ptr: rawptr = self.alloc(total)
if ptr is not None:
string.memset(ptr, 0, total)
return ptr
def realloc(self, ptr: rawptr, new_size: sizet) -> rawptr:
if ptr is None:
return self.alloc(new_size)
if new_size == 0:
self.free(ptr)
return None
new_ptr: rawptr = self.alloc(new_size)
if new_ptr is None:
return ptr
# Base class cannot track original allocation size; data will not be copied. Subclasses may override to preserve data
self.free(ptr)
return new_ptr
def __enter__(self) -> MemManager*:
return self
def __exit__(self):
self.reset()
def alloc_buf(self, capacity: sizet) -> viperio.Buf*:
buf: viperio.Buf* = self.alloc(viperio.Buf.__sizeof__())
if buf is None: return None
data_ptr: rawptr = self.alloc(capacity)
if data_ptr is None: return None
buf.__before_init__()
buf.__init__(data_ptr, capacity)
return buf
# ============================================================
# MemPool - Arena bump allocator
# Individual free operations are unsupported; reset reclaims all memory at once
# ============================================================
class MemPool(MemManager):
offset: sizet # Bump pointer offset
high_water: sizet # High-water mark
def __init__(self, base: rawptr, size: sizet):
self.base = base
self.size = size
self.offset = 0
self.high_water = 0
# Capacity pre-check (pure decision: does not move offset, does not exit(1)).
# Returns 1 = this alloc fits; 0 = it would exhaust (same criteria as alloc: size == 0, the
# same _align_up, the same offset+aligned > size). A caller that gives one big chunk at once
# (e.g. a per-entry reuse pool) can self-check and turn "exhausted => printf + exit(1)" into a
# controllable failure.
def AllocOk(self, size: sizet) -> int:
if size == 0:
return 0
aligned: sizet = _align_up(size, MEMHUB_ALIGN)
if self.offset + aligned > self.size:
return 0
return 1
# Exhaustion stays fail-fast (printf + exit(1)) rather than returning None:
# * MemPool is a free-less bump allocator;
# * its instances flow through `rawptr[memhub.MemManager]` parameters to many callers that do
# not check None (includes/ast/**, includes/llvmlite/**, includes/numpy/**, 200+ `.alloc(`
# sites), where None would regress a clear [OOM] diagnostic into a null dereference;
# * this round cannot compile/run to add per-site None checks.
# New callers needing a controllable failure should pre-check with AllocOk(). The None contract
# is provided by MemBuddy; MemPool's contract stays "never returns None, terminates on exhaustion".
def alloc(self, size: sizet) -> rawptr:
if size == 0: return None
aligned: sizet = _align_up(size, MEMHUB_ALIGN)
if self.offset + aligned > self.size:
stdio.printf("\n[OOM] 内存不足 (Out of Memory)\n")
stdio.printf("[OOM] 尝试分配: %zu 字节,失败\n", size)
stdio.printf("[OOM] MemPool: total=%zu used=%zu (bump allocator, 无释放)\n",
self.size, self.offset)
stdio.printf("[OOM] 说明: 内存池已耗尽。可能原因: (1) 池大小不足,需增大 POOL_SIZE; (2) 存在内存泄漏。\n")
stdio.printf("[OOM] 提示: 需要可控失败而非终止的调用方,请先用 MemPool.AllocOk(size) 预检。\n")
stdlib.exit(1)
ptr: rawptr = self.base + self.offset
self.offset += aligned
if self.offset > self.high_water:
self.high_water = self.offset
return ptr
def free(self, ptr: rawptr) -> int:
# Bump allocator does not support individual free operations
return 1
def reset(self) -> int:
self.offset = 0
self.high_water = 0
return 1
# ============================================================
# Heap-construction factories
#
# A class-construction expression (`memhub.MemPool(arena, cap)`) places its storage on the CALLER's
# stack frame (alloca -> __before_init__ -> __init__); storing that pointer in a module-level
# variable or handing it to another frame leaves the holder a soon-dead stack address.
#
# Managers that must outlive a frame are built by the factories below: take storage from a
# stdlib.malloc heap block, then construct in place. The two steps:
# * __before_init__ writes the vtable pointer and zeroes/defaults the fields;
# * __init__ fills base/size/offset/high_water.
# Afterwards, virtual calls through a MemManager base pointer (alloc/free/reset/realloc/calloc/
# AllocOk) hit the subclass implementation via the vtable.
#
# Usage (MemBuddy precedent):
# p: rawptr[memhub.MemPool] = memhub.NewMemPool(arena, cap)
# if p is None: ...
#
# Lifetime: the manager object is stdlib.malloc'd and shares the arena's process lifetime;
# long-lived (per-process reuse) pools are not freed.
# ============================================================
# Construct a MemPool (arena bump allocator) on the heap. base = arena start, size = its byte count.
# Returns None when malloc fails.
def NewMemPool(base: rawptr, size: sizet) -> MemPool*:
obj: MemPool* = stdlib.malloc(MemPool.__sizeof__())
if obj is None: return None
obj.__before_init__()
obj.__init__(base, size)
return obj
# [DETSITE] Tagged allocation for any MemManager: lets a caller that only holds a base pointer
# (MemManager*) register its site. It only sets _alloc_pending_site (location, independent of
# detection) and then calls the existing virtual alloc => no new virtual function, no vtable change,
# no contract change; for a MemBuddy pool its alloc_tagged is also available (that one additionally
# records a record-table tag for [FREEFRAME] reports).
def MemAllocTagged(pool: MemManager*, size: sizet, tag: sizet) -> rawptr:
if pool is None:
return None
prev: uint64 = _alloc_site_enter(uint64(tag))
r: rawptr = pool.alloc(size)
_alloc_site_leave(prev)
return r
# ============================================================
# MemSlab - Fixed-size block pool
# Bitmap stored at the start of arena; subsequent area split into block_size units managed by free list
# ============================================================
class MemSlab(MemManager):
block_size: sizet # Block size (aligned)
block_count: sizet # Total block count
used_count: sizet # Used block count
free_list: rawptr # Free list head
alloc_map: uint8* # Allocation bitmap (at start of arena)
alloc_map_size: sizet # Bitmap byte count
usable: rawptr # Usable block area start (after bitmap)
usable_size: sizet # Usable block area size
def __init__(self, base: rawptr, size: sizet, block_size: sizet):
self.base = base
self.size = size
self.block_size = 0
self.block_count = 0
self.used_count = 0
self.free_list = None
self.alloc_map = None
self.alloc_map_size = 0
self.usable = None
self.usable_size = 0
if base is None: return
bs: sizet = _align_up(block_size, MEMHUB_ALIGN)
if bs < MEMSLAB_MIN_BLOCK: bs = MEMSLAB_MIN_BLOCK
map_bytes: sizet = _align_up(MEMSLAB_BITMAP_BYTES, MEMHUB_ALIGN)
if size < map_bytes + bs: return
self.alloc_map = base
self.alloc_map_size = map_bytes
self.usable = base + map_bytes
self.usable_size = size - map_bytes
self.block_size = bs
# Clear bitmap
for idx in range(map_bytes):
self.alloc_map[idx] = 0
self.block_count = self.usable_size / bs
if self.block_count == 0: return
# Construct free list: the first 8 bytes of each block store pointer to next block
self.free_list = None
for i in range(self.block_count):
block: rawptr = self.usable + i * bs
*((uint64*)(block)) = uint64(self.free_list)
self.free_list = block
def alloc(self, size: sizet) -> rawptr:
# Slab mode: allocate one block only if requested size <= block_size
if self.free_list is None:
stdio.printf("\n[OOM] 内存不足 (Out of Memory)\n")
stdio.printf("[OOM] 尝试分配: %zu 字节,失败(free_list 已空)\n", size)
stdio.printf("[OOM] MemSlab: block_size=%zu block_count=%zu used=%zu\n",
self.block_size, self.block_count, self.used_count)
stdio.printf("[OOM] 说明: 内存池已耗尽。可能原因: (1) 池大小不足,需增大 POOL_SIZE; (2) 存在内存泄漏。\n")
stdlib.exit(1)
if size > self.block_size:
# Request larger than a block = caller error: fail gracefully with NULL (malloc-like,
# caller decides), distinct from free-list exhaustion (real OOM, prints then exit(1)).
stdio.printf("[WARN] MemSlab: 请求 %zu 字节超过 block_size=%zu,分配失败返回 NULL;应改用 MemBuddy/MemPool。\n",
size, self.block_size)
return None
block: rawptr = self.free_list
self.free_list = rawptr(*((uint64*)(block)))
self.used_count += 1
idx: sizet = (uint64(block) - uint64(self.usable)) / self.block_size
self.alloc_map[idx / 8] = self.alloc_map[idx / 8] | uint8(1 << (idx % 8))
return block
def free(self, ptr: rawptr) -> int:
if ptr is None: return 0
p: uint64 = uint64(ptr)
if p < uint64(self.usable): return 0
if p >= uint64(self.usable) + self.block_count * self.block_size: return 0
if (p - uint64(self.usable)) % self.block_size != 0: return 0
idx: sizet = (p - uint64(self.usable)) / self.block_size
byte_idx: sizet = idx / 8
bit_idx: int = int(idx % 8)
if (self.alloc_map[byte_idx] >> bit_idx) & 1 == 0: return 0
self.alloc_map[byte_idx] = self.alloc_map[byte_idx] & uint8(~(1 << bit_idx))
*((uint64*)(ptr)) = uint64(self.free_list)
self.free_list = ptr
self.used_count -= 1
return 1
def reset(self) -> int:
self.used_count = 0
self.free_list = None
for i in range(self.block_count):
block: rawptr = self.usable + i * self.block_size
*((uint64*)(block)) = uint64(self.free_list)
self.free_list = block
if i / 8 < self.alloc_map_size:
self.alloc_map[i / 8] = 0
return 1
# ============================================================
# MemBuddy - Buddy system allocator
# The free-list head array sits at the arena start; the rest is managed in power-of-two blocks
# with coalescing. Thread-safety is compile-time gated by MEMBUDDY_THREAD_SAFE (0 = single-threaded
# default with the spinlock compiled out; multi-threaded use must set 1).
#
# Checkpoint support (reclaim staged allocations in long tasks such as the compiler):
# mark() -> returns the current allocation watermark (allocates nothing)
# reset_to(m) -> frees every block allocated after watermark m (back onto the free list)
# The record stack is allocated from stdlib.malloc, separate from the buddy area.
# ============================================================
class MemBuddy(MemManager):
# Override parent __provides__: MemBuddy provides both __mbuddy__ and __memmgr__.
# Within a `with MemBuddy(...)` context, classes (e.g. _str) with __requires__=['__mbuddy__']
# can be injected automatically via _find_provider.
__provides__: list[str] = ['__mbuddy__', '__memmgr__']
max_order: int # maximum order
free_lists: uint64* # free-list head array (at the arena start)
lock_val: volatile int # spinlock
usable: rawptr # start of usable memory (skips free_lists)
usable_size: sizet # usable memory size (sum of decomposed blocks)
chk_stack: uint64* # checkpoint record stack (stores block addresses)
chk_cap: sizet # record-stack capacity (entries)
chk_depth: sizet # current record depth
chk_base: rawptr # record-stack base (stdlib.malloc, separate from the buddy area)
big_alloc_count: sizet # count of allocations >= 8KB (OOM diagnostics)
# Free bytes at mark() time (OOM diagnostics): distinguishes "this window filled the pool" from already-full.
mark_free_bytes: sizet = 0
# Record depth at mark() time: for OOM segmented diagnostics (prefix vs this window).
mark_depth: sizet = 0
# [DET] Allocation detection (off by default; enable to hunt overflows / live-block overwrites):
# detection=1 -> 2-byte 0xCDCD markers + record table + a probe after each alloc
detection: int = 0
# Deep-diagnostic switch (--detection-deep -> constructor argument): deep_detect=1 ignores
# MEMBUDDY_DET_PROBE_PERIOD (behaves as 1, probing every alloc) and enables the periodic
# full-table scan. Diagnostics only, very slow. Default 0 = identical to the historical build.
deep_detect: int = 0
detection_size: sizet = MEMBUDDY_DET_RECORD_SIZE
_det_arena: rawptr = None # record-area base (stdlib.malloc, for realloc growth)
_det_records: uint64* = None # record table (each 2×u64 = {ptr, size})
_det_count: sizet = 0 # number of records
_det_cap: sizet = 0 # record capacity (entries)
def __init__(self, base: rawptr, size: sizet,
detection: int = 0, deep_detect: int = 0):
self.base = base
self.size = size
self.detection = detection
# Default argument (same pattern as detection): existing 2-arg / 3-arg construction sites
# need no change.
self.deep_detect = deep_detect
# Explicit assignment: do not rely on field defaults for a CDefine constant (a garbage
# detection_size would malloc a 1-byte record area and then realloc double-free).
self.detection_size = MEMBUDDY_DET_RECORD_SIZE
self._det_arena = None
self._det_records = None
self._det_count = 0
self._det_cap = 0
self.max_order = 0
self.free_lists = None
self.lock_val = 0
self.usable = None
self.usable_size = 0
self.chk_stack = None
self.chk_cap = 0
self.chk_depth = 0
self.chk_base = None
self.big_alloc_count = 0
self.mark_free_bytes = 0
self.mark_depth = 0
self.lock_val = 0
fl_bytes: sizet = (MEMBUDDY_MAX_ORDERS + 1) * 8
self.free_lists = base
# Initialize all free list heads to be 0
i: int
for i in range(MEMBUDDY_MAX_ORDERS + 1):
self.free_lists[i] = 0
if size <= fl_bytes:
self.usable = None
self.usable_size = 0
self.max_order = 0
return
remaining: sizet = size - fl_bytes
# The checkpoint stack is not reserved here; mark() lazily allocates it from the buddy pool.
# Non-power-of-two arenas are split by binary decomposition into power-of-two free blocks;
# buddy coalescing stops at decomposition boundaries because the orders no longer match.
max_blk: sizet = _largest_pow2_le(remaining)
if max_blk < MEMBUDDY_MIN_BLOCK:
self.usable = None
self.usable_size = 0
self.max_order = 0
return
self.usable = base + fl_bytes
# max_order = log2(max_blk / MIN_BLOCK)
self.max_order = 0
bs: sizet = MEMBUDDY_MIN_BLOCK
while bs < max_blk:
bs = bs << 1
self.max_order += 1
# Binary decomposition: split [usable, usable+remaining) into power-of-two free blocks.
# The tail fragment (< MIN_BLOCK) is not pushed; usable_size counts only real blocks.
pos: sizet = 0
rem: sizet = remaining
used_area: sizet = 0
while rem >= MEMBUDDY_MIN_BLOCK:
blk: sizet = _largest_pow2_le(rem)
o: int = 0
ob: sizet = MEMBUDDY_MIN_BLOCK
while ob < blk:
ob = ob << 1
o += 1
self._fl_push(o, self.usable + pos)
pos += blk
used_area += blk
rem -= blk
self.usable_size = used_area
# Allocate the record area separately with stdlib.malloc (not from the buddy area, same
# policy as chk_stack). Capacity = detection_size // 16 (each entry {ptr, size}).
# Records + overlap detection are decoupled from detection: even without it the record area
# is allocated and same-address duplicate-allocation detection is on (no markers, no layout
# change), which locates free-list / duplicate-allocation corruption.
det_arena: rawptr = stdlib.malloc(self.detection_size)
if det_arena is not None:
self._det_arena = det_arena
self._det_records = uint64*(det_arena)
self._det_cap = self.detection_size // 16
self._det_count = 0
# === Checkpoint support ===
def mark(self) -> sizet:
"""Return the current allocation watermark (checkpoint).
The record stack is stdlib.malloc'd (independent of the buddy area, so it neither consumes
the pool nor risks reset_to freeing it or overlapping a buddy block). Initial capacity is
2M entries (16 MB): translating a single file can produce hundreds of thousands of allocs,
so a 64K initial capacity would grow on the first big file, and a failed growth would stop
recording blocks -> reset_to becomes a permanent no-op -> all allocations leak -> 1 GB main
pool OOM.
"""
if self.chk_stack is None:
cap: sizet = 2 * 1024 * 1024
base: rawptr = stdlib.malloc(cap * 8)
if base is None:
return 0
self.chk_base = base
self.chk_stack = (uint64*)(base)
self.chk_cap = cap
self.chk_depth = 0
# OOM diagnostics: record free bytes and record depth at this mark, so a failure report can
# distinguish "this window filled the pool" from already-full-before-window.
self.mark_free_bytes = self.free_bytes()
self.mark_depth = self.chk_depth
return self.chk_depth
def reset_to(self, mark: sizet):
"""Free every block allocated after watermark `mark`, returning them to the free list.
Each recorded block: if its header is still marked allocated (odd), _coalesce returns it;
if it was already freed individually (header == 0), skip to avoid a duplicate return.
"""
if self.chk_stack is None:
return
if mark > self.chk_depth:
# mark beyond the current depth: the caller's mark is bogus (wrong field / cross-
# instance), which makes reset a no-op and leaks every later allocation (1 GB OOM).
# fail-fast.
stdio.printf(
"[MEM] reset_to mark=%zu > chk_depth=%zu(mark 异常,reset 失效,终止)\n",
mark, self.chk_depth)
return
if mark >= self.chk_depth:
return
self._lock()
while self.chk_depth > mark:
self.chk_depth -= 1
block: rawptr = rawptr(self.chk_stack[self.chk_depth])
if block is None:
continue
stored: rawptr = rawptr(*((uint64*)(block)))
stored_val: uint64 = uint64(stored)
if (stored_val & 1) == 0:
continue # already freed individually: skip
order: int = int(stored_val >> 1)
if order < 0 or order > self.max_order:
continue
# [FREEFRAME] Pre-free owner check (detection only): a reclaimed block is an owner-known
# release point, so a broken tail canary means its owner is the writer that ran into the
# right free block's first word. Returns immediately when detection == 0.
self._ff_owner_check(block, order)
self._coalesce(block, order)
# [DET] reset-aware: the record table and chk stack have equal length (each alloc appends
# one, free only sets a flag), so truncating chk_depth must truncate _det_count too;
# otherwise old records stay "active" and a reused block is misreported as an overlap.
# The record table is decoupled from detection, so this condition must not be gated on it.
if self._det_records is not None:
self._det_count = mark
self._unlock()
def reset_all(self):
"""Free every allocated block (equivalent to reset_to(0))."""
if self.chk_stack is None:
return
self.reset_to(0)
# === Free list operations ===
def _fl_push(self, order: int, block: rawptr):
old_head: uint64 = self.free_lists[order]
*((uint64*)(block)) = old_head
self.free_lists[order] = uint64(block)
# [FREEFRAME] In detection mode also stamp the free block (words 1..3), checked by
# _fl_pop before popping. With detection == 0 this is one predictable branch and no
# memory write. This is the ONLY "block enters the free list" write site, so the stamps
# cover every path onto the list.
if self.detection != 0:
self._ff_push_mark(order, block, old_head)
def _fl_block_ok(self, block: rawptr) -> int:
"""Validate a free-list block address: it must lie in [usable, usable+usable_size) and
be aligned to MEMBUDDY_MIN_BLOCK (same range+alignment criteria as self_check).
Unlike _is_valid_ptr, this checks a FREE block whose header is 0 (allocated blocks write
(order<<1)|1), so the "header must be odd" test does not apply. Used by _fl_pop to reject
a list head / next pointer overwritten with garbage before dereferencing it.
"""
if block is None:
return 0
if self.usable is None:
return 0
if uint64(block) < uint64(self.usable):
return 0
if uint64(block) >= uint64(self.usable) + self.usable_size:
return 0
if (uint64(block) - uint64(self.usable)) % MEMBUDDY_MIN_BLOCK != 0:
return 0
return 1
# === [FREEFRAME] free-block stamps: forensics (read/write diagnostic words; detection only) ===
def _ff_word(self, block: rawptr, idx: int) -> uint64:
"""Read word idx of a free block (0=next, 1=order stamp, 2=check word, 3=next mirror)."""
return (uint64*)(block)[idx]
def _ff_word_set(self, block: rawptr, idx: int, val: uint64):
"""Write word idx of a free block (diagnostic words only; word0 is never touched)."""
p: uint64* = (uint64*)(block)
p[idx] = val
def _ff_stamp(self, order: int) -> uint64:
"""Order stamp of a free block of `order` = TAG ^ (order << 32) (order is recoverable)."""
return uint64(MEMBUDDY_FREEFRAME_TAG) ^ (uint64(order) << 32)
def _ff_push_mark(self, order: int, block: rawptr, next_val: uint64):
"""Stamp words 1..3 after linking. The caller has already checked detection != 0 and
written word0 = next_val.
Called only by _fl_push (the sole link-in site) and by _fl_find_and_remove's middle
unlink (mirror sync after rewriting prev's next), so while a block is on the list words
1..3 always equal the expected values; any deviation can only come from an off-list
write (overflow / wild pointer).
"""
self._ff_word_set(block, 1, self._ff_stamp(order))
self._ff_word_set(block, 2, uint64(MEMBUDDY_FREEFRAME_TAG2))
self._ff_word_set(block, 3, next_val)
def _ff_report(self, order: int, block: rawptr):
"""[FREEFRAME] forensic dump (the caller then fflush + exit(1)).
Three parts that together name the writer:
1. the free block's four words (actual vs expected) => the overwritten byte span;
2. left-neighbour probe: for each order o take L = A - (MIN_BLOCK<<o); if L's header is
odd and (hdr>>1) == o then L is the allocation whose tail lands exactly on A (the
only possible contiguous-overflow origin); its tail canary is checked, but geometry
plus canary alone cannot attribute, hence the evidence-based _ff_left_classify;
3. nearby records (|ptr-A| <= 256): what was allocated around this area and how big, to
trace back the writer's size and call site.
"""
w0: uint64 = self._ff_word(block, 0)
w1: uint64 = self._ff_word(block, 1)
w2: uint64 = self._ff_word(block, 2)
w3: uint64 = self._ff_word(block, 3)
exp1: uint64 = self._ff_stamp(order)
exp2: uint64 = uint64(MEMBUDDY_FREEFRAME_TAG2)
stdio.printf("[FREEFRAME] 空闲块四字: block=%llu order=%d w0(next)=%llu w1=%llu(期望=0x%llx) w2=%llu(期望=0x%llx) w3(next镜像)=%llu\n",
uint64(block), order, w0, w1, exp1, w2, exp2, w3)
dev_n: int = 0
if w3 != w0:
dev_n += 1
if w1 != exp1:
dev_n += 1
if w2 != exp2:
dev_n += 1
# Span inference: dev_n only bounds "how many 8-byte words inside THIS block were
# overwritten", so only a range is derivable: block-only span = [8*n, 8*(n+1)-1] bytes;
# if the left neighbour's tail canary is also broken the same run started 8 bytes earlier,
# shifting the span to [8*(n+1), 8*(n+2)-1]. The four stamped words sit at offsets
# 0/8/16/24, so a run reaching offset 0 covers word0.
lo_f: sizet = sizet(dev_n) * 8
hi_f: sizet = (sizet(dev_n) + 1) * 8 - 1
stdio.printf("[FREEFRAME] 跨度推断: 不符字数=%d/3 ⇒ 本空闲块内写入跨度 ∈ [%zu, %zu] 字节(通用式 [8n, 8n+7],n=不符字数;n=1⇒8..15, n=2⇒16..23, n=3⇒>=24);含左邻 canary 则 ∈ [%zu, %zu]\n",
dev_n, lo_f, hi_f, lo_f + 8, hi_f + 8)
# w1 / w3 cause: _fl_push is the only link-in site and always stamps w1 (non-zero), so
# w2 == TAG2 proves the stamp was applied and w1 == 0 must be an off-list write; if
# w2 != TAG2 the whole stamp is untrustworthy and w1 == 0 is not attributed on its own.
if w2 == exp2:
stdio.printf("[FREEFRAME] w1 成因: w2==TAG2 ⇒ 本块确经 _fl_push 打过印记(_ff_push_mark 三字同写)⇒ w1=0 是链外写入者清零,非入链漏打印记\n")
else:
stdio.printf("[FREEFRAME] w1 成因: w2!=TAG2 ⇒ 印记整体不可信(入链未打或三字俱被覆盖),w1=0 不单独归因\n")
if w3 == 0:
stdio.printf("[FREEFRAME] w3 成因: 若写入范围未达偏移 24 ⇒ w3 保持入链原值 = 入链时的 old_head ⇒ =0 表示入链时 free_lists[order]==0(本块是链尾,next=NULL)\n")
else:
stdio.printf("[FREEFRAME] w3 成因: w3!=0 但 !=w0 ⇒ 或 w0 被改(镜像失配)或 w3 本身被改,需结合跨度区间判读\n")
# Left-neighbour probe: clamp max_order if it was written through, to avoid spinning.
mo: int = self.max_order
if mo < 0 or mo > MEMBUDDY_MAX_ORDERS:
mo = MEMBUDDY_MAX_ORDERS
stdio.printf("[FREEFRAME] 左邻探测(块尾正好落在 A 的分配块 = 越界写来源): max_order=%d usable=%llu\n",
mo, uint64(self.usable))
o: int = 0
found_l: int = 0
while o <= mo:
bs_o: sizet = MEMBUDDY_MIN_BLOCK << sizet(o)
if uint64(block) >= uint64(self.usable) + bs_o:
l_addr: uint64 = uint64(block) - bs_o
lp: uint64* = (uint64*)(l_addr)
lhdr: uint64 = lp[0]
if (lhdr & 1) != 0 and int(lhdr >> 1) == o:
lcan: uint64 = *((uint64*)(block - MEMBUDDY_CANARY))
found_l += 1
if lcan != uint64(MEMBUDDY_CANARY_PATTERN):
stdio.printf("[FREEFRAME] 阶 o=%d L=%llu hdr=0x%llX 用户ptr=%llu canary=0x%llx ⇒ canary 损坏(本块可能是写入者,也可能是被同一写者波及的受害者)——归属见下方「左邻归属判据/判定」两行\n",
o, l_addr, lhdr, l_addr + MEMBUDDY_HEADER_SIZE,
lcan)
# Look up the writer's allocation: the record key is the block's returned
# user pointer (detection: block+10, else block+8). The hit's sz is the
# writer's REQUESTED size, which pins the allocation site.
if self._det_records is not None:
l_usr: uint64 = l_addr + MEMBUDDY_HEADER_SIZE
if self.detection != 0:
l_usr = l_usr + 2
lr_hit: int = 0
for lr_i in range(self._det_count):
if uint64(self._det_records[lr_i * 2]) == l_usr:
# Single load + explicit bit test (print and decision share the
# source word): the raw 8 bytes of the size slot are read once.
w_raw_r: uint64 = self._det_records[lr_i * 2 + 1]
w_sz_r: uint64 = w_raw_r & ((uint64(1) << 63) - 1)
w_fr_r: int = 0
if (w_raw_r & (uint64(1) << 63)) != 0:
w_fr_r = 1
# Call-site tag (same index as the record): a hit maps to
# file:line via a stable constant; no tag -> "?", never faked.
w_site_r: str = _det_site_name(
_det_site_get(lr_i, l_usr))
stdio.printf("[FREEFRAME] 写入者分配记录: [%zu] ptr=%llu sz=%llu freemark=%d 原始字=%llu site=%s(sz 即写入者 alloc 请求尺寸;freemark=1 表该 ptr 已 free、0 表存活;site=该记录的调用点 tag 映射,无 tag 打 ?)\n",
lr_i, l_usr, w_sz_r, w_fr_r, w_raw_r, w_site_r)
lr_hit += 1
if lr_hit == 0:
stdio.printf("[FREEFRAME] 写入者分配记录: 无(该用户指针无记录 ⇒ 记录表曾扩容/截断,或该块非经 alloc 得)\n")
# [FREEFRAME] Dump the writer's payload head/tail to separate an object
# overflow from a wild pointer landing on this block's tail (see
# _ff_owner_dump).
self._ff_owner_dump(rawptr(l_addr), o)
# [FREEFRAME] Evidence-based left-neighbour attribution (geometry plus
# canary alone cannot attribute; criteria in _ff_left_classify).
self._ff_left_classify(l_addr, bs_o, lcan, w0)
else:
stdio.printf("[FREEFRAME] 阶 o=%d L=%llu hdr=0x%llX 用户ptr=%llu canary=0x%llx(完好:不是它的连续越界写)\n",
o, l_addr, lhdr, l_addr + MEMBUDDY_HEADER_SIZE,
lcan)
o += 1
if found_l == 0:
stdio.printf("[FREEFRAME] 无候选:左邻不存在「块头奇数且阶与几何一致」的分配块 ⇒ **不能**用左邻几何归因(写入者可能是野指针落点,或来自更远处,需另证)\n")
# Nearby records: which objects were allocated in this area (detection: ptr = user + 2).
if self._det_records is not None:
stdio.printf("[FREEFRAME] 记录表近邻(|ptr-A| <= 256): ")
printed: int = 0
ri2: sizet = 0
while ri2 < self._det_count and printed < 8:
rp2: uint64 = self._det_records[ri2 * 2]
rs2: uint64 = self._det_records[ri2 * 2 + 1]
if rp2 >= uint64(block) - 256 and rp2 <= uint64(block) + 256:
# Single load + explicit bit test (same source as the writer-record line).
n_sz: uint64 = rs2 & ((uint64(1) << 63) - 1)
n_fr: int = 0
if (rs2 & (uint64(1) << 63)) != 0:
n_fr = 1
stdio.printf("[%zu]ptr=%llu sz=%llu freemark=%d ",
ri2, rp2, n_sz, n_fr)
printed += 1
ri2 += 1
stdio.printf("\n")
stdio.fflush(None)
def _ff_owner_dump(self, block: rawptr, order: int):
"""[FREEFRAME] Dump the writer (allocation) block's payload head/tail (16 bytes each) and
its requested size.
Purpose (separate the two readings of an overflow at the report level):
* payload head looks like an initialised object (pointers/strings/non-zero fields) while
the block capacity is far below the write span => an object larger than its block ran
over into the right neighbour;
* payload head/tail still hold the alloc fill (0xCD...) or zeros => more likely a wild
pointer landing exactly on this block's tail.
Also looks up this block's alloc requested size (= bytes the owner may write), same source
as _ff_report's writer-record line.
Boundary: a block outside / larger than the pool is never read (the dump path must not add
out-of-bounds reads).
"""
if self.detection == 0:
return
if self.usable is None:
return
if order < 0 or order > MEMBUDDY_MAX_ORDERS:
return
big: sizet = MEMBUDDY_MIN_BLOCK << sizet(order)
if uint64(block) < uint64(self.usable):
return
if uint64(block) + big > uint64(self.usable) + self.usable_size:
return
# Payload head 16B (start = block+8; in detection the first 2 bytes are the front marker).
head_addr: uint64 = uint64(block) + MEMBUDDY_HEADER_SIZE
# Payload tail 16B (16 bytes before the canary).
tail_addr: uint64 = uint64(block) + big - MEMBUDDY_CANARY - 16
stdio.printf("[FREEFRAME] 写入者 payload 头16B(=%llu): ", head_addr)
for bi_h in range(16):
bv_h: uint8 = *((uint8*)(head_addr + uint64(bi_h)))
stdio.printf("%02X ", uint32(bv_h))
stdio.printf("| 尾16B(=%llu): ", tail_addr)
for bi_t in range(16):
bv_t: uint8 = *((uint8*)(tail_addr + uint64(bi_t)))
stdio.printf("%02X ", uint32(bv_t))
stdio.printf("\n")
if self._det_records is not None:
usr_d: uint64 = uint64(block) + MEMBUDDY_HEADER_SIZE
if self.detection != 0:
usr_d = usr_d + 2
hit_d: int = 0
for ri_d in range(self._det_count):
if uint64(self._det_records[ri_d * 2]) == usr_d:
w_d: uint64 = self._det_records[ri_d * 2 + 1]
d_sz: uint64 = w_d & ((uint64(1) << 63) - 1)
d_fr: int = 0
if (w_d & (uint64(1) << 63)) != 0:
d_fr = 1
stdio.printf("[FREEFRAME] 写入者请求尺寸: [%zu] sz=%llu freemark=%d 原始字=%llu;本块可写=%zu 字节(bs=%zu − 头%zu − canary%zu)\n",
ri_d, d_sz, d_fr, w_d,
big - MEMBUDDY_HEADER_SIZE - MEMBUDDY_CANARY,
big, sizet(MEMBUDDY_HEADER_SIZE),
sizet(MEMBUDDY_CANARY))
hit_d += 1
if hit_d == 0:
stdio.printf("[FREEFRAME] 写入者请求尺寸: 无记录(记录表曾扩容/截断,或该块非经 alloc 得)\n")
stdio.fflush(None)
def _ff_addr_in_pool(self, a: uint64) -> int:
"""Whether a lies in this pool's range [usable, usable + usable_size). Predicate only."""
if self.usable is None:
return 0
if a < uint64(self.usable):
return 0
if a >= uint64(self.usable) + self.usable_size:
return 0
return 1
def _ff_left_classify(self, l_addr: uint64, l_bs: sizet,
lcan: uint64, w0: uint64):
"""[FREEFRAME] Evidence-based attribution of the left neighbour (geometry plus a damaged
canary alone cannot attribute).
Four measured criteria (read only L's bytes; L's payload word i = L + 8 + 8i):
1. canary == A.word0: the same 8-byte value in adjacent words => the write advanced in
8-byte units;
2. whether L's payload tail 16B is all zero: non-zero => the writer started inside the
payload; all zero => the writer never reached offsets 32..47, excluding a run growing
out of the payload tail;
3. how many payload words equal the written value (= canary): 0 => the written value is
unrelated to the payload (the write started at the canary);
4. count of in-pool addresses / out-of-pool non-zero words in L's payload: this pool's
alloc only writes in-pool addresses or 0, so an out-of-pool non-zero word means the
payload was not produced by "store a pointer per 8-byte slot" (write width != request
width).
A record only proves the block was allocated at that site; its sz is the REQUESTED size,
not the written range. Output states measured values and the value-implied conclusion
(A/B/C) only, never an unconditional attribution.
"""
nw: sizet = (l_bs - MEMBUDDY_HEADER_SIZE - MEMBUDDY_CANARY) / 8
tail_zero: int = 1
if nw >= 2:
w_a: uint64 = self._ff_word(rawptr(l_addr), int(nw) - 1)
w_b: uint64 = self._ff_word(rawptr(l_addr), int(nw))
if w_a != 0 or w_b != 0:
tail_zero = 0
hit_v: sizet = 0
in_pool: sizet = 0
out_pool: sizet = 0
for i in range(nw):
wv: uint64 = self._ff_word(rawptr(l_addr), int(i) + 1)
if wv == lcan:
hit_v += 1
if wv != 0:
if self._ff_addr_in_pool(wv) != 0:
in_pool += 1
else:
out_pool += 1
can_eq_w0: int = 0
if lcan == w0:
can_eq_w0 = 1
stdio.printf("[FREEFRAME] 左邻归属判据(实测): canary==A.word0=%d;L payload 尾16B全零=%d;写入值(=canary)在 L payload 出现=%zu 字;L payload 非零字 池内=%zu 池外=%zu(payload 共 %zu 字,起点 L+%zu)\n",
can_eq_w0, tail_zero, hit_v, in_pool, out_pool, nw,
sizet(MEMBUDDY_HEADER_SIZE))
case_n: int = 0
if tail_zero != 0 and can_eq_w0 != 0 and hit_v == 0:
case_n = 1
if out_pool != 0:
case_n = case_n + 2
if case_n == 1 or case_n == 3:
stdio.printf("[FREEFRAME] ⇒ 结论A:写入按 8 字节字推进(canary 与 A.word0 同值)且起点 ≤ L+%zu、**跳过**偏移 32..47 ⇒ 该次写**不是**「从 L 的 payload 尾部连续溢出」;若 L 的所有者确实越界,其写入宽度必然 > 申请尺寸 sz(对照上方「写入者请求尺寸」行)\n",
l_bs - MEMBUDDY_CANARY)
if case_n == 2 or case_n == 3:
stdio.printf("[FREEFRAME] ⇒ 结论B:L 的 payload 含**池外非零字**(本池 alloc 只产生池内地址或 0)⇒ 该 payload 不是「按 8 字节槽写元素指针」的结果 ⇒ 记录项的 sz 只是**申请**尺寸、不是写入范围:写入按更宽的结构体宽度跨字进行(或 payload 被第三方覆盖)⇒ 归属不能只看几何\n")
if case_n == 0:
stdio.printf("[FREEFRAME] ⇒ 结论C:可区分特征均未命中 ⇒ 不据「几何 + canary 损坏」断言归属(需结合记录表现场与 payload 全量转储另证)\n")
def _ff_verify(self, order: int, block: rawptr):
"""[FREEFRAME] Verify the free-block stamps before popping (detection only, caller-checked).
A stamp/mirror mismatch means the block's words were written off-list after linking, so
dump the scene and terminate at once -- earlier and more precise than "is next a valid
block address" (a writer that happened to write a valid 8-byte block address would pass
the old guard, not this one).
"""
# Range/alignment first: _fl_find_and_remove's head value has no range check, so if the
# slot itself was overwritten with an out-of-pool value, reading words 1..3 would read out
# of the pool (reading word0 is existing behaviour; the extra reads must not widen the
# boundary) => never verify a non-in-pool, unaligned address.
if self._fl_block_ok(block) == 0:
return
w0: uint64 = self._ff_word(block, 0)
w1: uint64 = self._ff_word(block, 1)
w2: uint64 = self._ff_word(block, 2)
w3: uint64 = self._ff_word(block, 3)
if w1 == self._ff_stamp(order) and w2 == uint64(MEMBUDDY_FREEFRAME_TAG2) and w3 == w0:
return
stdio.printf("[FREEFRAME] 空闲块元数据被写穿: order=%d block=%llu w0(next)=%llu w1=%llu(期望=0x%llx) w2=%llu(期望=0x%llx) w3(next镜像)=%llu(入链后被链外写入,立即转储)\n",
order, uint64(block), w0, w1, self._ff_stamp(order), w2,
uint64(MEMBUDDY_FREEFRAME_TAG2), w3)
self._ff_report(order, block)
stdio.fflush(None)
stdlib.exit(1)
def _ff_owner_check(self, block: rawptr, order: int):
"""[FREEFRAME] Pre-free owner check (detection only): verify the block-tail canary.
A contiguous overflow writer always overwrites its own block-tail canary before entering
the right neighbour's first word (the neighbour header is 8 bytes past the canary), so a
damaged canary means this block's owner is that writer. The report gives order / user
pointer / legal write length and dumps 24 bytes from the canary. Called from free's
live-block branch and reset_to's reclaimed blocks (both owner-known release points).
"""
if self.detection == 0:
return
bs: sizet = MEMBUDDY_MIN_BLOCK << sizet(order)
# Range pre-check: reset_to passes blocks straight from chk_stack (no pool-range check), so
# an out-of-range value would make the canary read land outside the pool; such cases are not
# reported (no attribution).
if self.usable is None:
return
if uint64(block) < uint64(self.usable):
return
if uint64(block) + bs > uint64(self.usable) + self.usable_size:
return
can_addr: uint64 = uint64(block) + bs - MEMBUDDY_CANARY
cp: uint64* = (uint64*)(can_addr)
can_val: uint64 = cp[0]
if can_val == uint64(MEMBUDDY_CANARY_PATTERN):
return
stdio.printf("[FREEFRAME] 释放前 canary 损坏(本块所有者**越过了本块可用区**;写入宽度见下方 payload 转储,`sz` 是申请尺寸、不是写入范围): block=%llu order=%d 用户ptr=%llu 合法写长=%zu canary=0x%llx 期望=0x%llx\n",
uint64(block), order, uint64(block) + MEMBUDDY_HEADER_SIZE,
bs - MEMBUDDY_HEADER_SIZE - MEMBUDDY_CANARY, can_val,
uint64(MEMBUDDY_CANARY_PATTERN))
# 24 bytes from the canary (8 = canary only, 16 = plus the neighbour header, 24 = plus its
# second word); a pool-tail block dumps only the in-pool part (no out-of-pool read).
dump_n: int = 8
if uint64(block) + bs + 16 <= uint64(self.usable) + self.usable_size:
dump_n = 24
stdio.printf("[FREEFRAME] canary 起 %d 字节: ", dump_n)
for bi in range(dump_n):
bv: uint8 = *((uint8*)(can_addr + uint64(bi)))
stdio.printf("%02X ", uint32(bv))
stdio.printf("\n")
# [FREEFRAME] Writer payload head/tail + requested size (same scene as the canary dump:
# a broken canary means this block's owner overflowed into the right free block's first word).
self._ff_owner_dump(block, order)
stdio.fflush(None)
stdlib.exit(1)
def _fl_pop(self, order: int) -> rawptr:
"""Pop the head of order `order`'s free list (O(1): no loop/recursion in this body).
Value validation (fail-fast): this function has only two memory accesses -- read
free_lists[order], then dereference that value as block[0]. Without validation a written-
through free_lists[order] (e.g. 0xFFFF...FF) makes the second access "dereference an
arbitrary address" (ASAN: access-violation on unknown address 0xffffffffffffffff), far from
the real writer. So validate that order is within capacity, then that the head and next
pointers are in-pool and aligned; otherwise print the scene and terminate -- allocator
corruption must be loud. No silent skip/downgrade: that would disguise "heap corrupted" as
"alloc failed" and shift the failure point along the call chain.
"""
if order < 0 or order > MEMBUDDY_MAX_ORDERS:
stdio.printf("[MEM] _fl_pop 调用者=_split_to_order order 越界: order=%d 容量=%d MAX_ORDERS=%d max_order=%d\n",
order, MEMBUDDY_MAX_ORDERS + 1, MEMBUDDY_MAX_ORDERS, self.max_order)
stdio.fflush(None)
stdlib.exit(1)
# Structural invariant (written once in __init__, never after): the free_lists array base
# == usable - 264 (264 = (MEMBUDDY_MAX_ORDERS+1)*8). Breaking it means the MemBuddy OBJECT
# itself was written through (not pool data), so the address of self.free_lists[order] is
# itself wild (e.g. free_lists=-1, order=0 => the first access is 0xffffffffffffffff). Must
# be rejected before ANY free_lists[...] access, or even the guard is too late.
if self.usable is not None and uint64(self.free_lists) != uint64(self.usable) - (MEMBUDDY_MAX_ORDERS + 1) * 8:
stdio.printf("[MEM] _fl_pop 调用者=_split_to_order free_lists 基址非法(MemBuddy 对象被写穿): free_lists=%llu usable=%llu max_order=%d\n",
uint64(self.free_lists), uint64(self.usable), self.max_order)
stdio.fflush(None)
stdlib.exit(1)
head_val: uint64 = self.free_lists[order]
if head_val == 0:
return None
if self._fl_block_ok(rawptr(head_val)) == 0:
stdio.printf("[MEM] _fl_pop 调用者=_split_to_order 链表头非法: order=%d head=%llu max_order=%d usable_size=%zu(free_lists[order] 被写穿:非池内对齐块地址)\n",
order, head_val, self.max_order, self.usable_size)
stdio.fflush(None)
stdlib.exit(1)
block: rawptr = rawptr(head_val)
# [FREEFRAME] detection mode: verify the free-block stamps/mirror before popping (earlier
# and more sensitive than the "is next a valid block address" guard below, which a writer
# that happened to write a valid 8-byte block address would pass). With detection == 0 it
# is one predictable non-taken branch, no extra memory access.
if self.detection != 0:
self._ff_verify(order, block)
next_ptr: rawptr = rawptr(*((uint64*)(block)))
next_val: uint64 = uint64(next_ptr)
if next_val != 0 and self._fl_block_ok(rawptr(next_val)) == 0:
stdio.printf("[MEM] _fl_pop 调用者=_split_to_order next 指针非法: order=%d block=%llu next=%llu max_order=%d usable_size=%zu(空闲块首字被写穿:next 非池内对齐块地址)\n",
order, head_val, next_val, self.max_order, self.usable_size)
stdio.fflush(None)
# [FREEFRAME] In detection mode add a forensic dump (left-neighbour probe + nearby
# records) to pin down which allocation wrote over the block.
if self.detection != 0:
self._ff_report(order, block)
stdlib.exit(1)
self.free_lists[order] = next_val
return block
def _fl_find_and_remove(self, order: int, target: rawptr) -> int:
head_val: uint64 = self.free_lists[order]
if head_val == 0:
return 0
head: rawptr = rawptr(head_val)
# [FREEFRAME] detection mode: this function walks the chain (following next), so a
# written-through node would send the walk into a wild chain (a cycle/over-limit report
# only says "heap is broken", naming nothing). Verify the head and every visited node, so
# a hit dumps the scene and terminates, turning a drifting walk into a named block.
if self.detection != 0:
self._ff_verify(order, head)
if uint64(head) == uint64(target):
next_ptr: rawptr = rawptr(*((uint64*)(head)))
self.free_lists[order] = uint64(next_ptr)
return 1
prev: rawptr = head
cur: rawptr = rawptr(*((uint64*)(head)))
# Cycle guard (same bound as self_check): a corrupted next pointer can make the free list
# cyclic, and without a step bound this loop would never end (free/coalesce hang with a
# drifting crash). The bound is "all minimum blocks + 2", which no valid free count can
# exceed; over the bound the heap is damaged: print the scene and stop (return 0 = not
# found) instead of spinning.
max_steps: sizet = self.usable_size / MEMBUDDY_MIN_BLOCK + 2
steps: sizet = 0
while uint64(cur) != 0:
steps += 1
if steps > max_steps:
stdio.printf("[MEM] free_lists[%d] 链成环或节点数超上限(next 指针被破坏): head=%llu target=%llu;停止遍历避免自旋,堆已损坏\n",
order, head_val, uint64(target))
return 0
# [FREEFRAME] Stamp check of the visited node (detection): the next hop comes from
# cur's word0, so if cur was written through, name it here and terminate (otherwise
# the walk drifts along a wild chain).
if self.detection != 0:
self._ff_verify(order, cur)
if uint64(cur) == uint64(target):
next_ptr: rawptr = rawptr(*((uint64*)(cur)))
*((uint64*)(prev)) = uint64(next_ptr)
# [FREEFRAME] Middle unlink rewrote prev's word0 (next), so its mirror word3 must
# be synced too; otherwise _ff_verify would misreport memhub's own valid unlink as
# "written through". prev's order is this list's order (the stamp written by
# _fl_push).
if self.detection != 0:
self._ff_word_set(prev, 3, uint64(next_ptr))
return 1
prev = cur
cur = rawptr(*((uint64*)(cur)))
return 0
# === Buddy system core ===
def _buddy_of(self, block: rawptr, order: int) -> rawptr:
# Buddy = block with the order's size bit flipped in its pool byte offset.
offset: sizet = uint64(block) - uint64(self.usable)
bs: sizet = MEMBUDDY_MIN_BLOCK << sizet(order)
buddy_offset: sizet = offset ^ bs
return self.usable + buddy_offset
def _order_for_size(self, size: sizet) -> int:
"""Smallest order with MEMBUDDY_MIN_BLOCK << order >= size.
Upper bound (termination): for a huge size (near 2^64 after corruption/overflow) `bs << 1`
would wrap to 0 and `0 < size` would stay true forever; order (i32) would also go negative
(undefined in `MIN_BLOCK << order`). The bound is semantics-preserving: only order <=
max_order is meaningful, so anything larger returns max_order + 1, which both callers
(alloc's `order <= self.max_order`, realloc's `new_order <= old_order`) reject as "too big"
exactly as before.
"""
order: int = 0
bs: sizet = MEMBUDDY_MIN_BLOCK
while bs < size and order <= self.max_order:
bs = bs << 1
order += 1
return order
def _split_to_order(self, to_order: int) -> rawptr:
# max_order must be in range before it indexes free_lists (a corrupt field would
# otherwise read out of bounds and feed garbage to _fl_pop). fail-fast.
if self.max_order < 0 or self.max_order > MEMBUDDY_MAX_ORDERS:
stdio.printf("[MEM] _split_to_order max_order 非法(字段被写穿): to_order=%d max_order=%d 容量=%d MAX_ORDERS=%d usable_size=%zu\n",
to_order, self.max_order, MEMBUDDY_MAX_ORDERS + 1, MEMBUDDY_MAX_ORDERS,
self.usable_size)
stdio.fflush(None)
stdlib.exit(1)
found_order: int = to_order
while found_order <= self.max_order:
if self.free_lists[found_order] != 0:
break
found_order += 1
if found_order > self.max_order:
return None
block: rawptr = self._fl_pop(found_order)
while found_order > to_order:
# Split the block in half, pushing the upper half back onto the free list.
found_order -= 1
bs: sizet = MEMBUDDY_MIN_BLOCK << sizet(found_order)
buddy: rawptr = block + bs
self._fl_push(found_order, buddy)
return block
def _coalesce(self, block: rawptr, order: int):
# Value validation (fail-fast, same rule as _fl_pop/_split_to_order): the first loop
# indexes self.free_lists[order] directly, and order comes from the header `stored >> 1`
# (free/reset_to paths) -- a written-through header yields an arbitrary index => out-of-
# bounds free_lists access. Valid callers (free via _is_valid_ptr, reset_to with an
# explicit bound) already guarantee order in [0, max_order], so behaviour changes only on
# corruption.
if order < 0 or order > MEMBUDDY_MAX_ORDERS:
stdio.printf("[MEM] _coalesce order 非法(块头被写穿): order=%d 容量=%d MAX_ORDERS=%d max_order=%d block=%llu\n",
order, MEMBUDDY_MAX_ORDERS + 1, MEMBUDDY_MAX_ORDERS,
self.max_order, uint64(block))
stdio.fflush(None)
stdlib.exit(1)
while order < self.max_order:
# Early exit (no semantic change, just skips two doomed calls): an empty free list at
# this order makes _fl_find_and_remove return 0 immediately, so coalescing is
# impossible; checking first saves this round's _buddy_of and _fl_find_and_remove.
# The test is identical to _fl_find_and_remove's entry condition and neither function
# has side effects, so the free-list semantics and order are fully equivalent.
if self.free_lists[order] == 0:
break
buddy: rawptr = self._buddy_of(block, order)
found: int = self._fl_find_and_remove(order, buddy)
if found == 0:
break
if uint64(buddy) < uint64(block):
block = buddy
order += 1
self._fl_push(order, block)
def _is_valid_ptr(self, ptr: rawptr) -> int:
if ptr is None: return 0
if self.usable is None: return 0
block: rawptr = ptr - MEMBUDDY_HEADER_SIZE
if uint64(block) < uint64(self.usable): return 0
if uint64(block) >= uint64(self.usable) + self.usable_size: return 0
offset: sizet = uint64(block) - uint64(self.usable)
if offset % MEMBUDDY_MIN_BLOCK != 0: return 0
stored: rawptr = rawptr(*((uint64*)(block)))
stored_val: uint64 = uint64(stored)
if (stored_val & 1) == 0: return 0
order: int = int(stored_val >> 1)
if order < 0: return 0
if order > self.max_order: return 0
# Live-set check: every alloc pushes its block onto chk_stack, and reset_to(mark)
# pops [mark, chk_depth) and frees those blocks. A stale pointer with a still-odd
# header would pass the checks above, so require the block to sit in the live
# window [0, chk_depth). Once chk_stack is full (chk_depth == chk_cap) new blocks
# are no longer recorded, so the check is skipped to avoid rejecting valid frees.
if self.chk_stack is not None and self.chk_depth < self.chk_cap:
alive: int = 0
for ci in range(self.chk_depth):
if self.chk_stack[ci] == uint64(block):
alive = 1
break
if alive == 0:
return 0
return 1
# === Spinlock ===
def _lock(self): # can use the spinlock library for decoupling
# With MEMBUDDY_THREAD_SAFE = 0 this if folds away and the body is empty; with 1 the atomic
# spinlock is kept as before.
if MEMBUDDY_THREAD_SAFE != 0:
while atom.__atomic_test_and_set(&self.lock_val, atom.ATOMIC_ACQUIRE):
pass
def _unlock(self):
if MEMBUDDY_THREAD_SAFE != 0:
atom.__atomic_clear(&self.lock_val, atom.ATOMIC_RELEASE)
# === Virtual function overrides ===
def alloc(self, size: sizet) -> rawptr:
self._lock()
result: rawptr = None
if size != 0 and self.usable is not None:
# OOM diagnostics: count requests >= 8KB so the report can tell "many small blocks" from
# "few large blocks"; threshold matches the field comment.
if size >= 8192:
self.big_alloc_count += 1
# +CANARY: reserve 8 sentinel bytes past the user area for overflow detection.
needed: sizet = size + MEMBUDDY_HEADER_SIZE + MEMBUDDY_CANARY
if self.detection != 0:
# [DET] 2 bytes of 0xCDCD on each side: 4 bytes more than requested.
needed = needed + 4
# Size-overflow guard (fail-fast): when size is near the u64 limit, `size + 16 (+4)`
# wraps to a tiny value => a smaller order => a block far smaller than requested is
# "successfully" returned and the caller overflows the neighbours. A valid request can
# never wrap, so terminate loudly here; never wrap, never silently return a small block,
# and leave the existing "too big -> None" contract (the `order <= self.max_order`
# branch below) untouched.
if needed < size:
stdio.printf("[MEM] alloc 调用者=上层 size 溢出: size=%zu needed=%zu(size+HEADER+CANARY 回绕,拒绝分配)\n",
size, needed)
stdio.fflush(None)
stdlib.exit(1)
order: int = self._order_for_size(needed)
if order <= self.max_order:
block: rawptr = self._split_to_order(order)
if block is not None:
*((uint64*)(block)) = uint64((order << 1) | 1) # header: odd = allocated
result = block + MEMBUDDY_HEADER_SIZE
bs: sizet = MEMBUDDY_MIN_BLOCK << sizet(order)
# Write the block-tail canary (just past the user area).
*((uint64*)(block + bs - MEMBUDDY_CANARY)) = uint64(MEMBUDDY_CANARY_PATTERN)
if MEMBUDDY_FILL_GUARD != 0:
# Diagnostic fill: uninitialised fields read back as 0xCD.
fill_size: sizet = bs - MEMBUDDY_HEADER_SIZE - MEMBUDDY_CANARY
if fill_size > 0:
string.memset(result, MEMBUDDY_FILL_PATTERN, fill_size)
# Record on the checkpoint stack (for reset_to reclamation). When full, double
# it with stdlib.realloc (a full stack would stop recording -> reset misses
# blocks and _is_valid_ptr's stack check degrades).
#
# Lazily build the record stack (also records before the first mark()); else blocks
# allocated before it are missing from chk_stack and their free is permanently rejected.
if self.chk_stack is None:
self.mark()
if self.chk_stack is not None:
if self.chk_depth >= self.chk_cap:
new_cap: sizet = self.chk_cap * 2
new_stack: rawptr = rawptr(stdlib.realloc(self.chk_base, new_cap * 8))
if new_stack is not None:
self.chk_base = new_stack
self.chk_stack = (uint64*)(new_stack)
self.chk_cap = new_cap
else:
# realloc failed: the block is not recorded -> reset_to cannot
# reclaim it -> a gradual leak ending in 1 GB main-pool OOM.
# fail-fast instead of silent.
stdio.printf(
"[MEM] 记录栈 realloc 失败 cap=%zu depth=%zu(reset 将失效,终止)\n",
self.chk_cap, self.chk_depth)
if self.chk_depth < self.chk_cap:
self.chk_stack[self.chk_depth] = uint64(block)
self.chk_depth += 1
if result is not None and self._det_records is not None:
base_ret_d: uint64 = uint64(result)
# Returned pointer (in detection layout = user area + 2, past the front marker).
rec_ptr_d: uint64 = base_ret_d
if self.detection != 0:
# [DET] Layout [0xCDCD][user][0xCDCD]: returned ptr = user area + 2. Record
# {ptr, size} and stamp both markers; a broken marker = out-of-bounds write.
m_f_d: uint16* = (uint16*)(base_ret_d)
m_f_d[0] = uint16(MEMBUDDY_DET_MARKER)
tail_d: uint64 = base_ret_d + 2 + uint64(size)
m_t_d: uint16* = (uint16*)(tail_d)
m_t_d[0] = uint16(MEMBUDDY_DET_MARKER)
rec_ptr_d = base_ret_d + 2
# Sampling: run the O(n) full-table scan every MEMBUDDY_DET_PROBE_PERIOD allocs
# (allocs otherwise only append a record, O(1)); corruption is persistent so a
# delayed scan still finds it. Gated by the compile-time switch (default 0 =
# never run) and by deep_detect (--detection-deep = scan on every alloc).
if MEMBUDDY_PERIOD_SCAN != 0 or self.deep_detect != 0:
if (self._det_count & (MEMBUDDY_DET_PROBE_PERIOD - 1)) == 0 or self.deep_detect != 0:
# Overlap check: the new block's address must not already be an active
# record (same block allocated twice = free list corruption).
ff_d: uint64 = uint64(1) << 63
for dup_i in range(self._det_count):
drp: uint64 = self._det_records[dup_i * 2]
drs: uint64 = self._det_records[dup_i * 2 + 1]
if drp == rec_ptr_d and (drs & ff_d) == 0:
stdio.printf("[DET] 分配重叠: 新alloc(size=%llu)与记录[%zu](sz=%llu)同地址 ptr=%llu(同一块重复分配,free list 损坏)\n",
uint64(size), dup_i, drs & (ff_d - 1), rec_ptr_d)
stdio.fflush(None)
stdlib.exit(1)
if _memdbg_enable != 0:
# [MEMDBG] A freshly popped block must not still be on its own free
# list (a duplicate put would leave a residual node -> reallocated).
dbg_block: rawptr = result - MEMBUDDY_HEADER_SIZE
dbg_hdr: rawptr = rawptr(*((uint64*)(dbg_block)))
dbg_order: int = int(uint64(dbg_hdr) >> 1)
# dbg_order comes from the block header; if the header were corrupted
# (the very case this branch hunts), indexing free_lists[dbg_order]
# would read out of bounds and feed garbage to a deref. fail-fast.
if dbg_order < 0 or dbg_order > MEMBUDDY_MAX_ORDERS:
stdio.printf("[MEMDBG] alloc dbg_order 非法(块头被写穿): hdr=0x%llX dbg_order=%d 容量=%d block=%llu(终止)\n",
uint64(dbg_hdr), dbg_order, MEMBUDDY_MAX_ORDERS + 1,
uint64(dbg_block))
stdio.fflush(None)
stdlib.exit(1)
dbg_cur: rawptr = rawptr(self.free_lists[dbg_order])
dbg_n: sizet = 0
while uint64(dbg_cur) != 0:
if uint64(dbg_cur) == uint64(dbg_block):
stdio.printf("[MEMDBG] alloc 重复出块: size=%llu block=%llu 仍残留 free list order=%d(重复 put 实锤,终止)\n",
uint64(size), uint64(dbg_block), dbg_order)
stdio.fflush(None)
stdlib.exit(1)
dbg_n += 1
if dbg_n > self.usable_size / MEMBUDDY_MIN_BLOCK + 2:
stdio.printf("[MEMDBG] alloc 扫描 free list order=%d 成环/超长(自检异常,终止)\n", dbg_order)
stdio.fflush(None)
stdlib.exit(1)
dbg_cur = rawptr(*((uint64*)(dbg_cur)))
if self._det_count >= self._det_cap:
# Record table full: double it so is_alive/_det_probe never miss live blocks.
new_cap: sizet = self._det_cap * 2
new_rec: rawptr = rawptr(stdlib.realloc(self._det_arena, new_cap * 16))
if new_rec is not None:
self._det_arena = new_rec
self._det_records = (uint64*)(new_rec)
self._det_cap = new_cap
if self._det_count < self._det_cap:
self._det_records[self._det_count * 2] = rec_ptr_d
self._det_records[self._det_count * 2 + 1] = uint64(size)
self._det_count += 1
# Probe decision: never entered when detection is off; with it on, deep_detect
# probes every alloc, otherwise the historical sampling is used.
if self.detection != 0 and ((self._det_count & (MEMBUDDY_DET_PROBE_PERIOD - 1)) == 0 or self.deep_detect != 0):
self._det_probe()
if self.detection != 0:
result = result + 2
if result is None:
# Allocation-failure diagnostics. A zero size and "pool exhausted" are different
# cases: the entry `size != 0` guard skips the former and returns NULL. A zero
# size almost always means the caller's size expression evaluated to 0
# (unresolved or shadowed constant). Never exit here: alloc failures return None
# for the caller to handle (tests rely on free-and-retry after OOM).
if size == 0:
stdio.printf("[MEM] alloc 请求尺寸为 0:返回 NULL(非池耗尽!)——"
"请检查调用处尺寸表达式是否被静默求值为 0"
"(常量未解析/被同名符号遮蔽/被算成 0)\n")
# Fields below are read from existing state only (this branch runs only on
# failure): site tag, pool alloc sequence, zero-size hit count, pool state,
# and the stage/detail context set by TPV.
usable_addr: uint64 = 0
if self.usable is not None:
usable_addr = uint64(self.usable)
marked: int = 0
if self.chk_stack is not None:
marked = 1
zero_hits: sizet = _zero_alloc_note()
stdio.printf("[MEM] site=%s ctx=%s/%s marked=%d 本池分配序号=%zu 0尺寸累计=%zu "
"池基址=%llu 池容量=%zu 已用=%zu 空闲=%zu "
"max_order=%d detection=%d deep=%d\n",
_det_site_name(_alloc_site_current()),
_alloc_ctx_stage_str(), _alloc_ctx_detail_str(), marked,
self.chk_depth,
zero_hits, usable_addr, self.usable_size, self.used_bytes(),
self.free_bytes(), self.max_order,
self.detection, self.deep_detect)
stdio.fflush(None)
elif self.usable is not None:
needed2: sizet = size + MEMBUDDY_HEADER_SIZE + MEMBUDDY_CANARY
order2: int = self._order_for_size(needed2)
stdio.printf("[MEM] alloc 失败 size=%zu needed=%zu order=%d max_order=%d used=%zu pool_base=%llu\n",
size, needed2, order2, self.max_order, self.used_bytes(),
uint64(self.usable))
# Attribution (always-on, no --detection needed): site comes from the alloc_tagged site
# tag, ctx=stage/current file from TPV's SetAllocCtx (? if unset). Failure path only.
stdio.printf("[MEM] 归属 site=%s ctx=%s/%s 池容量=%zu\n",
_det_site_name(_alloc_site_current()),
_alloc_ctx_stage_str(), _alloc_ctx_detail_str(),
self.usable_size)
# Window-start free bytes (recorded at mark()): ~pool size => this window filled the
# pool; ~0 => already full before the window (current file is only a victim).
win_start_free: sizet = self.mark_free_bytes
win_now_free: sizet = self.free_bytes()
win_net: sizet = 0
if win_start_free > win_now_free:
win_net = win_start_free - win_now_free
stdio.printf("[MEM] 本窗口起点空闲=%zu 当前空闲=%zu(窗口内净分配=%zu)\n",
win_start_free, win_now_free, win_net)
# Size profile: see which orders of live blocks filled the pool (read-only, no --detection).
self.dump_alive_orders()
# self_check != 0 means the free list structure is damaged (a block header or
# next pointer was written through), so alloc finds no block despite free space.
sc: int = self.self_check()
stdio.printf("[MEM] alloc 失败 self_check=%d free_bytes=%zu free_count=%zu chk_depth=%zu big=%zu\n",
sc, self.free_bytes(), self.free_count(), self.chk_depth, self.big_alloc_count)
# Huge chk_depth => allocation storm (per-statement/per-call); small chk_depth
# with huge used => repeated single large allocations (buffers/table growth).
if order2 > self.max_order:
stdio.printf("[MEM] 原因:请求 size 超 pool 最大块(order %d > max %d)\n",
order2, self.max_order)
else:
stdio.printf("[MEM] 原因:pool 耗尽(空闲 %zu 字节不足)\n", self.free_bytes())
# Failure (alloc(0) or exhausted): unlock and return None; the caller decides.
# Must not exit -- tests free a block and retry after an OOM.
self._unlock()
return None
self._unlock()
return result
def alloc_tagged(self, size: sizet, tag: sizet) -> rawptr:
"""alloc wrapper that records the caller's site tag for [FREEFRAME] / zero-size reports.
Same semantics as alloc, plus two registrations:
1. site tag (independent of detection): set _alloc_pending_site before forwarding and
restore it after, so alloc's zero-size / failure reports print site=file:line;
2. when detection != 0: after self.alloc(size), store (returned ptr, tag) into the
module-level side table _det_site_arena at index = the pre-call self._det_count.
If the record table is full and realloc fails, _det_count does not advance, nothing is
registered, and the report prints site=? instead of fabricating one.
"""
prev: uint64 = _alloc_site_enter(uint64(tag))
if self.detection == 0 or self._det_records is None or tag == 0:
r_plain: rawptr = self.alloc(size)
_alloc_site_leave(prev)
return r_plain
before: sizet = self._det_count
r: rawptr = self.alloc(size)
if r is not None and self._det_count == before + 1:
_det_site_put(before, uint64(r), uint64(tag))
_alloc_site_leave(prev)
return r
def free(self, ptr: rawptr) -> int:
self._lock()
if ptr is not None:
p_eff: rawptr = ptr
if self.detection != 0:
# [DET] Returned ptr is 2 bytes past the user area; step back on free.
p_eff = ptr - 2
# Mark the record free (independent of detection): otherwise a later alloc of the
# same address (normal reuse) would be misreported as an overlap.
if self._det_records is not None:
# Read this record's real size (detection fills 0xCD after free, so a later
# write-through breaks the fill = use-after-free). Keep the record and set the
# free bit; the probe checks the 0xCD fill of still-free blocks.
for ri in range(self._det_count):
rp: uint64 = self._det_records[ri * 2]
rs: uint64 = self._det_records[ri * 2 + 1]
if rp == uint64(ptr) and (rs & (uint64(1) << 63)) == 0:
if self.detection != 0 and rs > 0:
string.memset(ptr, MEMBUDDY_FILL_PATTERN, rs)
self._det_records[ri * 2 + 1] = rs | (uint64(1) << 63)
break
if self._is_valid_ptr(p_eff) != 0:
block: rawptr = p_eff - MEMBUDDY_HEADER_SIZE
stored: rawptr = rawptr(*((uint64*)(block)))
stored_val: uint64 = uint64(stored)
order: int = int(stored_val >> 1)
if (stored_val & 1) == 0:
# Double free (even header = already freed): silent no-op when debug is
# off (never coalesce again, to avoid a duplicate free-list put); with
# debug on, terminate and expose the stale/duplicate free call site.
if _memdbg_enable != 0:
stdio.printf("[MEMDBG] 悬空/重复 free: ptr=%llu block=%llu header=0x%llX(偶数=已释放/未初始化,终止)\n",
uint64(ptr), uint64(block), stored_val)
stdio.fflush(None)
stdlib.exit(1)
self._unlock()
return 1
# [FREEFRAME] Pre-free owner check (detection only): a corrupted block-tail
# canary means this block's owner is the writer that ran over into the right
# neighbour's first word (the neighbour header sits 8 bytes past the canary).
# Done before the [MEMDBG] scan: double-put and overflow are different faults.
self._ff_owner_check(block, order)
if _memdbg_enable != 0:
# [MEMDBG] Double-return check: a live block (odd header) must never
# already be on this order's free list; a hit = the same block put twice.
dbg_f: rawptr = rawptr(self.free_lists[order])
dbg_fn: sizet = 0
while uint64(dbg_f) != 0:
if uint64(dbg_f) == uint64(block):
stdio.printf("[MEMDBG] 重复归还: free(ptr=%llu) block=%llu 已在 free list order=%d(double-put 实锤,终止)\n",
uint64(ptr), uint64(block), order)
stdio.fflush(None)
stdlib.exit(1)
dbg_fn += 1
if dbg_fn > self.usable_size / MEMBUDDY_MIN_BLOCK + 2:
stdio.printf("[MEMDBG] free 扫描 free list order=%d 成环/超长(自检异常,终止)\n", order)
stdio.fflush(None)
stdlib.exit(1)
dbg_f = rawptr(*((uint64*)(dbg_f)))
*((uint64*)(block)) = 0
self._coalesce(block, order)
elif _memdbg_enable != 0:
# [MEMDBG] Invalid/stale free (previously silent): the block is inside this pool
# but the header is even = already freed / never initialised => stale free.
# Pointers outside the pool belong to another pool and are not reported.
dbg_eff: uint64 = uint64(p_eff)
dbg_blk: uint64 = dbg_eff - MEMBUDDY_HEADER_SIZE
if (dbg_eff >= MEMBUDDY_HEADER_SIZE and dbg_blk >= uint64(self.usable)
and dbg_blk < uint64(self.usable) + self.usable_size):
dbg_hv: uint64 = *((uint64*)(dbg_blk))
if (dbg_hv & 1) == 0:
stdio.printf("[MEMDBG] 悬空/重复 free: ptr=%llu block=%llu header=0x%llX(偶数=已释放/未初始化,终止)\n",
uint64(ptr), dbg_blk, dbg_hv)
stdio.fflush(None)
stdlib.exit(1)
else:
# _is_valid_ptr invalid = block never returns to a free list (silent pool leak); print once.
# Criteria: in_pool=0 not in pool, aligned=0 not 32B-aligned, hdr_odd=0 even header, in_win=0 outside window.
global _rej_free_reported
if _rej_free_reported == 0:
_rej_free_reported = 1
e_eff: uint64 = uint64(p_eff)
e_blk: uint64 = e_eff - MEMBUDDY_HEADER_SIZE
e_in_pool: int = 0
e_aligned: int = 0
e_hdr: uint64 = 0
e_hdr_odd: int = 0
if (e_eff >= MEMBUDDY_HEADER_SIZE and self.usable is not None
and e_blk >= uint64(self.usable)
and e_blk < uint64(self.usable) + self.usable_size):
e_in_pool = 1
if (e_blk - uint64(self.usable)) % MEMBUDDY_MIN_BLOCK == 0:
e_aligned = 1
e_hdr = *((uint64*)(e_blk))
if (e_hdr & 1) != 0:
e_hdr_odd = 1
e_in_win: int = 0
if self.chk_stack is not None and self.chk_depth < self.chk_cap:
e_ci: sizet = 0
while e_ci < self.chk_depth:
if self.chk_stack[e_ci] == e_blk:
e_in_win = 1
break
e_ci += 1
else:
e_in_win = -1 # record stack full (chk_depth==chk_cap) -> check skipped
stdio.printf("[MEM] free 被拒绝: ptr=%llu block=%llu 在池内=%d 对齐=%d "
"块头=0x%llX 块头奇数=%d 在记录窗口=%d chk_depth=%zu "
"本池 base=%llu usable=%llu usable_size=%zu\n",
e_eff, e_blk, e_in_pool, e_aligned, e_hdr, e_hdr_odd,
e_in_win, self.chk_depth,
uint64(self.base), uint64(self.usable), self.usable_size)
stdio.fflush(None)
self._unlock()
return 1
def _rebuild_free_lists(self):
"""Rebuild the free lists by binary decomposition (reset restores the initial layout).
Same decomposition as __init__: a non-power-of-two arena splits into power-of-two free
blocks (buddy coalescing naturally stops at decomposition boundaries because the orders
no longer match); a power-of-two arena is a single max_order block.
"""
i: int
for i in range(MEMBUDDY_MAX_ORDERS + 1):
self.free_lists[i] = 0
pos: sizet = 0
rem: sizet = self.usable_size
while rem >= MEMBUDDY_MIN_BLOCK:
blk: sizet = _largest_pow2_le(rem)
o: int = 0
ob: sizet = MEMBUDDY_MIN_BLOCK
while ob < blk:
ob = ob << 1
o += 1
self._fl_push(o, self.usable + pos)
pos += blk
rem -= blk
def reset(self) -> int:
if self.usable is None:
return 0
self._rebuild_free_lists()
# [DET] reset frees every block, so clear the record table too; otherwise a later
# alloc reusing an old block would be misreported (same reason as reset_to's truncation).
if self._det_records is not None:
self._det_count = 0
return 1
# realloc override: buddy system reads original order from block header; old_size is not required
def realloc(self, ptr: rawptr, new_size: sizet) -> rawptr:
if ptr is None:
return self.alloc(new_size)
if new_size == 0:
self.free(ptr)
return None
p_eff: rawptr = ptr
if self.detection != 0:
# [DET] Same as free: step back 2 bytes before block-level math.
p_eff = ptr - 2
if self._is_valid_ptr(p_eff) == 0:
return None
block: rawptr = p_eff - MEMBUDDY_HEADER_SIZE
stored: rawptr = rawptr(*((uint64*)(block)))
stored_val: uint64 = uint64(stored)
old_order: int = int(stored_val >> 1)
# Capacity must match alloc exactly (single source of truth): alloc's block capacity is
# `needed = size + HEADER + CANARY (+4 for detection markers)`, so the real writable
# bytes are bs-16. Without CANARY the in-place test would be one boundary too wide
# (new_size in (bs-16, bs-8] would return in place and let the caller overflow its own
# canary), so CANARY is required here for both sides to agree.
needed: sizet = new_size + MEMBUDDY_HEADER_SIZE + MEMBUDDY_CANARY
if self.detection != 0:
needed = needed + 4 # [DET] front/back markers
new_order: int = self._order_for_size(needed)
if new_order <= old_order:
return ptr
new_ptr: rawptr = self.alloc(new_size)
if new_ptr is None:
return ptr
old_block_size: sizet = _block_size_at_order(old_order)
old_data_size: sizet = old_block_size - MEMBUDDY_HEADER_SIZE
cp_len: sizet = old_data_size
if self.detection != 0:
cp_len = old_data_size - 4 # [DET] exclude the 2-byte front/back markers
string.memcpy(new_ptr, ptr, cp_len)
self.free(ptr)
return new_ptr
# ============================================================
# [DET] is_alive - read-side liveness check: is ptr an active record in
# the current record table (allocated, not freed)? 1 = alive / 0 = dangling.
# Returns 1 when detection is off or the table is unavailable (no blocking).
# Covers _det_probe's blind spot: write-side markers cannot catch use-after-free
# reads (a dangling pointer into a reused block).
# ============================================================
def is_alive(self, ptr: rawptr) -> int:
if self.detection == 0:
return 1
if self._det_records is None:
return 1
target: uint64 = uint64(ptr)
free_flag: uint64 = uint64(1) << 63
for i in range(self._det_count):
rp: uint64 = self._det_records[i * 2]
rs: uint64 = self._det_records[i * 2 + 1]
if rp == target and (rs & free_flag) == 0:
return 1
return 0
# ============================================================
# [DET] _det_probe - walk the record table and check every live block's tail
# marker (2 bytes at ptr + size), which must still read 0xCDCD. A broken
# marker = out-of-bounds write: report ptr/size and terminate, turning a
# random crash into a deterministic "which allocation overflowed".
# ============================================================
def _det_probe(self):
if self._det_records is None:
return
free_flag: uint64 = uint64(1) << 63
i: sizet = 0
while i < self._det_count:
p: uint64 = self._det_records[i * 2]
s: uint64 = self._det_records[i * 2 + 1]
if (s & free_flag) != 0:
# Free record: skip. Buddy coalescing/reallocation makes a freed block's memory
# non-exclusive (a merged-in neighbour can overwrite the old front marker while
# the old header stays 0), so marker checks would false-positive. Only active
# records are checked: a live block is exclusive (odd header => never merged),
# so its front/tail markers are reliable.
i += 1
continue
if s == 0:
i += 1
continue # no size recorded (defensive; should not happen)
# Active record: front marker (underflow) + tail marker (overflow).
m_fa: uint16* = (uint16*)(p - 2)
mfv_a: uint16 = m_fa[0]
if mfv_a != uint16(MEMBUDDY_DET_MARKER):
stdio.printf("[DET] 下溢写: ptr=%llu size=%llu 前标记=0x%04X(期望 0xCDCD)\n",
p, s, uint32(mfv_a))
stdio.fflush(None)
stdlib.exit(1)
m16: uint16* = (uint16*)(p + s)
mv: uint16 = m16[0]
if mv != uint16(MEMBUDDY_DET_MARKER):
stdio.printf("[DET] 越界写: ptr=%llu size=%llu 后标记=0x%04X(期望 0xCDCD)块内容前16B: ",
p, s, uint32(mv))
for bi3 in range(16):
bv3: uint8 = *((uint8*)(p + uint64(bi3)))
stdio.printf("%02X ", uint32(bv3))
# Overflow region (24 bytes from ptr+size): 0x00 = written through, 0xCD = not
# written, other = written value; shows how far/direction the writer went.
stdio.printf("越界区: ")
for bj3 in range(24):
bvj: uint8 = *((uint8*)(p + s + uint64(bj3)))
stdio.printf("%02X ", uint32(bvj))
# Record-table context: the 3 records before/after, to infer the phase.
stdio.printf("\n记录上下文: ")
ci2: int = -3
while ci2 <= 3:
idx2: sizet = sizet(int(i) + ci2)
if idx2 < self._det_count:
cip: uint64 = self._det_records[idx2 * 2]
cis: uint64 = self._det_records[idx2 * 2 + 1]
stdio.printf("[%d]ptr=%llu sz=%llu ", ci2, cip, cis & (free_flag - 1))
ci2 += 1
# Overlap check: a ptr with several records means alloc handed the same block to
# different size requests (they overwrite each other).
stdio.printf("\n同ptr记录: ")
for ci3 in range(self._det_count):
cp3: uint64 = self._det_records[ci3 * 2]
cs3: uint64 = self._det_records[ci3 * 2 + 1]
if cp3 == p and (cs3 & free_flag) == 0:
stdio.printf("[%zu]sz=%llu ", ci3, cs3 & (free_flag - 1))
stdio.printf("\n")
stdio.fflush(None)
stdlib.exit(1)
i += 1
# === Status Queries ===
@property
def mem_size(self) -> sizet:
# Total usable region size (power of two). Maximum allocatable size = mem_size - MEMBUDDY_HEADER_SIZE
return self.usable_size
def stats(self) -> sizet:
# Return total usable region size, for status statistics
return self.usable_size
def free_count(self) -> sizet:
# Count blocks in all order lists for status statistics.
# Cycle guard (same bound as self_check): this is a diagnostic/OOM data path, so a cyclic
# free list must not hang the reporter; on overrun stop that order's walk and return the
# partial count.
count: sizet = 0
i: int
max_steps: sizet = self.usable_size / MEMBUDDY_MIN_BLOCK + 2
for i in range(MEMBUDDY_MAX_ORDERS + 1):
head_val: uint64 = self.free_lists[i]
cur: rawptr = rawptr(head_val)
if uint64(cur) != 0 and self._fl_block_ok(cur) == 0:
# Free-list head itself invalid (e.g. uninitialized 0xBE...): dereferencing would AV; report and stop this order.
stdio.printf("[MEM] free_lists[%d] 头非法(非池内对齐块地址,疑元数据未初始化/被写穿): "
"head=%llu chk_depth=%zu usable=%llu usable_size=%zu\n",
i, head_val, self.chk_depth, uint64(self.usable), self.usable_size)
stdio.fflush(None)
break
steps: sizet = 0
while uint64(cur) != 0:
steps += 1
if steps > max_steps:
stdio.printf("[MEM] free_lists[%d] 链成环或节点数超上限(next 指针被破坏): head=%llu;停止计数避免自旋,堆已损坏\n",
i, head_val)
break
count += 1
next_p: rawptr = rawptr(*((uint64*)(cur)))
if uint64(next_p) != 0 and self._fl_block_ok(next_p) == 0:
# Head/node invalid (uninitialized 0xBE... or non-pool-aligned): further dereference
# would AV, so just report and stop counting this order (metadata corruption proof).
stdio.printf("[MEM] free_lists[%d] 节点非法(非池内对齐块地址,疑元数据未初始化/被写穿): "
"head=%llu cur=%llu next=%llu chk_depth=%zu usable=%llu usable_size=%zu\n",
i, head_val, uint64(cur), uint64(next_p), self.chk_depth,
uint64(self.usable), self.usable_size)
stdio.fflush(None)
break
cur = next_p
return count
def free_bytes(self) -> sizet:
# Count bytes in all free blocks for status statistics.
# Cycle guard (same bound as self_check): same as free_count.
total: sizet = 0
i: int
max_steps: sizet = self.usable_size / MEMBUDDY_MIN_BLOCK + 2
for i in range(MEMBUDDY_MAX_ORDERS + 1):
head_val: uint64 = self.free_lists[i]
cur: rawptr = rawptr(head_val)
if uint64(cur) != 0 and self._fl_block_ok(cur) == 0:
# Free-list head itself invalid (e.g. uninitialized 0xBE...): dereferencing would AV; report and stop this order.
stdio.printf("[MEM] free_lists[%d] 头非法(非池内对齐块地址,疑元数据未初始化/被写穿): "
"head=%llu chk_depth=%zu usable=%llu usable_size=%zu\n",
i, head_val, self.chk_depth, uint64(self.usable), self.usable_size)
stdio.fflush(None)
break
steps: sizet = 0
while uint64(cur) != 0:
steps += 1
if steps > max_steps:
stdio.printf("[MEM] free_lists[%d] 链成环或节点数超上限(next 指针被破坏): head=%llu;停止统计避免自旋,堆已损坏\n",
i, head_val)
break
total += _block_size_at_order(i)
next_b: rawptr = rawptr(*((uint64*)(cur)))
if uint64(next_b) != 0 and self._fl_block_ok(next_b) == 0:
stdio.printf("[MEM] free_lists[%d] 节点非法(非池内对齐块地址,疑元数据未初始化/被写穿): "
"head=%llu cur=%llu next=%llu chk_depth=%zu usable=%llu usable_size=%zu\n",
i, head_val, uint64(cur), uint64(next_b), self.chk_depth,
uint64(self.usable), self.usable_size)
stdio.fflush(None)
break
cur = next_b
return total
def used_bytes(self) -> sizet:
# Used bytes = total usable region size - free bytes in all free blocks
return self.usable_size - self.free_bytes()
def dump_stats(self, label: char*):
# Print memory usage statistics
fb: sizet = self.free_bytes()
ub: sizet = self.usable_size - fb
stdio.printf("[MEM] %s: total=%zu used=%zu (%zu%%) free=%zu free_blocks=%zu\n",
label, self.usable_size, ub,
(ub * 100) / self.usable_size if self.usable_size > 0 else 0,
fb, self.free_count())
def dump_alive_orders(self):
"""OOM 归属画像:按 order 统计「记录在案且仍存活」的块,并给出整池账目。
数据全部来自既有状态,无需 --detection:
- `chk_stack[0, chk_depth)` 是 alloc 时压栈的块地址;
- 块头为 `(order << 1) | 1` 表示存活,偶数表示已被单独 free(跳过,与
reset_to / check_alive_blocks 同判据)。
块大小 = MEMBUDDY_MIN_BLOCK << order,故这份分布能直接区分
“大量小块” 与 “少量大块”,是定位池被谁吃满的关键画像。
整池账目(关键判据):池内每一字节必然属于「存活块」或「空闲块」二者之一,
故 `usable_size - Σ存活 - Σ空闲链表` 若显著不为 0,说明有块既不在空闲链表、
也没被任何记录项持有 —— 即空闲链表被写穿后**链被静默截断**(`_fl_pop` 允许
next==0 与池内任意对齐地址,截断不会报错),这些块从此再也分配不到:池漏满。
"""
if self.chk_stack is None or self.usable is None:
return
max_o: int = MEMBUDDY_MAX_ORDERS
counts: uint64* = (uint64*)(stdlib.malloc((max_o + 1) * 8))
if counts is None:
return
string.memset(counts, 0, (max_o + 1) * 8)
live: sizet = 0
alive_bytes: sizet = 0
bad: sizet = 0
pfx_live: sizet = 0
pfx_bytes_a: sizet = 0
win_live: sizet = 0
win_bytes_a: sizet = 0
pool_end: uint64 = uint64(self.usable) + self.usable_size
ci: sizet = 0
while ci < self.chk_depth:
block: rawptr = rawptr(self.chk_stack[ci])
ci += 1
if block is None:
bad += 1
continue
bv: uint64 = uint64(block)
if bv < uint64(self.usable) or bv >= pool_end:
bad += 1
continue
stored: rawptr = rawptr(*((uint64*)(block)))
stored_val: uint64 = uint64(stored)
if (stored_val & 1) == 0:
continue
order: int = int(stored_val >> 1)
if order < 0 or order > max_o:
bad += 1
continue
counts[order] = counts[order] + 1
bs_a: sizet = MEMBUDDY_MIN_BLOCK << sizet(order)
alive_bytes += bs_a
live += 1
if ci - 1 >= self.mark_depth:
win_live += 1
win_bytes_a += bs_a
else:
pfx_live += 1
pfx_bytes_a += bs_a
fb: sizet = self.free_bytes()
lost: sizet = 0
if self.usable_size > alive_bytes + fb:
lost = self.usable_size - alive_bytes - fb
stdio.printf("[MEM] 整池账目:镜像存活=%zu 块/%zu 字节 空闲链表=%zu 字节 "
"丢失=%zu 字节(丢失=既不在链表也不被持有 ⇒ 空闲链被写穿后截断)\n",
live, alive_bytes, fb, lost)
# Segmentation: only the part allocated after mark() is "this window"; the total would also
# include the historical prefix (mark1 runs once in Phase1 1b => prefix=1a, window=1b).
stdio.printf("[MEM] 分段(mark_depth=%zu chk_depth=%zu):前缀[0,mark)=%zu 块/%zu 字节 "
"本窗口[mark,end)=%zu 块/%zu 字节\n",
self.mark_depth, self.chk_depth, pfx_live, pfx_bytes_a,
win_live, win_bytes_a)
stdio.printf("[MEM] 存活块 order 分布(记录在案共 %zu 块;单块=%llu<<order 字节):\n",
live, uint64(MEMBUDDY_MIN_BLOCK))
for oi in range(max_o + 1):
cnt: uint64 = counts[oi]
if cnt != 0:
blk: uint64 = uint64(MEMBUDDY_MIN_BLOCK) << oi
stdio.printf("[MEM] order=%d 块=%llu 单块=%llu 合计=%llu\n",
oi, cnt, blk, cnt * blk)
if bad != 0:
stdio.printf("[MEM] 记录项异常(NULL/越界/块头 order 非法)共 %zu 条;"
"抽样 4 条(addr / 块头):\n", bad)
else:
stdio.printf("[MEM] 记录项抽样 4 条(addr / 块头):\n")
si: sizet = 0
dstep: sizet = self.chk_depth / 4
if dstep == 0:
dstep = 1
shown: int = 0
while si < self.chk_depth and shown < 4:
raw: uint64 = self.chk_stack[si]
hv: uint64 = 0
hv2: uint64 = 0
hv_m8: uint64 = 0
hv_p8: uint64 = 0
if raw >= uint64(self.usable) and raw < pool_end:
bp: rawptr = rawptr(raw)
# Cross-check two reads at the same address: hv direct, hv2 via the two-step memhub
# form; a mismatch means "header read as 0" is a codegen/read issue, not a zeroed pool.
hv = *((uint64*)(bp))
hv2 = uint64(rawptr(*((uint64*)(bp))))
# Neighbor probe: an all-zero region implies the adjacent words are also 0 (signature of a huge zero-fill write).
if raw >= uint64(self.usable) + 8:
bp_m: rawptr = rawptr(raw - 8)
hv_m8 = *((uint64*)(bp_m))
if raw + 8 < pool_end:
bp_p: rawptr = rawptr(raw + 8)
hv_p8 = *((uint64*)(bp_p))
stdio.printf("[MEM] [%zu] addr=%llu hdr=0x%llX hdr2=0x%llX "
"-8=0x%llX +8=0x%llX\n", si, raw, hv, hv2, hv_m8, hv_p8)
si += dstep
shown += 1
stdlib.free(counts)
stdio.fflush(None)
def self_check(self) -> int:
# Validate buddy allocator internal consistency, return 0=OK, non-0=damage
if self.usable is None:
return 0
# Cycle guard: a corrupted next pointer can make the free list cyclic; without a step
# bound the walk would spin forever (a corruption sentinel must terminate).
max_steps: sizet = self.usable_size / MEMBUDDY_MIN_BLOCK + 2
i: int
for i in range(MEMBUDDY_MAX_ORDERS + 1):
head_val: uint64 = self.free_lists[i]
cur: rawptr = rawptr(head_val)
if uint64(cur) != 0 and self._fl_block_ok(cur) == 0:
# Free-list head itself invalid (e.g. uninitialized 0xBE...): dereferencing would AV; report and stop this order.
stdio.printf("[MEM] free_lists[%d] 头非法(非池内对齐块地址,疑元数据未初始化/被写穿): "
"head=%llu chk_depth=%zu usable=%llu usable_size=%zu\n",
i, head_val, self.chk_depth, uint64(self.usable), self.usable_size)
stdio.fflush(None)
break
steps: sizet = 0
while uint64(cur) != 0:
steps += 1
if steps > max_steps:
return 4 # free list cyclic or over the node limit (next pointer damaged)
# Check pointer is in usable region range
if uint64(cur) < uint64(self.usable):
return 1
if uint64(cur) >= uint64(self.usable) + self.usable_size:
return 2
# Check alignment: order i blocks must be aligned to size _block_size_at_order(i)
offset: sizet = uint64(cur) - uint64(self.usable)
bs: sizet = _block_size_at_order(i)
if offset % bs != 0:
return 3
cur = rawptr(*((uint64*)(cur)))
return 0
def check_alive_blocks(self, from_mark: sizet) -> int:
# Check the header integrity of every allocated block in [from_mark, chk_depth).
# An out-of-bounds write corrupts a neighbour's header (parity/order), so this returns
# 0=OK / non-zero=damage (rc meanings in the caller).
# Used as a reset_to sentinel: turns a drifting crash into a deterministic file.
if self.chk_stack is None:
return 0
if from_mark > self.chk_depth:
# from_mark > chk_depth: the caller's mark is bogus (wrong field / cross-instance),
# which would make the check a no-op and miss overflows. fail-fast.
stdio.printf(
"[MEM] check_alive_blocks from_mark=%zu > chk_depth=%zu(mark 异常)\n",
from_mark, self.chk_depth)
return 0
if from_mark >= self.chk_depth:
return 0
if self.usable is None:
return 0
ci: sizet = from_mark
while ci < self.chk_depth:
block: rawptr = rawptr(self.chk_stack[ci])
if block is not None:
if uint64(block) < uint64(self.usable):
# Block address below the pool start.
stdio.printf("[MEM] 存活块损坏 rc=1(块地址低于 usable): block=%llu usable=%llu ci=%zu from_mark=%zu\n",
uint64(block), uint64(self.usable), ci, from_mark)
return 1
if uint64(block) >= uint64(self.usable) + self.usable_size:
# Block address past the pool end.
stdio.printf("[MEM] 存活块损坏 rc=2(块地址超出池末尾): block=%llu usable=%llu ci=%zu from_mark=%zu\n",
uint64(block), uint64(self.usable), ci, from_mark)
return 2
stored: rawptr = rawptr(*((uint64*)(block)))
stored_val: uint64 = uint64(stored)
if (stored_val & 1) != 0:
# Live block: header must be (order<<1)|1 with a valid order and alignment.
order: int = int(stored_val >> 1)
if order < 0:
# Negative order (header written through).
stdio.printf("[MEM] 存活块损坏 rc=3(order 为负): block=%llu order=%d ci=%zu from_mark=%zu\n",
uint64(block), order, ci, from_mark)
return 3
if order > self.max_order:
# order beyond max_order.
stdio.printf("[MEM] 存活块损坏 rc=4(order 超 max_order): block=%llu order=%d max_order=%d ci=%zu from_mark=%zu\n",
uint64(block), order, self.max_order, ci, from_mark)
return 4
offset: sizet = uint64(block) - uint64(self.usable)
bs: sizet = _block_size_at_order(order)
if offset % bs != 0:
# Block not aligned to its order's block size.
stdio.printf("[MEM] 存活块损坏 rc=5(块未对齐 order 大小): block=%llu order=%d bs=%zu offset=%zu ci=%zu from_mark=%zu\n",
uint64(block), order, bs, offset, ci, from_mark)
return 5
# Block-tail canary check: the key sentinel for overflow damage. The canary is
# an 8-byte value, so it must be read through a uint64* view, and the byte
# address is computed on the byte pointer first (arithmetic on a typed pointer
# scales by sizeof(T)=8 and would read a different block).
can_stored: rawptr = rawptr(*((uint64*)(block + bs - MEMBUDDY_CANARY)))
if uint64(can_stored) != uint64(MEMBUDDY_CANARY_PATTERN):
# Overflow scene: the block-tail canary was overwritten (rc=6). Report the
# block/order/bs plus the index and window start for location.
stdio.printf(
"[MEM] canary 损坏: block=%llu order=%d bs=%zu ci=%zu from_mark=%zu canary=0x%llx 期望=0x%llx\n",
uint64(block), order, bs, ci, from_mark,
uint64(can_stored), uint64(MEMBUDDY_CANARY_PATTERN))
# Width diagnostic: print the high/low 32-bit halves of the read value and
# the expectation, to tell a low-32-only write from a sign-extended compare
# (low halves derived by subtracting the high half, no large literal).
can_d: uint64 = uint64(can_stored)
exp_d: uint64 = uint64(MEMBUDDY_CANARY_PATTERN)
can_hi: uint64 = can_d >> 32
can_lo: uint64 = can_d - (can_hi << 32)
exp_hi: uint64 = exp_d >> 32
exp_lo: uint64 = exp_d - (exp_hi << 32)
stdio.printf(
"[MEM] canary 位宽诊断: canary_lo=0x%llx canary_hi=0x%llx 期望_lo=0x%llx 期望_hi=0x%llx\n",
can_lo, can_hi, exp_lo, exp_hi)
# Flush so the scene survives a redirected stdout (the caller prints then exits).
stdio.fflush(None)
return 6
ci += 1
return 0