Strata is a C++20/CUDA inference engine for dense and mixture-of-experts models whose weights exceed local VRAM, and often exceed local RAM as well. It splits a single model checkpoint across GPU VRAM, host RAM, and read-only NVMe storage, and executes it with the precision, router semantics, expert count, and top-k the checkpoint actually declares. There is no Python or ML framework in the runtime; it builds as a static library plus a set of CLI binaries.
The motivation is ordinary: most people who want to run a large model do not have a rack of data-center GPUs. They have one or a few consumer GPUs and a reasonable amount of system RAM. Strata's job is to make that combination usable for models that would otherwise only run on much larger machines, without quietly making the model worse to get there.
This is the machine the project is built and measured on. It's an ordinary workstation, not a server:
| Memory Type | Hardware |
|---|---|
| GPUs | RTX 5060 Ti 16 GiB, RTX 3090 24 GiB, RTX 3090 24 GiB (64 GiB total) |
| RAM | 251 GiB DDR4 |
| CPU | Xeon E5-2680 v4 |
On this hardware Strata supports four checkpoints: Gemma 4 31B-IT W8A16 (35.1 GB, dense text and vision), resident across the three GPUs; Laguna S 2.1-NVFP4 (99.7 GB, 48 layers, 256 experts, top-10), whose routed experts exceed combined VRAM but fit in host RAM, so decode streams them over PCIe without touching storage after warm-up; DeepSeek-V4-Flash-0731 (167 GB, 43 layers, 256 experts, top-6), staged into host RAM so decode does not touch storage after warm-up; and GLM-5.2 W4A16 (388 GB, 78 layers, 256 experts, top-8), which is larger than the machine's combined VRAM and RAM and therefore remains I/O-dependent.
Fitting a large model into limited memory is easy if you're willing to degrade it — drop below four bits, cut experts, reduce top-k, or fall back to a cheaper path when the fast one isn't available. Strata's execution model is built against doing that:
- Weight precision does not go below four bits anywhere in the system: not in canonical weights, not in caches, not in KV storage, not in speculative drafts, not in prefetch predictors.
- Precision, router semantics, expert count, and top-k are read from the checkpoint and never changed silently.
- A predictor may only influence scheduling or prefetch order. It never changes what gets computed.
- Exact mode either completes the exact computation or reports a failure. It does not fall back to something cheaper without saying so.
- If a dense model is larger than the machine's combined resident memory, Strata reports it as I/O-dependent rather than hiding the cost behind a cache.
These are enforced in code, not just documented, and they're part of why some numbers below (GLM's decode speed, for instance) are unglamorous: the honest number for a checkpoint that doesn't fit in memory is a slow one.
You need a C++20 compiler, CMake 3.20+, Make, and CUDA 12.8 (or a compatible toolchain) for the GPU backend. Gemma 4 image requests additionally use the installed ImageMagick decoder for bounded PNG, JPEG, and WebP inputs.
git clone https://github.com/ro99/strata
cd strata
make checkPlace a checkpoint under models/ in its original Safetensors form. Strata reads the shards as published — there's no conversion step and no second copy of the weights on disk.
./build/strata-chat \
--model models/dsv4f --model-type deepseek \
--context-size 8192 --max-new 256 --devices 0,1,2 \
--vram-fraction 0.95 --pin-resident-arena --flash-attentionGemma 4 uses its native W8A16 checkpoint directly and enables CUDA attention:
./build/strata-chat \
--model models/gemma4 --model-type gemma4 \
--context-size 2048 --max-new 256 --devices 0,1,2Laguna S 2.1 runs its NVFP4 experts as shipped; the resident spine is pinned first and the routed experts fill whatever VRAM is left:
./build/strata-chat \
--model models/laguna-s-21 --model-type laguna \
--context-size 2048 --max-new 256 --devices 0,1,2Startup prints the selected devices, the VRAM budget per device, load progress, elapsed load time, and the active sampler. Decoding defaults to greedy (--temperature 0) for reproducible output. Multi-turn chat reuses the cached prefix and only prefills new tokens.
Whether a model is fast usually comes down to one question that is invisible
until the weights are already loaded: does everything fit on the GPUs, or is
some class of weight being streamed on every decode step? --dry-run answers it
in about a second, without reading a single weight:
./build/strata-chat \
--model models/gemma4 --model-type gemma4 \
--context-size 8192 --devices 0,1,2 --dry-runcomponent size tier cuda:0 cuda:1 cuda:2 per step
-------------------------------------------------------------------------------------
attention 8.39 GiB device 3.36 GiB 3.49 GiB 1.55 GiB 8.39 GiB
feed-forward 20.59 GiB device 8.24 GiB 8.58 GiB 3.78 GiB 20.59 GiB
norm 5.06 MiB device 2.02 MiB 2.11 MiB 950.00 KiB 5.06 MiB
output-head 2.62 GiB device - - 2.62 GiB 2.62 GiB
vision 1.03 GiB device 1.03 GiB - - -
kv-cache 1.41 GiB device 576.00 MiB 592.00 MiB 272.00 MiB 1.41 GiB
workspace 2.25 GiB device 768.00 MiB 768.00 MiB 768.00 MiB -
-------------------------------------------------------------------------------------
device total 36.29 GiB 13.93 GiB 13.40 GiB 8.96 GiB
admitted budget 19.81 GiB 19.81 GiB 13.04 GiB
layer blocks cuda:0=0..23, cuda:1=24..48, cuda:2=49..59
hops 2 cross-device activation transfers per decode step
decode reads 33.02 GiB device, 0 B host-to-device, 0 B nvme (bytes per step, not a duration)
max context 200599 tokens at this placement
verdict fits: every weight, cache, and workspace is device resident
The plan is then cached under ~/.cache/strata/plans and the next real load
reuses it, so what runs is what the dry run printed. A plan is keyed by
checkpoint contents, GPU set, context size, and device list; change any of them
and it is recomputed automatically. --replan forces a fresh one,
--no-plan-cache neither reads nor writes one, and --plan-cache DIR (or
$STRATA_PLAN_CACHE) moves the directory.
Read the table as an instance of the cost model in
research/moe-tiered-memory-decode-optimization.md: the per step column is the
W_r volume each component contributes to one batch-1 decode step, split by the
resource that serves it. It is a byte count, not a duration — the planner
measures no bandwidth and converts nothing to milliseconds.
What each verdict means:
| Verdict | Meaning |
|---|---|
fits |
Every weight, cache, and workspace is device resident. Decode reads no PCIe and no NVMe. |
fits with a host tier |
Sparse classes are cached in VRAM over a host-resident copy. Decode streams the misses over PCIe. |
I/O dependent |
The resident set exceeds VRAM plus host RAM, so steady-state decode reads NVMe. Caching cannot manufacture sparsity in a dense model. |
Placement is prescriptive for Gemma 4: the plan chooses contiguous, byte-balanced layer blocks sized to each GPU's admitted budget, and the load performs exactly that assignment. For GLM and DeepSeek the plan is descriptive — it sizes and admits the placement those runtimes already perform and reports it without changing it, so a planning defect cannot regress a validated runtime.
--dry-run exits 0 when the configuration fits and 1 when it does not, so it
works as a preflight check in a script. strata-server takes the same flags.
The DeepSeek command above favors decode throughput: pinning adds about 17 seconds to startup on the development machine, then avoids pageable host-staging stalls while loading expert weights into VRAM.
Flags worth knowing:
| Flag | Purpose |
|---|---|
--devices 0,1,2 |
CUDA devices to use |
--context-size N |
Context ceiling enforced by the runtime |
--vram-fraction F |
Fraction of free VRAM budgeted for weight caching (default 0.85) |
--host-memory 216G |
Host RAM ceiling for the resident weight arena |
--pin-resident-arena |
Page-lock DeepSeek's resident weights for faster host-to-device demand loads |
--flash-attention |
Use the exact CUDA attention fast path where it is faster |
--no-prepack-mhc |
Disable the default exact AVX2-packed mHC projection path |
--dry-run |
Size and place every component against this machine, print the plan, cache it, and exit without reading weights |
--replan |
Recompute and overwrite a cached placement plan |
--plan-cache DIR |
Directory for cached plans (default ~/.cache/strata/plans) |
--no-plan-cache |
Neither read nor write a cached plan |
The sampler runs as a fixed pipeline. Penalties rewrite the logits, every truncation stage then reads the model's own distribution, and temperature rescales only the surviving candidates before a seeded Gumbel-max draw:
presence/frequency/repetition → DRY → n-gram ban → logit bias
→ top_k → top_p → min_p → typical_p → XTC → future entropy
→ temperature → draw
Truncating on the natural distribution is what makes the thresholds mean what
they say: --min-p 0.05 is "at least 5% as likely as the best token according
to the model" at any temperature.
Just want temperature? Pass --temperature with no --preset and nothing
else runs — no truncation, no penalties, no XTC:
./build/strata-chat --model models/dsv4f --model-type deepseek --temperature 0.9The startup banner echoes exactly what's active, e.g. [sampler] temperature=0.9; if other stages were silently on, they'd show there too. In
the TUI, this means leaving the SAMPLER field on PRECISE — it writes no
stages of its own — and editing TEMPERATURE directly; an edited temperature
always overrides whatever the preset last wrote there.
Four presets bundle the stages that are otherwise on separate flags:
| Preset | Stages | What to expect |
|---|---|---|
--preset precise |
none (temperature only, default 0) |
Deterministic at temperature 0; with temperature raised, plain softmax sampling over the full vocabulary — the most expensive path per token, ~4.6 ms/token at DeepSeek's vocabulary, and the one most prone to picking an implausible tail token at high temperature since nothing truncates it |
--preset balanced |
min_p 0.05, repetition_penalty 1.05 over the last 256 tokens |
Cuts tokens under 5% as likely as the best one, so the tail can't get picked; mild pushback on restating the same tokens. Closest to "temperature sampling, but safe" |
--preset creative |
min_p 0.02, XTC 50% at threshold 0.1, DRY, repetition_penalty 1.03 over the last 512 tokens |
Half the time, removes whichever safe/obvious token was in reach and forces a still-plausible alternative instead — the mechanism aimed at "goes to the mean" prose. DRY leans against the loops that removing the safe choice tends to invite. Expect more variance run to run; verify on your own material before trusting it unsupervised |
--preset future-entropy |
min_p 0.05, 20-candidate lookahead over the top-30 future, alpha 0, DRY, repetition_penalty 1.03 over the last 512 tokens |
Scores each candidate by how much future choice it unlocks. Costs 21 forward passes per token, so it decodes roughly 21× slower than the same settings without it — this is a quality knob, not a throughput one |
A preset writes defaults; any flag after it overrides them — including
--temperature, which is how you'd run e.g. --preset creative --temperature 0.7 to keep XTC/DRY but sample less aggressively.
| Flag | Purpose |
|---|---|
--temperature F |
Rescales survivors before the draw; 0 is greedy |
--top-k N |
Keep the N most likely tokens |
--top-p F |
Keep the smallest set carrying mass F |
--min-p F |
Keep tokens at least F times as likely as the best one |
--typical-p F |
Keep the tokens whose surprisal is nearest the distribution's entropy |
--xtc-probability F --xtc-threshold F |
With probability F, drop every candidate above the threshold except the least likely of them |
--presence-penalty F --frequency-penalty F |
Subtractive repetition penalties |
--repetition-penalty F --penalty-window N |
Multiplicative penalty, optionally bounded to the last N tokens |
--dry-multiplier F --dry-base F --dry-allowed-length N --dry-window N |
Penalize the token extending the longest repeated suffix |
--no-repeat-ngram N |
Hard ban on completing an N-gram already in the output |
--future-entropy N --future-entropy-top-n N |
Look ahead one step past the N likeliest survivors and score them by the entropy of the top-n future |
--alpha F --future-entropy-curve NAME |
Crossfade between probability and future entropy; which exponent mapping to use |
--alpha-wave-amplitude F --alpha-wave-period F |
Oscillate alpha over the generated tokens |
XTC is the stage aimed squarely at flat prose: it removes the safe continuation and leaves a plausible one in its place. It draws from the generator even at temperature zero, so a run using it is seeded-reproducible but not greedy, and the startup banner reports it as sampled rather than exact.
Prefers tokens that keep the next step's options open, after
Count Bayesie, Making LLMs better at creative writing using
entropy.
The model is run one step past each surviving candidate w to get
q_w = p(V | c + w); the normalized entropy of that distribution's top-n,
H(w) = [-Σ q̃ ln q̃] / ln n ∈ [0, 1], reweights the candidate:
s(w) = p(w | c)^a · H(w)^b
and the draw is made from s renormalized over the candidates, so
--temperature rescales the blended score rather than the raw probability.
--alpha sets both exponents at once: -1 is ordinary sampling and the stage
becomes a no-op, +1 scores on the future alone. Two mappings are available
because the article and the reference implementation disagree between the
endpoints — --future-entropy-curve article (default) uses
a = 1 − max(0, α), b = 1 − max(0, −α), so α = 0 is exactly the article's
headline s = p · H; crossfade uses a = 1 − t, b = t for t = (α+1)/2,
so α = 0 is √(p · H). They agree at α = ±1 and nowhere else.
Two things are load-bearing:
- It costs forward passes. One per candidate, sequentially, because the KV
cache holds a single sequence —
--future-entropy 20makes a token 21 decode steps instead of one. Every other stage in the pipeline is arithmetic on logits that are already in hand; this one is not. The batched form the reference implementation uses needskforked sequences decoded at one position, which the GLM cache cannot express today. - A relative-plausibility cut in front of it is not optional. Broken
word-fragments have maximally uncertain futures, so entropy selects them
unless something has already removed them. Keep
--min-pat 0.05 or above; the preset does. The lookahead runs last, on the survivors only, which is also what keeps the cost proportional to what the cheaper stages accepted.
The lookahead is exact: each speculative pass is rolled back to a
bit-identical cache before the next candidate runs, so the emitted token is
decoded from the same state it would have been without the stage. It draws
nothing from the generator, so --temperature 0 with future entropy is still
reported as exact greedy decoding — the argmax of s rather than of p.
Every knob is also accepted by the OpenAI-compatible server, under the same
names, on both /v1/chat/completions and /v1/completions.
Reported logprobs are the model's natural log probabilities, computed from the
unmodified logits before penalties, truncation, and temperature. They describe
the model rather than the sampler settings, so they stay comparable across
requests that used different knobs.
strata-tui is a Ratatui frontend over the same runtime process: a launch form, streamed output, a throughput graph, prefill/decode telemetry, context usage, and exact-versus-sampled status.
make tui
./target/release/strata-tuiF1 shows the keyboard map, Ctrl+L shows runtime diagnostics. Rust is only a dependency for this frontend, not for the runtime itself. Details in docs/tui.md.
strata-server exposes the same runtime over an OpenAI-compatible HTTP API. It is a thin layer on top of the C++ runtime, not a second inference path:
./build/strata-server --model models/glm52 --model-type glm \
--model-id glm52 --context-size 2048 --devices 0,1,2
curl http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"glm52","messages":[{"role":"user","content":"Hello"}],"stream":true}'It serves /v1/models, /v1/health, /v1/chat/completions, /v1/completions, and /v1/tokenize, with streaming over SSE. An endpoint the loaded model has no exact implementation for returns an explicit error. Measured serving overhead is about 0.04% of a decode step.
| Model | Layout | Native precision | Status on this hardware |
|---|---|---|---|
| Gemma 4 31B-IT | 60 dense hybrid-attention layers; 27-layer vision tower | INT8 group-32 text linears, BF16 vision/embeddings and BF16 KV | Fully resident across GPU VRAM; text and OpenAI image-content generation |
| DeepSeek-V4-Flash-0731 | 43 layers, 256 experts, top-6 | FP4 E2M1 experts, FP8 E4M3 spine, BF16/F32 | Resident in RAM; zero checkpoint reads during decode |
| Laguna S 2.1-NVFP4 | 48 layers in a 1:3 global/sliding attention pattern, 256 experts + 1 shared, top-10 | NVFP4 (E2M1 pairs, group-16 E4M3 scales, FP32 per-tensor global scale) routed experts in layers 1-39, BF16 everywhere else; W4A16 execution | Spine resident across GPU VRAM; routed experts exceed VRAM and stream from host RAM |
| GLM-5.2 | 78 layers, 256 experts, top-8 | INT4 group-128 linears, BF16/F32 sensitive tensors; W4A16 execution | Larger than combined memory; I/O-dependent |
Each runs its declared model semantics as-is: local/global grouped-query
attention, proportional RoPE, GeGLU, soft-capped logits, and bidirectional
image blocks for Gemma 4; hybrid compressed attention, manifold-constrained
hyper-connections, and sqrtsoftplus/noaux_tc routing for DeepSeek;
MLA-style projections, compressed KV, and sigmoid/noaux_tc top-8 routing for
GLM; and per-layer head counts, QK RMSNorm, per-head softplus output gating,
YaRN rotary on the global layers against plain rotary on the sliding ones, and
sigmoid top-10 routing with a correction bias and a 2.5x routed scale for
Laguna.
The current 0731 checkpoint has not yet been benchmarked. For context, the last validated DeepSeek measurement used the now-unsupported preview checkpoint:
| DeepSeek V4 preview (historical) | GLM-5.2 | |
|---|---|---|
| Checkpoint size | 167 GB | 388 GB |
| Decode | ~4.0 tok/s | 0.283 tok/s |
| Checkpoint reads during decode | 0 | 910 GB/run |
| Load time | ~22 s | ~23 s |
The historical DeepSeek number is from an 18-token prompt, 152 generated tokens,
three GPUs, 216 GiB host ceiling, --flash-attention --pin-resident-arena; it
must not be attributed to 0731. GLM's is a median of three runs, 30-token prompt,
128 generated tokens. Neither number transfers to a different context length or
prompt — see docs/experiments/ for the full records and
their operating points.
The difference between the two rows is mostly explained by whether the checkpoint fits in RAM. DeepSeek does, so decode after warm-up doesn't touch storage. GLM doesn't, so every decode step pays for storage traffic. Reducing that cost for checkpoints in GLM's position, without changing precision or routing, is the current research direction.
- Loading. A content-addressed sidecar manifest references byte ranges inside the original Safetensors shards. The shards are opened read-only and are never duplicated or repacked.
- Admission. Before generation starts, the runtime computes a placement for
every tensor — VRAM spine, host arena, or storage — and either commits to
that plan or refuses to start.
--admission-only --jsonprints the plan without loading the model. - Residency. The dense/shared spine is pinned in VRAM. Routed experts live in a host RAM arena and are leased into VRAM per decode step through an LRU cache with a capacity-weighted schedule across GPUs.
- Kernels. Native INT4 group-128, INT8 group-32, FP4 E2M1, and FP8 E4M3 CUDA kernels for compute capabilities 8.6 and 12.0, checked against a CPU reference implementation.
- Instrumentation. Every run reports checkpoint reads, H2D/D2H bytes, cache hits/misses/evictions, per-phase timings, RSS, and per-GPU VRAM as JSON.
For more detail: docs/current-architecture.md describes what's implemented, docs/architecture.md describes the target scheduler design, and docs/deepseek-v4-runtime.md covers the DeepSeek contract specifically.
Model adapters are kept narrow by design — an adapter owns its tokenizer, tensor roles, router semantics, and operations, and nothing outside that. Qwen and diffusion models are the intended next architectures once the current correctness gates clear.
The main open research problem is making decode independent of storage for checkpoints like GLM's, where the model doesn't fit in the machine's combined memory. Work on custom quantization or pruning to shrink checkpoints further is also of interest, but it will be held to the same rule as everything else here: it has to clear a measured quality gate before it ships, and the four-bit floor is not something that moves to get there.
Strata is run as a research project, not just an engine. Changes are expected to start from a measured bottleneck against the cost model in research/moe-tiered-memory-decode-optimization.md, not from a guess about what's slow. Results are reported as medians over interleaved repetitions, with the full operating point stated alongside the number. Rejected experiments are recorded with the same care as accepted ones in docs/experiments/.
If you're contributing, CLAUDE.md has the full rules. In short: state a hypothesis and a kill criterion before building, measure before optimizing, run make check before claiming a result, and don't call something a win if it's within run-to-run variance.
apps/ command-line tools and the Rust TUI
include/strata/ public C and C++ interfaces
src/ runtime, model adapters, checkpoint, scheduling
kernels/cpu/ numerical reference implementations
kernels/cuda/ CUDA backend
tests/ dependency-free correctness tests and fixtures
scripts/ reproducible benchmarks and determinism checks
docs/ architecture, contracts, and dated experiment records
Start with the docs index — it distinguishes current behavior from target design, active plans, and historical evidence.
Apache-2.0.