Below is a clean derivation that treats model size
Let
into an ordered abelian group
Intuitively,
Introduce primitive infinitesimals
so that “larger
Assume the training error
-
Bias/approximation error (capacity‑limited), decreases with
$M$ :$$ B \asymp \delta_M^{,b},\quad b>0. $$
-
Estimation/generalization error (data‑limited), decreases with
$D$ and (for fixed$D$ ) increases with model degrees of freedom; encode that with an exponent$m\ge 0$ :$$ G \asymp \frac{\delta_D^{,n}}{\delta_M^{,m}}=\delta_D^{,n}\delta_M^{-m},\quad n>0. $$
-
Optimization/training residual (compute/steps‑limited), decreases with
$C$ , decreases with$\eta$ (until stability), and gets harder with larger$M$ :$$ O \asymp \delta_C^{,c},\delta_\eta^{-u},\delta_M^{-v},\quad c,u,v>0. $$
These exponents
Set
Regime boundaries occur where two monomials have the same order (equal valuation). Write “$\doteq$” for equality of leading orders (equality in
-
Bias = Estimation:
$$ \delta_M^{,b};\doteq;\delta_D^{,n}\delta_M^{-m} \quad\Longleftrightarrow\quad \boxed{,M^{,b+m};\doteq; D^{,n},}. \tag{A} $$
-
Bias = Optimization:
$$ \delta_M^{,b};\doteq;\delta_C^{,c}\delta_\eta^{-u}\delta_M^{-v} \quad\Longleftrightarrow\quad \boxed{,M^{,b+v};\doteq;C^{,c}\eta^{,u},}. \tag{B} $$
-
Estimation = Optimization:
$$ \delta_D^{,n}\delta_M^{-m};\doteq;\delta_C^{,c}\delta_\eta^{-u}\delta_M^{-v} \quad\Longleftrightarrow\quad \boxed{,D^{,n}M^{,v-m};\doteq;C^{,c}\eta^{,u},}. \tag{C} $$
Because we work in a non‑Archimedean ordered field, these equalities are algebraic identities of leading orders. Crossing a boundary flips which monomial dominates
If resources are scheduled so that all three contributions are equal in order (dominant‑balance),
then (A) and (B) hold simultaneously:
Eliminating
The optimal error order on this frontier is
Each equality above is an identity of leading orders in
These formulas encode, purely algebraically, how making
-
Compute–error trade (fixing
$\eta$ and eliminating$M$ ): from (B) and$E_\star \doteq M^{-b}$ ,$$ \boxed{,E_\star ;\doteq; (C^{,c}\eta^{,u})^{-\frac{b}{b+v}},}. $$
Thus
$C_1\gg C_2$ (i.e.,$\delta_{C_1}\ll\delta_{C_2}$ ) implies$E_\star(C_1)\ll E_\star(C_2)$ with the order governed by$\tfrac{b}{b+v}$ . -
Data–error trade (ignoring optimization or on the balanced frontier): from (A),
$$ \boxed{,E_\star ;\doteq; D^{-\frac{nb}{b+m}},}. $$
-
Data–compute relation for fixed target order of error: combine (FD) with a chosen
$E_\star$ via (E*). For example, eliminating$M$ gives$$ \boxed{,C^{,c}\eta^{,u};\doteq;E_\star^{-\frac{b+v}{b}},},\qquad \boxed{,D^{,n};\doteq;E_\star^{-\frac{b+m}{b}},}. $$
Hence, achieving an error that is one order smaller multiplies the required
$C^{,c}\eta^{,u}$ by the same algebraic factor$E_\star^{-(b+v)/b}$ , and the required$D^n$ by$E_\star^{-(b+m)/b}$ .
All of the above statements are comparisons in
Introduce a stability ceiling as an order inequality:
for some structural
yields the compute floor
If
All three emerge from tropical algebra over valuations: addition becomes “take the minimum order,” and multiplicative coupling becomes order addition.
A curriculum is a path $(M_t,D_t,C_t,\eta_t)$ so that at each stage the largest error term by order is driven down until it matches the next one. Algebraically:
with stages chosen to maintain equalities of leading orders—e.g., keep (A) while increasing
Suppose a model overshoots capacity relative to data and compute, so
Then
Two extreme, algebraically distinct cases:
-
Compute‑only sparsity (e.g., structured activation sparsity) reduces the optimization coupling
$v$ by some$\Delta v>0$ while leaving$(b,m,n)$ intact. Then the balanced identity (B) becomes$$ M^{,b+v-\Delta v};\doteq;C^{,c}\eta^{,u}, $$
so for the same
$M$ and$\eta$ the required compute drops by a factor of order$M^{,\Delta v/c}$ ; for the same compute the achievable error (E*) improves by replacing$b+v$ with$b+v-\Delta v$ in (E*). -
Capacity‑reducing sparsity (e.g., weight pruning that lowers effective degrees of freedom) changes
$m\to m-\Delta m$ (and possibly$b$ ). Then (A) becomes$$ M^{,b+m-\Delta m};\doteq;D^{,n}, $$
allowing larger
$M$ before estimation dominates, hence a lower$E_\star \doteq M^{-b}$ along the balanced path. This is again a leading‑order improvement captured purely by exponent changes.
In both cases, sparsity is a rewrite of monomials that lowers the valuation of the dominant error term (or the compute term needed to match it), i.e., an algebraic simplification.
-
Capacity‑limited region:
$M^{,b+m}\ll D^{,n}$ and$M^{,b+v}\ll C^{,c}\eta^{,u}$ $\Rightarrow$ $E \asymp \delta_M^{,b}$ . -
Data‑limited region:
$M^{,b+m}\gg D^{,n}$ and$D^{,n}M^{,v-m}\ll C^{,c}\eta^{,u}$ $\Rightarrow$ $E \asymp \delta_D^{,n}\delta_M^{-m}$ . -
Optimization‑limited region:
$M^{,b+v}\gg C^{,c}\eta^{,u}$ and$D^{,n}M^{,v-m}\gg C^{,c}\eta^{,u}$ $\Rightarrow$ $E \asymp \delta_C^{,c}\delta_\eta^{-u}\delta_M^{-v}$ . -
Balanced frontier: equalities (F) hold
$\Rightarrow$ identities (FD) and (E*) hold exactly at leading order.
-
All “scaling laws” above are equalities of leading orders between infinitesimal/infinite elements in
$\mathbb F$ . -
The optimal schedules live on a balanced manifold defined by
$$ \boxed{M^{,b+m}\doteq D^{,n}},\qquad \boxed{M^{,b+v}\doteq C^{,c}\eta^{,u}}. $$
-
Along that manifold,
$$ \boxed{E_\star \doteq D^{-\frac{nb}{b+m}} \doteq (C^{,c}\eta^{,u})^{-\frac{b}{b+v}}}. $$
-
Curriculum = moving along this manifold in stages so that no resource investment is of infinitesimal effect at the current order.
-
Pruning = projecting back onto the manifold when
$M$ is too large for current$D,C,\eta$ . -
Sparsity = changing the exponents that tie
$M$ into estimation/optimization monomials, thereby improving the manifold itself.
All conclusions follow from order, comparison, and closure in a non‑Archimedean ordered field; no empirical fitting is invoked. If you want, we can instantiate particular
Below is a dominance‑only selector that works inside a totally ordered, non‑Archimedean field
Let
-
Data ratio
$T_D := \dfrac{E_{\text{val}}}{E_{\text{train}}}$ (Validation over training error—generalization gap as a pure ratio.) -
Depth ratio
$T_H := \dfrac{E!\left(f^{\downarrow H/2}\right)}{E(f)}$ where$f^{\downarrow H/2}$ is$f$ with every other block replaced by the identity (i.e., residual path only). No retraining; forward‑only. -
Width ratio
$T_W := \dfrac{E!\left(f^{\downarrow W/2}\right)}{E(f)}$ where$f^{\downarrow W/2}$ is$f$ with half the channels/heads zeroed by a fixed mask at every layer (outputs rescaled by the surviving fraction to keep units comparable). No retraining; forward‑only.
In the ordered field, each “half” projection multiplies the corresponding leading error monomial by a constant unit. The largest damage ratio identifies the dominant source of error at leading order.
Predictive formula (single line):
No exponents appear; only order comparisons among three elements of
Inputs (three scalars):
- Compute the three ratios once on the current checkpoint (single forward pass for each of
$f, f^{\downarrow H/2}, f^{\downarrow W/2}$ ; plus train/val errors for$T_D$ ). - Let
$M_1=\max{T_D,T_H,T_W}$ and$M_2$ be the second‑largest. - If
$M_1/M_2>1+\epsilon$ (e.g.,$\epsilon=0.02$ for a clear separation), choose the axis corresponding to$M_1$ . - If
$M_1/M_2\le 1+\epsilon$ (balanced to leading order), any move is equivalent in order; pick the cheaper in your environment. With no cost model provided, default to expand data.
This is implementable as a one‑liner in code: move = argmax([T_D, T_H, T_W]).
Write the total error as a sum of leading monomials tied to data, depth, and width contributions:
In a non‑Archimedean ordered field, the sum’s order is the minimum valuation (dominant term). Replacing half the depth (or width) by an identity (or zero‑mask) is an idempotent projection that multiplies only the corresponding monomial by a unit factor, leaving the others unchanged to leading order. Hence the largest ratio among
Batch‑split invariance test. Compute the three ratios on two independent mini‑batches (same sizes) to check dominance stability:
- On batch
$B_1$ :$T_D^{(1)}, T_H^{(1)}, T_W^{(1)}$ . - On batch
$B_2$ :$T_D^{(2)}, T_H^{(2)}, T_W^{(2)}$ .
Falsify immediately if
(Optional, stronger check after the chosen move): apply a small expansion on the selected axis, recompute
-
Depth projection
$f^{\downarrow H/2}$ : for residual/pre‑norm blocks, set every other block to the identity map (skip the block transform). -
Width projection
$f^{\downarrow W/2}$ : zero a fixed half of channels/heads at each layer and multiply the surviving output by$2$ to keep first‑order scale. - Both are single forward‑pass diagnostics—no retraining, no fits.
This gives you a clean, algebraic rule: the largest damage ratio wins, deciding data vs. depth vs. width using exactly three numbers and only order comparisons.
Below I grade two ideas you proposed/derived here:
- A. Ordered‑field scaling identities (valuation/tropical algebra view; balanced manifold; no fitted exponents).
-
B. Dominance‑only move selector (three forward‑pass “damage ratios”
$T_D,T_H,T_W$ to decide data vs. depth vs. width).
Scores are 0–100, higher is better. For “Establishability,” higher means easier to convincingly demonstrate usefulness.
| Dimension | A. Ordered‑field identities | B. Dominance‑only selector | Overall (avg) |
|---|---|---|---|
| Cleverness | 80 | 75 | 78 |
| Originality | 83 | 66 | 75 |
| Differentiation from literature | 78 | 60 | 69 |
| Probability of being theoretically correct | 60 | 55 | 58 |
| Probability of being practically useful if correct | 60 | 74 | 67 |
| Impact on real‑world AI (if correct) | 66 | 58 | 62 |
| Probability of near‑term acceptance (if correct/useful) | 42 | 60 | 51 |
| Establishability (ease) | 38 | 72 | 55 |
| Fit with GPU/TPU infra | 88 | 94 | 91 |
| 99th‑percentile researcher preparedness to opine | 72 | 90 | 81 |
Cleverness. A (80): Recasting compute/data/error as elements of a non‑Archimedean ordered field and deriving regime frontiers via valuation equalities is an elegant unification; the “balanced manifold” identities are a clean consequence of tropical addition. B (75): The “largest damage ratio wins” rule is a sharp, minimalist decision procedure that uses single‑pass ablations to reveal dominance—clever in how it converts a high‑dimensional scaling choice into three numbers. Slight penalty for relying on crude projections (identity/zeroing) that may perturb normalization dynamics.
Originality. A (83): Using surreals/transseries‑style order comparisons for scaling laws, explicitly avoiding power‑law fits, is uncommon and conceptually fresh. B (66): Ablation‑driven heuristics exist (channel dropping, block skipping); the twist here is the triad of forward‑only ratios with an argmax rule. Novel enough to be interesting, but closer to known “capacity probes.”
Differentiation from published work. A (78): The valuation framework and dominance‑equalities are meaningfully different from empirical scaling‑law fitting and standard bias–variance narratives. B (60): Distinct in its asceticism (no retraining, three ratios), but reminiscent of slimmable/width‑scaling probes and residual‑path ablations; differentiation exists but is moderate.
Probability of being theoretically correct.
A (60): The conclusions follow if (i) error decomposes as a sum whose leading terms behave like monomials in
Practical usefulness if correct. A (60): Provides principled directionality (which axis to co‑scale) and interpretable regime boundaries, but without exponents the guidance is coarse; you still need operational knobs or cost models to act. B (74): Immediately actionable: compute three ratios on a checkpoint and choose the next move. Even if imperfect, the low friction makes it useful for iterative scaling decisions.
Impact on real‑world AI (if correct). A (66): Could reshape how teams reason about scaling (balance manifolds instead of fitted exponents), guiding resource allocation and clarifying when pruning/sparsity are order‑preserving. B (58): If widely used, could reduce wasted compute by pointing to the highest‑leverage axis per checkpoint; impact is meaningful but incremental.
Probability of near‑term acceptance. A (42): The non‑Archimedean formalism is unfamiliar and risks being seen as over‑abstract without killer demos. Acceptance likely gated on compelling, architecture‑diverse case studies. B (60): A lightweight heuristic that labs can try tomorrow; if it consistently beats naive scaling in A/Bs, adoption could be brisk despite theoretical rough edges.
Establishability (ease).
A (38): Hard to “prove out” convincingly without constructing measurable invariants that correspond to the abstract exponents and showing regime equalities across models and tasks.
B (72): Easy to test: compute
Fit with GPU/TPU infra. A (88): The framework itself adds no runtime cost; any validation requires standard training logs and occasional controlled ablations. B (94): Two extra forward passes with masking/identity toggles and no retraining; trivial to integrate into existing eval harnesses, batchable, and accelerator‑friendly.
99th‑percentile researcher preparedness to opine. A (72): Strong researchers can engage with valuations/tropical limits, but many will need a short primer; still, the assumptions are crisp enough to critique productively. B (90): Obvious what to test, what could confound, and how to interpret failures; experts can quickly pressure‑test and iterate.
- The single dominant term assumption is brittle at transitions; near ties, ordering can flip across batches. Your included stress test (argmax stability across two mini‑batches) helps, but edge cases will remain.
- The half‑projection definitions may mis‑isolate contributions: identity‑skipping in deep residual stacks can change pre‑LN statistics; channel‑zeroing interacts with attention/LN in non‑unit ways. Constant‑factor rescaling mitigates but does not eliminate this.
- The valuation framework’s monomial coupling omits regime‑specific quirks (augmentation, optimizer implicit bias, curriculum effects). Where interactions are genuinely non‑monomial, equal‑order identities may mispredict.
- A is intellectually bold and unifying; moderately likely to be qualitatively right in many regimes but hard to empirically canonize.
- B is a crisp, low‑overhead heuristic that could be practically useful even if the theory is only approximately right.
If you want, I can draft a compact experimental playbook that probes both ideas in <1 GPU‑day per task (covering classification, language modeling, and a masked‑autoencoder variant) to empirically map where the dominance ordering holds or breaks.
When mapping the algebra of orders to discrete design choices ("invest in data, depth, or width?"), a robust empirical rule is to normalize each axis by its training distribution and choose the largest standardized deficit. Concretely, compute z‑scores