The RankedAGI Engine

The data

Picture a grid with one row per model and one column per benchmark. Only about a sixth of the cells hold an actual result; the engine fills the rest. Benchmarks use different formats such as percentages, rating scales, and currencies, so every value is first mapped into a common normalized space before anything is compared.

Two estimators

Local, from the nearest models. For a missing cell, the engine ranks every other model by how closely it agrees with the target model on the benchmarks they both have. The closest models, weighted by agreement, effectively vote on the target benchmark, with a signed correction for how much stronger or weaker the target tends to be. This is sharpest where a model has many results and the benchmark is widely covered.

Global, from matrix completion. Separately, one model is fit to the whole grid at once. Writing z for a normalized score, it learns z[model, benchmark] ≈ bias[model] + θ[model] · V[benchmark]. The bias term is the model's general strength and is barely regularized, so even a thinly tested model gets a sensible level. The short vectors θ and V capture interactions through their dot product, for example that one family tends to over-perform on a particular kind of benchmark; they are regularized more strongly so a model with little data cannot be thrown to an extreme. The fit runs by alternating least squares with a fixed seed and a fixed number of passes, so the output is fully reproducible, and it works in per-benchmark percentile space with an empirical-quantile inverse, which handles saturated and bimodal benchmarks without per-benchmark tuning. A final pass adds a small correction learned from the residuals of the global fit.

Combining them

The global view sets the level and the local view sharpens it. The blend leans on the local view when a benchmark is widely covered and on the global view when it is sparse. A floor stops a high-strength model from being pulled below its global estimate, which was the weakness of the older nearest-models-only method, where weaker neighbours used to drag strong models down.

Confidence and how estimates enter a score

Each estimate gets a confidence between 0 and 1 from how much evidence supports it: how many similar models exist, how well the target benchmark is covered, and how cleanly the model fits. Confidence then governs how the estimate is used. Each source is normalized to a 0 to 1 score; an actual result has trust 1.0, while a simulated value must clear a confidence of 0.35 and is used with trust = confidence × 0.75, so a real result always counts for more than an estimate. The composite then combines its sources:

signal_weight    = (source_relevance / 50) ^ 1.25
effective_weight = signal_weight * trust * sampling_reliability

family_score     = sum(source_score * effective_weight) / sum(effective_weight)
family_evidence  = min(sum(effective_weight), strongest_source_weight * 1.2)

observed_score   = sum(family_score * family_evidence)
evidence         = sum(family_evidence)
final_score      = (prior_score * prior_weight + observed_score) / (prior_weight + evidence)

The family cap stops several versions of one benchmark from counting as if they were independent evidence. In the leaderboard, each simulated value shows its confidence as a colored dot, green, amber, or red, and anything below the 0.35 threshold is left out of scoring entirely.

How we know it works

Accuracy is measured by hiding values we already have and predicting them back, across several patterns: a few values hidden, many values hidden, a whole benchmark thinned out, a single model thinned out, and the hardest case, hiding a benchmark's top scorer to see whether the engine can still place it near the top. A change ships only when it lowers these held-out errors. Actual results always override estimates, a simulated value is never used to generate another, and the full set of estimates regenerates deterministically whenever the data changes.