Evaluation methodology and ecosystem

LocalArena is the open evidence layer for a team's own model decision. It runs the exact endpoints, model IDs, parameters, and tasks selected by the user, then keeps the sample-level evidence portable. A rating is one view of that evidence, not the product's source of truth.

The practical question is not “which model wins everywhere?” It is “which exact model and configuration should we ship for this workload, given quality, failures, latency, token use, and the uncertainty in this experiment?”

Two ranking methods with different jobs

Arena.standings() retains sequential Elo. It is useful for a changing stream of human or external verdicts where history order is part of the process. It is not the default statistical answer for a completed static evaluation: replaying the same cyclic evidence in another order can change online Elo.

Arena.bradley_terry() in Python and Arena.bradleyTerry() in JavaScript fit the complete match set in batch. The implementation:

• canonicalizes contestants and outcomes before fitting, so match replay order does not change the result;
• uses one symmetric virtual draw on each observed comparison edge to keep complete-separation estimates finite;
• converts fitted log abilities onto the familiar 400-point rating scale;
• uses a seeded cluster-parametric bootstrap and reports a confidence interval;
• identifies disconnected comparison components instead of inventing a global order between models that never share evidence; and
• marks a result inconclusive when intervals overlap, evidence is absent, or a global comparison is impossible.

For an EvaluationRun, pair outcomes sharing a task ID form one bootstrap cluster. Each replicate resamples those clusters, then simulates outcomes from the fitted Bradley–Terry probabilities. Repeated observations of the same model pair inside one selected task share a simulated outcome, preserving within-task dependence instead of treating every pair row as independent. Degenerate replicates cannot produce false certainty: even one observed match gets a finite uncertainty interval and remains inconclusive. The saved summary uses 500 seeded samples at 95% confidence; the lower-level arena API defaults to 1,000 and exposes all three settings.

This follows the important methodological direction of modern arenas without claiming their exact estimator. LMArena moved its static leaderboard from online Elo to order-independent Bradley–Terry with bootstrap confidence intervals after documenting the replay-order problem in its leaderboard methodology update.

Failure and coverage policy

A raw average over surviving scores can reward a flaky model. LocalArena keeps that raw mean for diagnosis, but ranks the default report by reliability_adjusted_score:

sum of successful scores / every expected scored row

A failed generation or scorer therefore contributes zero to the decision score. The report shows both the raw mean and score_coverage, so a model with one perfect answer and nine errors cannot outrank a model with eight correct answers out of ten.

Provider reliability across scored and unscored tasks remains separately visible as coverage. No missing token, latency, judge, or score measurement is silently converted into a successful observation.

Native task packs

A native task pack is strict JSON with:

• schema version, pack name, version, license, and optional source provenance;
• provider-independent messages or prompt text;
• deterministic evaluator configuration; and
• JSON-safe task metadata for categories and later score slices.

The loader validates the pack, rejects duplicate IDs and unsupported fields, and assigns a cross-runtime SHA-256 content identity. Python and JavaScript use the same typed canonical encoding, including IEEE-754 number bytes, so the same manifest has the same digest in both runtimes.

That identity is not a security sandbox. Packs should be trusted or reviewed before evaluation. In particular, regex and extract patterns execute in the host runtime's native regular-expression engine and pathological patterns can consume excessive CPU.

JSONL is supported in two forms:

• one native task object per line; or
• simple rows containing input and ideal, which map to live tasks with beginning-of-answer multi-reference matching.

The latter follows the open-source Evals row shape described in the evaluation-building guide. It is intentionally narrow: it does not claim to reproduce an external framework's custom code, model graders, or hosted execution.

Current deterministic scorers cover:

• multi-reference exact, beginning, end, or substring matching;
• boundary-safe leading choice labels;
• regular-expression answer extraction followed by multi-reference matching;
• whitespace-token multiset F1 with a pass threshold;
• exact, containment, format, strict JSON, and numeric checks; and
• an explicitly configured live pointwise judge.

These cover common generation-based classification, extraction, instruction following, structured output, and final-answer tasks. The Inspect standard scorer catalog and lm-evaluation-harness task guide informed the must-have set: accepted targets, output extraction, choice grading, partial credit, filters, and reproducible task configuration recur across mature systems.

External harness boundary

LocalArena should not reimplement every benchmark runtime. It should normalize their authoritative evidence without weakening their methodology.

Ecosystem	Appropriate integration
Inspect AI	First full external adapter. Keep its dataset, solver, scorer, tools, epochs, sandbox, and published log semantics; import its structured log evidence.
lm-evaluation-harness	Invoke the installed harness and import aggregate plus sample logs. Do not clone its YAML/Jinja task language or pretend chat generation reproduces official log-likelihood tasks.
Evals-style JSONL	Native import is implemented for simple input and ideal rows. Custom or hosted graders stay in their original runner.
Promptfoo	Add test-case and result interoperability while keeping raw prompt/output retention opt-in.
LightEval, DeepEval, Ragas, and HELM	Add evidence importers when a user needs them; avoid runtime dependencies in the core package.

Code execution, terminal work, browser tasks, tool use, simulators, multi-turn state, log-probability multiple choice, and benchmark-specific graders must remain in the official environment. A prompt-only chat call is not an official run of those tasks.

Product moat

The defensible advantage is not another universal leaderboard and not Elo. Centralized arenas already provide broad anonymous comparisons, and current LMArena ranking is neither simply online Elo nor closed methodology.

LocalArena's moat is the combination of:

• live bring-your-own execution across local and hosted endpoints;
• private user-owned tasks that do not need to be uploaded to a central arena;
• content-addressed task and run evidence instead of copied leaderboard rows;
• a fully paired model × task experiment with failures preserved;
• uncertainty-aware, order-independent ranking and explicit inconclusive outcomes;
• portable Python/JavaScript evidence and standalone offline reports; and
• an adapter boundary that can unify external harness evidence without misrepresenting it as a native live run.

This is complementary to general-purpose arenas. Their strength is population- level preference evidence; LocalArena's strength is a reproducible deployment decision on a team's exact workload.

Next methodological priorities

The highest-value additions after the current native layer are:

1. Blind pairwise judging with both answer orders, position-conflict reporting, judge disagreement, and length/style diagnostics. Position and verbosity bias are documented in LMArena style control, Arena-Hard, and AlpacaEval.
2. Paired baseline-versus-candidate regression reports with task-level intervals and CI-friendly release gates.
3. Category-balanced scorecards, critical-task gates, and quality/latency/token Pareto views.
4. Inspect and lm-evaluation-harness evidence adapters.
5. Trusted public packs with pinned source revision, license, attribution, and download digest. Good first generation-based candidates include MMLU-Pro, GSM8K, and TruthfulQA, provided their official prompts, answer extraction, splits, and licenses are preserved.

Imported historical results must always be labeled imported. They must never be presented as live LocalArena execution.