Provider and task support

LocalArena evaluates outputs produced during the current run. It does not ship recorded model answers, copy scores from public leaderboards, or substitute cached responses when an endpoint is unavailable.

For provider installation, readiness checks, credentials, and a first scored run, start with the one-minute getting-started guide. This page documents the architecture and support boundary behind those commands.

The evidence and ranking engine remains offline and dependency-free. Provider calls are an optional execution layer around it: generate live answers, score every configured row, derive pairwise outcomes, and retain provenance.

Provider architecture

The first provider set uses one OpenAI-compatible Chat Completions transport with five named presets. A preset supplies safe defaults such as the base URL and expected authentication; the request, response, error, and model-listing logic stays shared.

Provider	Location	Default base URL	Chat and model endpoints	Authentication	Release behavior and notes	LocalArena adapter
llama.cpp server	Local or self-hosted	http://127.0.0.1:8080/v1	POST /chat/completions; GET /models	None by default; Bearer token when the server is started with --api-key	Bounded non-streaming requests; use a server alias for a stable model ID, and configure a compatible chat template	Named preset over the shared transport
Ollama	Local	http://127.0.0.1:11434/v1	POST /chat/completions; GET /models	No server credential by default; some clients require a non-secret placeholder value, which Ollama ignores	Bounded non-streaming requests; models must be pulled before use	Named preset over the shared transport
LM Studio	Local	http://127.0.0.1:1234/v1	POST /chat/completions; GET /models	None when authentication is disabled; Bearer token when authentication is enabled	Bounded non-streaming requests; the selected model must be available to the local server	Named preset over the shared transport
OpenRouter	Hosted	https://openrouter.ai/api/v1	POST /chat/completions; GET /models	Bearer API key	Bounded non-streaming requests; optional HTTP-Referer and X-OpenRouter-Title attribution headers	Named preset over the shared transport
OpenAI	Hosted	https://api.openai.com/v1	POST /chat/completions; GET /models	Bearer API key; optional organization and project headers	Bounded non-streaming requests	Reference preset for the shared transport

For all five presets, this release sends stream: false, reads the answer from choices[0].message.content, and reads model IDs from data[].id. Responses are buffered only up to the configured byte limit. The transport accepts an explicit base URL, so another compatible local or hosted server can be used without adding a new protocol implementation.

Why one transport

All five servers deliberately expose the same small interoperability surface: Chat Completions, model listing, Bearer-style headers when authentication is needed, and the same core response envelope. Sharing that transport provides:

• the same core request fields and normalized result contract in the Python and JavaScript packages;
• one place for timeouts, bounded response parsing, retries, response validation, error normalization, and secret redaction;
• contract tests that apply to every preset;
• custom-base-URL support without a provider-specific dependency; and
• a small portable parameter set: model, messages, max_tokens, temperature, seed, and stop sequences.

A compatibility layer is not a claim that every provider supports every vendor-specific feature. Provider-only request fields remain explicit pass-through options, and callers should use only fields supported by their selected endpoint. A genuinely different wire protocol belongs in a separate transport rather than in conditionals scattered through the shared one.

Live execution and network guarantees

Provider-backed evaluation follows an explicit-network contract:

1. Importing LocalArena, constructing an arena or provider, scheduling pairs, reading standings, and loading or saving snapshots perform no network I/O.
2. A network request occurs only when the caller explicitly asks to list models, generate an answer, or run a provider-backed task.
3. Each request goes only to the configured base URL. There is no silent provider fallback, cloud fallback, endpoint discovery, model download, or model pull.
4. Local presets default to loopback addresses. Hosted presets must be chosen explicitly and require their own credential.
5. Generation is live. An unreachable endpoint, missing model, invalid response, or timeout is reported as an error; it is never replaced with a bundled or previously recorded answer.
6. Generation makes one attempt by default. Retries are opt-in; when enabled, they repeat the same configured request and remain visible in run metadata. They do not switch models or providers.

Custom HTTP base URLs are useful for a trusted machine on a private network. Hosted credentials should be sent only over HTTPS. The caller remains responsible for trusting a custom endpoint and for deciding whether prompts may leave the local machine.

Secret handling

API keys are runtime configuration, not arena data.

• Keys may be supplied directly for the current process or read from a provider-specific environment variable.
• A key is placed only in the authentication header for the configured endpoint. It is not added to a URL, prompt, response, match metadata, or snapshot.
• Request diagnostics and exceptions redact authentication and caller-supplied secret headers.
• Result content retention defaults off. Prompts, evaluator criteria and references, answer text, score reasons, and error details are retained only when the caller explicitly opts in. Non-secret evaluator type and judge target provenance remain available for auditability.
• Unknown custom exceptions are recorded with generic details so an endpoint or credential embedded in an exception cannot leak into a result. Structured provider failures may add sanitized status, retryability, and attempt counts.
• Local endpoints work without a real credential unless their operator has enabled authentication.
• OpenAI and OpenRouter use separate credentials. A missing key fails before a hosted request is sent.
• Snapshots can contain sanitized provenance such as provider name, model ID, latency, token usage, finish reason, and a non-secret endpoint label. They must never contain authorization headers or raw keys.

Task taxonomy

The execution boundary is determined by what is needed to run and grade a task, not by the benchmark's name.

Task class	Examples	Execution path	Result recorded by LocalArena
Prompt preference	Two answers judged by a person or a live judge model	Native	left, right, or draw, plus prompt and run provenance
Prompt with a deterministic scorer	Multi-reference match, choice label, extraction, token F1, structured-output validation, or numeric checks	Native	Sample score plus pairwise outcomes derived from comparable model rows
Prompt with a rubric or reference	Free-form answer evaluated live against criteria or a reference answer	Native	The live evaluator's verdict and sanitized evidence
Stateful tool or simulator task	A task that needs browser state, an external simulator, a service topology, or another controlled runtime	External adapter	A normalized verdict imported from the task's authoritative runner
Containerized terminal benchmark	Repository repair, shell interaction, tests, filesystem mutation, environment reset, or image-based grading	Official environment and agent harness through an external adapter	The official harness outcome, task ID, and reproducibility metadata
Historical or third-party result	A result created outside the current run	Explicit import only	Marked as imported; never presented as a live LocalArena execution

Native prompt-scored tasks

A task runs natively when it can be represented by JSON-safe messages and its score can be derived from returned content plus task-local criteria. LocalArena can schedule the pair, call each configured provider, invoke the scorer or live judge, and record the outcome without taking control of a shell or mutable environment. A target marked judge_only can score other targets without appearing as a contestant or receiving an arena rating.

Native task runs should preserve enough provenance to reproduce the comparison: task and prompt identifiers or hashes, provider and model IDs, sampling parameters, judge configuration, timestamps, latency, token usage when reported, finish reason, and the scorer version. Secrets and sensitive prompt content remain opt-in metadata and are excluded by default.

Versioned native task packs preserve that configuration separately from model targets. They require a license field, reject duplicate task IDs, and receive a cross-runtime SHA-256 content identity. A pack may contain inline JSON tasks, native JSONL rows, or simple input/ideal JSONL rows. Pack loading performs no network I/O and executes no callbacks or downloaded code.

Task packs are portable data, not a sandbox boundary. Load only packs you trust or have reviewed. A digest proves identity, not safety: regex and extract patterns run in the host runtime's native regular-expression engine during scoring and a pathological pattern can consume excessive CPU.

External terminal and container tasks

A terminal benchmark is more than a prompt. Its result depends on the exact repository state, container image, installed tools, shell policy, time limits, network policy, environment reset, agent loop, and grader. Reimplementing only its prompt through a chat endpoint would not reproduce the benchmark and must not be described as an official run.

These benchmarks therefore stay under their published environment and agent harness. A narrow external adapter starts or observes that harness and returns a normalized run record to LocalArena. At minimum, that record should include:

• benchmark and harness version;
• task ID and attempt ID;
• contestant and model configuration;
• container image digest or equivalent environment identifier;
• seed, start and finish timestamps, status, and timeout state;
• authoritative score or pass/fail outcome;
• references to logs or grader evidence; and
• the mapping from that outcome to left, right, or draw.

The adapter boundary keeps container lifecycle, tool execution, credentials, and benchmark-specific grading outside the dependency-free rating core. It also allows an official harness to evolve independently while LocalArena continues to schedule comparisons, retain provenance, and calculate standings.

Support rule

Use the native path when the full task is “messages in, content out, score the content.” Use an external adapter when correctness depends on state or tools outside that exchange. In both paths, results come from an execution requested by the caller; no precomputed leaderboard data is treated as a live run.

The statistical policy, ecosystem integration order, and fair comparison with centralized arenas are documented in evaluation methodology and ecosystem.