Skip to content

Getting started: first live score in one minute

LocalArena can go from an installed Python package to a live, scored model response with one command:

localarena quickstart PROVIDER MODEL_ID

That command makes a real Chat Completions request, scores the returned text, writes localarena-results.json, and renders localarena-report.html. It does not use a recorded answer or cached leaderboard result.

What the one-minute path includes

The one-minute clock starts when:

  • LocalArena is installed;
  • the selected endpoint is reachable;
  • a local model is already downloaded and available to its server; or
  • a hosted provider key and accessible model are ready.

Model downloads are deliberately outside that clock. Even small downloads can take several minutes, and LocalArena never downloads, pulls, loads, or starts a model on the caller's behalf. A provider's first cold load can also take longer than one minute.

The quickstart uses one fixed smoke-test task with a deterministic scorer. It asks the model for 42 and passes when the returned text contains 42. This verifies the full live path—provider request, response parsing, scoring, result serialization, and report rendering—without pretending to be a meaningful benchmark. The 512-token output ceiling leaves room for models that deliberate before emitting visible text; a model that follows the prompt normally stops after 42. Override it with --max-tokens when an endpoint has a stricter limit.

Install and verify the CLI

Python 3.10 or newer is required. A virtual environment avoids externally-managed Python errors.

macOS or Linux:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade localarena
localarena providers

Windows PowerShell:

py -m venv .venv
.venv\Scripts\Activate.ps1
py -m pip install --upgrade localarena
localarena providers

The last command should print:

llamacpp
ollama
lmstudio
openrouter
openai
custom

If the localarena console command is not on PATH, use python -m localarena on macOS/Linux or py -m localarena on Windows. Every command below accepts that substitution.

Pick a provider

Provider profile Runs where Ready before the timer starts Default base URL
llamacpp Local or self-hosted llama-server is serving a downloaded GGUF with a stable alias http://127.0.0.1:8080/v1
ollama Local Ollama is running and the pulled model is loaded and warm http://127.0.0.1:11434/v1
lmstudio Local LM Studio's server is running and the model is available http://127.0.0.1:1234/v1
openrouter Hosted OPENROUTER_API_KEY is set and the account can use the model https://openrouter.ai/api/v1
openai Hosted OPENAI_API_KEY is set and the project can use the model https://api.openai.com/v1
custom Local or hosted A trusted compatible endpoint is reachable and the selected model is available and warm Required explicitly

localarena models PROVIDER asks the configured endpoint for its exact model IDs. Model discovery is helpful but optional: if an endpoint implements POST /chat/completions without GET /models, use a known model ID directly. Listing an ID confirms discovery, not necessarily that the account can run that model.

llama.cpp

Before starting the timer, install llama.cpp, download a chat-capable GGUF, and start the server with a stable alias:

llama-server -m /absolute/path/to/model.gguf --alias local-model --host 127.0.0.1 --port 8080

Windows PowerShell:

llama-server -m "C:\models\model.gguf" --alias local-model --host 127.0.0.1 --port 8080

The alias matters. Without it, llama.cpp normally reports the model's file path as the model ID. A compatible chat template is also required for useful Chat Completions output.

In a second terminal:

localarena models llamacpp --timeout 120
localarena quickstart llamacpp local-model

For a server on another trusted machine, override the complete API base URL:

localarena quickstart llamacpp local-model --base-url http://SERVER:8080/v1

If server authentication is enabled, put the key in a runtime environment variable and name that variable rather than putting the key on the command line.

macOS or Linux:

printf 'llama.cpp API key: '
read -r -s LOCALARENA_LLAMACPP_API_KEY
printf '\n'
export LOCALARENA_LLAMACPP_API_KEY
localarena quickstart llamacpp local-model --api-key-env LOCALARENA_LLAMACPP_API_KEY

Windows PowerShell:

$secret = Read-Host "llama.cpp API key" -AsSecureString
$credential = [System.Net.NetworkCredential]::new("", $secret)
$env:LOCALARENA_LLAMACPP_API_KEY = $credential.Password
Remove-Variable secret, credential
localarena quickstart llamacpp local-model --api-key-env LOCALARENA_LLAMACPP_API_KEY

See the official llama.cpp server documentation for installation, model loading, aliases, chat templates, and server authentication.

Ollama

Before starting the timer, install Ollama and launch ollama serve if the Ollama application or service is not already running. Then pull and warm a model. This small example is about 523 MB:

ollama pull qwen3:0.6b
ollama run qwen3:0.6b "Reply with only READY."
ollama ps

The warm-up prompt triggers the initial load; start the timer only after ollama ps shows the model, then run:

localarena models ollama
localarena quickstart ollama qwen3:0.6b

Ollama's local API does not require a credential by default. LocalArena calls it directly and does not need the placeholder key used by some client SDKs.

See Ollama's official CLI reference, OpenAI compatibility guide, and qwen3:0.6b model page.

LM Studio

Before starting the timer, install and launch LM Studio at least once. Download a model in the application, or use the CLI:

lms get ibm/granite-4-micro
lms ls

Copy the model key reported by lms ls, give the loaded model a stable API identifier, and start the server:

lms load MODEL_KEY --identifier local-model
lms server start --port 1234

The same operations are available in LM Studio's Developer page. Once ready:

localarena models lmstudio
localarena quickstart lmstudio local-model

Authentication is disabled by default. If you enable it in LM Studio, load the generated token into a runtime variable and pass only the variable name.

macOS or Linux:

printf 'LM Studio API token: '
read -r -s LM_API_TOKEN
printf '\n'
export LM_API_TOKEN
localarena quickstart lmstudio local-model --api-key-env LM_API_TOKEN

Windows PowerShell:

$secret = Read-Host "LM Studio API token" -AsSecureString
$credential = [System.Net.NetworkCredential]::new("", $secret)
$env:LM_API_TOKEN = $credential.Password
Remove-Variable secret, credential
localarena quickstart lmstudio local-model --api-key-env LM_API_TOKEN

See LM Studio's official API quickstart, lms reference, model-list endpoint, and authentication guide.

OpenRouter

Hosted calls send the smoke-test prompt to OpenRouter and the selected model provider. Account limits and model pricing apply.

Create a key using OpenRouter's account controls, then load it into the current shell without placing the secret in shell history.

macOS or Linux:

printf 'OpenRouter API key: '
read -r -s OPENROUTER_API_KEY
printf '\n'
export OPENROUTER_API_KEY

Windows PowerShell:

$secret = Read-Host "OpenRouter API key" -AsSecureString
$credential = [System.Net.NetworkCredential]::new("", $secret)
$env:OPENROUTER_API_KEY = $credential.Password
Remove-Variable secret, credential

The official free router is convenient for a connectivity smoke test:

localarena models openrouter --timeout 120
localarena quickstart openrouter openrouter/free --timeout 120

openrouter/free can resolve to different models over time. For reproducible comparisons, select a concrete provider/model slug returned by localarena models openrouter.

See OpenRouter's official quickstart and model guide.

Anthropic models through OpenRouter

LocalArena's native transport currently targets the Chat Completions protocol. To evaluate an Anthropic model today, choose a concrete Anthropic model slug returned by OpenRouter:

localarena models openrouter
localarena quickstart openrouter ANTHROPIC_MODEL_SLUG

The direct Anthropic Messages API is not presented as a built-in profile because it is a different wire protocol. A gateway that deliberately exposes a compatible Chat Completions endpoint can instead use the custom profile.

OpenAI

Hosted calls send the smoke-test prompt to OpenAI and may incur usage charges. Create a project key and follow OpenAI's official API key setup guide. Load the key into the current shell without placing the secret in shell history.

macOS or Linux:

printf 'OpenAI API key: '
read -r -s OPENAI_API_KEY
printf '\n'
export OPENAI_API_KEY

Windows PowerShell:

$secret = Read-Host "OpenAI API key" -AsSecureString
$credential = [System.Net.NetworkCredential]::new("", $secret)
$env:OPENAI_API_KEY = $credential.Password
Remove-Variable secret, credential

Discover the models available to the authenticated project, then run one that supports Chat Completions:

localarena models openai --timeout 120
localarena quickstart openai gpt-4.1-mini-2025-04-14 --timeout 120

If that snapshot is not returned for the project, substitute an accessible Chat Completions model ID from the discovery command. The authenticated /v1/models response is more authoritative for project access than a general catalog page.

See OpenAI's official model catalog and Chat Completions reference.

Custom compatible endpoint

Use custom for any trusted endpoint with a compatible POST /chat/completions response shape. GET /models is optional and needed only for localarena models. The base URL must include its API prefix, normally /v1. Before starting the timer, make sure the chosen model is available and warm it first if the endpoint loads models lazily.

Unauthenticated loopback example:

localarena models custom --base-url http://127.0.0.1:9000/v1
localarena quickstart custom model-id --base-url http://127.0.0.1:9000/v1

For a Bearer-authenticated hosted endpoint, load the key without placing it in shell history.

macOS or Linux:

printf 'Custom endpoint API key: '
read -r -s LOCALARENA_CUSTOM_API_KEY
printf '\n'
export LOCALARENA_CUSTOM_API_KEY
localarena quickstart custom model-id --base-url https://inference.example.com/v1 --api-key-env LOCALARENA_CUSTOM_API_KEY

Windows PowerShell:

$secret = Read-Host "Custom endpoint API key" -AsSecureString
$credential = [System.Net.NetworkCredential]::new("", $secret)
$env:LOCALARENA_CUSTOM_API_KEY = $credential.Password
Remove-Variable secret, credential
localarena quickstart custom model-id --base-url https://inference.example.com/v1 --api-key-env LOCALARENA_CUSTOM_API_KEY

Use HTTPS whenever a credential crosses the machine boundary. If authentication uses a non-Bearer header, move to a reusable JSON configuration and use headers_env:

{
  "name": "Authenticated custom endpoint",
  "models": [
    {
      "name": "custom-model",
      "provider": "custom",
      "base_url": "https://inference.example.com/v1",
      "model": "model-id",
      "headers_env": {
        "X-Service-Token": "LOCALARENA_CUSTOM_HEADER"
      }
    }
  ],
  "tasks": [
    {
      "id": "arithmetic",
      "prompt": "Reply with exactly 42.",
      "evaluator": {
        "type": "contains",
        "expected": "42"
      }
    }
  ]
}

Run it with:

localarena run evaluation.json --output results.json --report report.html

Read the quickstart result

A passing run normally looks like:

[1/1] MODEL_ID × arithmetic-smoke-test: ok

 1. MODEL_ID: score=1.000 decision=1.000 arena=1000.0 95%CI=1000.0–1000.0 inconclusive errors=0
Results: localarena-results.json
Report: localarena-report.html

Interpret the fields separately:

  • status: ok means generation and scoring completed.
  • score=1.000 means the answer passed the smoke-test scorer.
  • score=0.000 errors=0 is a valid live evaluation in which the model did not pass. It is not a transport failure.
  • decision=1.000 includes every expected scored row in the denominator. A failed scored row counts as zero instead of disappearing from the mean.
  • errors=1 means the provider call or scoring process failed; the command exits with status 2.
  • arena=1000.0 ... inconclusive is expected for a single contestant because there is no head-to-head evidence. Multi-model runs use order-independent Bradley–Terry ratings and a task-clustered 95% confidence interval.

The result and report exclude prompts, answers, evaluator details, and detailed errors by default. If the content is safe to store and you need to inspect the answer:

localarena quickstart ollama qwen3:0.6b --include-content

Both default output paths are replaced on a subsequent quickstart. Use --output and --report to preserve named runs:

localarena quickstart ollama qwen3:0.6b --output smoke.json --report smoke.html

Move from a smoke test to a real comparison

Create evaluation.json with two already-available Ollama models and several scored tasks:

{
  "name": "Local comparison",
  "concurrency": 2,
  "models": [
    {
      "name": "qwen-small",
      "provider": "ollama",
      "model": "qwen3:0.6b",
      "parameters": {
        "max_tokens": 64
      }
    },
    {
      "name": "gemma-small",
      "provider": "ollama",
      "model": "gemma3:270m",
      "parameters": {
        "max_tokens": 64
      }
    }
  ],
  "tasks": [
    {
      "id": "arithmetic",
      "prompt": "Return only the result of 6 multiplied by 7.",
      "evaluator": {
        "type": "numeric",
        "expected": 42,
        "tolerance": 0
      }
    },
    {
      "id": "capital",
      "prompt": "Reply with exactly Paris.",
      "evaluator": {
        "type": "exact",
        "expected": "Paris"
      }
    },
    {
      "id": "structured",
      "prompt": "Return only JSON with an answer field equal to 42.",
      "evaluator": {
        "type": "json",
        "compare": true,
        "expected": {
          "answer": 42
        }
      }
    }
  ]
}

Run the complete model-by-task matrix:

localarena run evaluation.json --output results.json --report report.html

Local and hosted targets can appear in the same models array. Each entry keeps its own provider, model ID, base URL, credential variable, request policy, and generation parameters. The complete six-profile reference configuration shows all available fields, but it is not expected to run unchanged because it assumes all six endpoints and credentials exist at once.

Put reusable tasks in a pack

Model/provider settings and tasks can evolve independently. The executable task-pack example demonstrates accepted answers, multiple choice, answer extraction, and partial-credit token F1.

Reference a pack from evaluation.json using a path relative to that config:

{
  "name": "Packed local comparison",
  "models": [
    {
      "name": "qwen-small",
      "provider": "ollama",
      "model": "qwen3:0.6b",
      "parameters": {
        "max_tokens": 64
      }
    }
  ],
  "task_files": [
    "examples/task-pack.localarena"
  ]
}

Run it with the same command:

localarena run evaluation.json --output results.json --report report.html

A native pack must declare schema_version, name, version, license, and at least one task. LocalArena validates all fields, rejects duplicate IDs, computes a portable SHA-256 identity, and records safe pack version/digest provenance in the result. JSONL files are also accepted. Rows with input and ideal are detected as simple Evals-style tasks and run live; no recorded answers are imported.

Use a separate live judge

A model with "judge_only": true can score candidates without becoming a ranked contestant:

{
  "name": "Locally judged answer",
  "models": [
    {
      "name": "candidate",
      "provider": "ollama",
      "model": "qwen3:0.6b",
      "parameters": {
        "max_tokens": 128
      }
    },
    {
      "name": "judge",
      "provider": "ollama",
      "model": "gemma3:1b",
      "judge_only": true,
      "parameters": {
        "max_tokens": 128
      }
    }
  ],
  "tasks": [
    {
      "id": "explanation",
      "prompt": "Explain in one sentence why the sky appears blue.",
      "evaluator": {
        "type": "model_judge",
        "model": "judge",
        "rubric": "Score factual correctness and directness.",
        "reference_answer": "Shorter blue wavelengths are scattered more strongly by Earth's atmosphere.",
        "pass_threshold": 0.7
      }
    }
  ]
}

A live judge receives the task, rubric, reference answer, and candidate answer. If the judge is hosted, that content leaves the machine, adds another billable request, and is subject to the judge provider's data handling. Default output redaction does not prevent network disclosure.

Privacy and secrets

There are three different boundaries:

  1. Saved output: prompts, answers, evaluator content, score reasons, and detailed errors are omitted from result JSON and HTML reports by default.
  2. Provider request: every selected provider receives the task messages. A hosted endpoint therefore receives the prompt even when saved output is redacted.
  3. Input configuration: the JSON file itself contains prompts, reference answers, base URLs, and environment-variable names. Treat it according to that content.

Credentials and configured headers are never serialized into a run. Keep raw keys out of JSON, command-line arguments, source control, .env files that might be committed, retained reports, and bug reports. Use api_key_env and headers_env for reusable configurations.

Privacy-safe results still retain non-secret provenance such as provider profile, model ID, timing, token use when supplied, finish reason, evaluator type, and live-judge identity.

Containers, WSL, and remote servers

127.0.0.1 means the network namespace in which LocalArena runs. A CLI inside Docker, a devcontainer, or WSL may not reach a provider running on the host. Supply a reachable address explicitly, for example with Docker Desktop:

localarena quickstart ollama qwen3:0.6b --base-url http://host.docker.internal:11434/v1

Linux containers may require an explicit host-gateway mapping. WSL may require the Windows host address. Do not bind an unauthenticated model server to a public interface. When credentials leave loopback, use HTTPS and a server you trust.

Troubleshooting

Symptom What to check
localarena: command not found Activate the virtual environment or use python -m localarena.
Connection refused Start the provider server and confirm the port from the same network namespace as the CLI.
404 from models or chat/completions The base URL usually must end in /v1. Avoid both a missing prefix and a doubled /v1/v1.
API key required Set the profile's key variable in the current process, or pass the correct variable name with --api-key-env.
401 or 403 Confirm the key belongs to the selected service, has model access, and was not accidentally sent to an overridden endpoint.
Model not found Copy an exact ID from localarena models; for llama.cpp use --alias, and for LM Studio load with --identifier.
Model discovery fails but chat is supported Skip localarena models and use the endpoint's documented model ID directly.
Invalid chat completion Confirm the endpoint really implements the compatible response envelope and that a local model has a usable chat template.
Timeout or a run longer than one minute Increase --timeout, wait for a cold model to load, or choose a smaller already-loaded model.
score=0.000 errors=0 The request worked but the answer missed the scorer. Re-run with --include-content only when the prompt and answer are safe to store.
errors=1 and exit status 2 Inspect provider logs and sanitized run metadata; test model discovery and endpoint authentication separately.

Next steps