Scorers
Create, read, update, delete, and list evaluation scorers over HTTP.
Scorers define how benchmark rows are judged (for example rubric prompts, structured outputs, or model-based checks).
Base path: /api/v1/scorers. Use HTTP authentication.
List and create
List scorers
GET /api/v1/scorers
Returns JSON with scorer definitions available to the project.
Create scorer
POST /api/v1/scorers
- Body: JSON including name and a config object with evaluation prompt and output_schema (optional temporal). Scoring backend is chosen per benchmark result row at run time, not on the scorer document.
- Returns: JSON including the scorer id for follow-up operations.
Single scorer
Replace {id} with the scorer id.
| Method | Path | Purpose |
|---|---|---|
| GET | /api/v1/scorers/{id} | Fetch the full scorer configuration. |
| PUT | /api/v1/scorers/{id} | Update mutable fields such as name or config fragments supported by the API. |
| DELETE | /api/v1/scorers/{id} | Remove the scorer from the project. |
Design tips
- Keep scorer prompts small and explicit; large prompts increase latency and cost.
- Version scorers by creating new records when you make breaking config changes, so historical benchmark runs remain interpretable.
- Pair with Running benchmarks for operational guidance.