> ## Documentation Index
> Fetch the complete documentation index at: https://gligabao.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval experiments

> Named series of eval runs for comparing models and settings.

An **experiment** is a named series of eval runs. Every time the eval runner (`go run ./evals/knowledge`, `./run_evals.sh`) executes, it records one **run** of an experiment in the **eval** database (`build/gligabao-eval.db`), which keeps the evaluation data apart from the main database and is never reset. Each run works in datasets of its own (`<case>_run<run id>`), so runs do not interfere with each other. A run records which cases of which evals it executed, how each ended, and the IDs of the [ATIF trajectories](https://gligabao.mintlify.site/evals/evals.md) of their calls. Go code lives in `internal/experiments`; the tables are in `internal/db/relational/{sqlite,postgres}/db_schema.go`, the structs in `internal/db/relational/model`.

## Data model

The knowledge-extraction evals and the [evals of trained models](https://gligabao.mintlify.site/evals/training.md) share one set of tables. The **universal** tables hold what every eval (knowledge or training) has and carry a `type` column (`knowledge` or `training`); what only one kind has is in that kind's **metadata** table, one row per row of `gb_experiment_case_runs`. A composite foreign key `(parent id, type)` keeps the type the same down the chain: the database refuses a training run under a knowledge experiment, a result under a run of the other type, or metadata on a result of the other type. Experiment names are unique within a type, so a knowledge and a training experiment can have the same name. Everything on this page is the `knowledge` type; the functions of this package only see those rows.

**Terms.** An *eval case* (a *case*) is one input/output combination, plus metadata, to be evaluated (an input and what is expected or checked). Cases are grouped into an *eval* (a set of cases). A *run* executes some cases once, and a *result* is the outcome of one case in one run: a case has many results, one per run it was in. An *experiment* is a named series of runs. The case definitions live in the eval files; the database holds the names (`gb_experiment_cases_defs`) and the results (`gb_experiment_case_runs`).

Universal tables:

| table | struct | meaning |
| - | - | - |
| `gb_experiments_defs` | `Experiment` | `type`, `name`, `description`, `config` (free-form JSON: the settings under test) |
| `gb_experiment_cases_defs` | `Experiment.Cases` (`CaseRef`) | the cases (`eval` + `case_name`) that any run of the experiment executed; added the first time a case is recorded |
| `gb_experiment_runs` | `ExperimentRun` | one execution: `type`, `experiment_id` (always set for knowledge), `status` (`running`, then: knowledge `passed` or `failed`; training `completed`, `error` or `cancelled`), `params` (how it was invoked), totals (`total_cases`, `passed_cases`, `failed_cases`, `error_cases`, `judged_cases`, `cost_usd`), `started_at`, `finished_at` |
| `gb_experiment_case_runs` | `ExperimentCaseRun` | one case's result in a run: `type`, `eval`, `case_name`, `status` (knowledge: `passed`, `failed`: a check did not hold, `error`: it could not run to the end; training: `judged`: whatever the score, `error`: the model or the judge errored), `error`, `cost_usd`, times |
| `gb_experiment_case_runs_trajectories` | `ExperimentTrajectory` | the ATIF trajectories of a case, in call order (`seq`): `kind` (`remember` / `query`), `trajectory_id`, `agent_name`, `steps`, `cost_usd`, `path` of the JSON file, `artifact_ids` (the IDs, in the case's dataset, of the call's artifacts: the artifact a remember stored, or the artifacts a query's evidence edges were sourced from via the edge provenance links; `experiments.GetRun` and the admin API resolve them to `artifacts` by reading the dataset) |

Metadata tables (the struct fields are those of `ExperimentCaseRun`; the result's `Type` is `knowledge`):

| table | meaning |
| - | - |
| `gb_knowledge_case_meta` | `answer`, `problems` (failed checks), `passed_checks`, `dataset_id` (the dataset the case ran in, which holds the knowledge graph it built; null if it was never created), `steps` (ATIF steps over its trajectories) |
| `gb_training_eval_run_meta` (one row per training run), `gb_training_case_meta` (one row per training case run) | the training eval runs' own columns (trained model, eval, judge, counters) and the per-case outcomes (question, the expected answer, the model's, the judge's score). See [training/EVALS.md](https://gligabao.mintlify.site/evals/training.md#tables) |

Both kinds have **evals**: a case (`case_name`) belongs to an eval (`eval`). Knowledge evals are the files of `build/evals_knowledge/`. Training evals are the files of `build/evals_training/` (named after the training dataset their cases ask about); an *eval run* (one run of the eval against one trained model) is a run with one case run per case (`case_name` is the case's name, as for a knowledge case).

The graph and the artifacts of a case are not copied into the experiment tables: they are read from the dataset the case ran in, via `dataset_id` and `artifact_ids`. Datasets are only ever soft-deleted, so they stay readable; if a dataset (or an artifact) is purged from the eval database, its case shows no graph (or no artifact).

An eval is a file in `build/evals_knowledge/` and its name is the file name without `.json`; a case is one of its cases, so `eval` + `case_name` identifies it. Rows cascade: deleting an experiment removes its runs, cases and trajectory references.

Only the trajectory *reference* is stored in the database; the trajectory is the file `<trajectory_id>.json` in the trajectory directory. The runner stores the files in an `atif/` directory next to the eval database (`build/atif/` by default) so the IDs stay resolvable; `-atif <dir>` changes it. `gb_experiment_case_runs_trajectories.path` records where each file went.

A run is `passed` when every recorded case passed. A case fails when a stored-edge check does not hold or the answer lacks an expected string (`problems` lists them).

The status sets depend on the type (the schema's `CHECK`s enforce the sets). `failed` is a verdict about checks, so only knowledge has it; `error` means "could not be run to the end" in both types:

| | run | case |
| - | - | - |
| knowledge | `running`, then `passed` or `failed` | `passed`, `failed` (a check did not hold), `error` (could not be run to the end) |
| training | `running`, then `completed` (it ran to the end), `error` (the run itself errored) or `cancelled` | `judged`, `error` (the model or the judge errored) |

A training score is not a verdict: a judged case with a score of 0 is still `judged`. A knowledge run is `failed` when at least one case did not pass (a check did not hold, or it errored). The totals of a run are one count per case status and nothing else: `passed_cases`, `failed_cases`, `error_cases`, `judged_cases` (knowledge uses the first three, training the last two; the unused ones are 0). `total_cases` minus the four is the cases not reached (the examples left when a training run is cancelled).

## Runner

```
./run_evals.sh [-experiment name] [-experiment-desc text] [-v] [case-name ...]
```

`-experiment` (default `default`) names the experiment; it is created on first use and later runs with the same name are further runs of it. The runner records into the eval database (`-db`, default `build/gligabao-eval.db`), which it never resets; the datasets of a run are named `<case>_run<run id>`. The runner prints `experiment "<name>" run <id>: <status>, cost $...` at the end.

## From Go

```go theme={null}
t, err := experiments.Start(ctx, d, experiments.Options{
    Name: "baseline", Description: "...", Config: map[string]any{"model": "claude-haiku-4-5"},
    Params: map[string]any{"cases": []string{"job_change"}},
})                                   // get-or-create the experiment, start a run
_, err = t.Record(ctx, experiments.CaseOutcome{
    Eval: "basic", Name: "job_change", Answer: answer,
    Problems: problems,              // failed checks; PassedChecks: the ones that held; Err: why it could not run
    DatasetID: dsID,                 // the dataset the case ran in (holds its graph and artifacts)
    ArtifactIDs: artIDs,             // per trajectory: the IDs of its artifacts in that dataset
    Trajectories: trajs, TrajectoryDir: dir,
})
run, err := t.Finish(ctx)            // totals the cases, sets passed / failed
```

`d` is the eval database's `Relational`. Reading: `experiments.Get(ctx, d, name)` (with `Cases`), `List`, `Runs(ctx, d, experimentID)`, `GetRun(ctx, d, runID)` (its `CaseRuns` and their trajectory references included, with each trajectory's `Artifacts` read from the case's dataset), `CaseRunDataset(ctx, d, caseRunID)` (the dataset a case ran in; `memory.ExportDatasetGraph` reads its graph) and `FindTrajectory(ctx, d, trajectoryID)`, which leads from a trajectory ID back to its case (`CaseRunID`), run and experiment. `Record` fails with `ErrRunNotRunning` after `Finish`.

## Admin UI

With the admin UI enabled (`serve --admin-ui`, see the README), the **Evals** block shows the experiments of the eval database (`--eval-db`; without it the block has nothing to show): totals (experiments, runs, distinct cases, the case runs by their counts, cost), the experiment list (with the runs of each by status) and, by clicking an experiment, its runs. The UI shows the model's own words, not statuses of its own: a run and a case run have the status of `internal/evals_neo/core` (`pending`, `running`, `completed`, `errored`, `cancelled`), a case run also its verdict (`pass` or `fail`, none if it has none) and score, and a run its counts (`total`, `pending`, `running`, `completed`, `errored`, `cancelled`, `passed`, `failed`), which the JSON carries as `status`, `verdict`, `score` and `counts`. The history of a case plots its verdict (or its status when it has no verdict). Clicking a run opens its cases; expanding a case shows its answer, all its checks (passed and failed) and error, plus its ATIF traces. Each case also has a **View graph** button that opens the knowledge graph it built, read live from the case's dataset, in the same graph view as the datasets' **View graph** (`showGraphAt` in `graph.ts`). Each trace also has a **View artifacts** button: the stored artifact for a `remember` call, the source artifacts of the answer for a `query`; they are read from the case's dataset. **View trace** draws a trajectory as a labeled process flow (d3): start → steps (system / user / agent; edges are labeled with the time between steps) → tool calls (`calls`) → tool results (`returns`) → end. Clicking a node shows its full content (message, reasoning, arguments, result, tokens, cost).

The JSON behind it is under `/v1/gb/admin/api/evals`: `GET /` (overview), `GET /experiments/{id}`, `GET /runs/{id}`, `GET /runs/{id}/web-searches` (the [web searches](https://gligabao.mintlify.site/guides/web-search.md#admin-ui) each case made, read from its trajectories), `GET /case_runs/{id}/graph` (the graph of the case's dataset), `GET /experiments/{id}/history?eval=..&name=..` (the results of one case over the experiment's runs) and `GET /trajectories/{trajectory_id}`. A trajectory is served only if an experiment recorded its ID. Its file is read from the `atif/` directory next to the eval database, then from the `path` stored with the reference, so the server must be able to read the directory the evals wrote to.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.