> ## Documentation Index > Fetch the complete documentation index at: https://gligabao.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Sidecar training API > The REST endpoints of the sidecar web API: training jobs and progress, inference and eval examples. There are no dataset endpoints. `sidecar/routes.py`, `sidecar/backends/`, `sidecar/ai_train/api/api.py`, `sidecar/ai_train/api/models.py` There are two layers with the same routes. The **sidecar** (`sidecar/routes.py`, see [the service](https://gligabao.mintlify.site/operations/sidecar.md#the-service)) is the REST API the Go server calls; it does not implement the routes, it forwards each one unchanged to its backend and streams the answer back. The **Modal web API**, described on this page, is what the `modal` backend forwards to: a FastAPI app created by `create_api` in `api/api.py` and served by `modal deploy app.py` (or `modal serve app.py` in dev, with `SIDECAR_MODAL_WEB_URL`). It is provider neutral: everything provider-specific (spawning a job, calling a function, committing and reloading volumes) is injected as callables by `app.py`. Only the sidecar calls it; the Go server never does. The request and answer formats below are the same on both layers, since the sidecar changes nothing. What the sidecar adds: `GET /health` (answered by the sidecar itself, not forwarded), the routes that manage the Modal app itself (`/backends/...`: status, deploy, cancel, stop, volumes; see [Modal app lifecycle](https://gligabao.mintlify.site/training/modal.md#the-api)) and `502 {"error": ...}` when the backend cannot be reached (for `modal`, also when the deployed app's web URL cannot be found). | Endpoint | Used by the Go server (through the sidecar) | Purpose | | - | - | - | | `POST /training/job_start` | yes | Start a training job. | | `GET /training/job_progress` | no | The progress entries of a job, as one JSON answer. | | `GET /training/job_events` | yes | The events of a job as Server-Sent Events. | | `POST /infer/query` | no | Generate with a fine-tuned adapter (`model_path` picks it). | | `GET /models/file` | yes | Stream one file of a trained model, for a [deployment](https://gligabao.mintlify.site/training/deploying.md#fetching-the-weights). | | `POST /evals/add_to_dataset` | no | Add an eval example. | | `GET /evals/get_dataset_count` | no | Count the eval examples. | | `GET /healthz` | no | Liveness of the Modal web API: `{"status": "ok", "timestamp": ...}`. Not a sidecar route (the sidecar's own is `GET /health`), so it is not reachable through the sidecar. | There are **no dataset endpoints** (`/datasets/upload` and `/datasets/list` were removed in gligabao 0.31.0): training datasets belong to the Go server, which hands one to the sidecar inside `POST /training/job_start`; see [Training datasets](https://gligabao.mintlify.site/training/datasets.md#how-a-dataset-reaches-the-sidecar). The routes were put under these prefixes in gligabao 0.26.2. They used to be `/start_training_job`, `/get_training_progress`, `/generate_response`, `/add_to_eval_dataset`, `/get_eval_dataset_count`, `/upload_dataset` and `/list_datasets`. ## `POST /training/job_start` Body `{dataset, num_train_steps?, model_size?, dataset_content?}` (defaults `100` and `4b`; `dataset` is required, there is no default dataset). `dataset_content` is the dataset's jsonl file, base64; when it is given the sidecar stores it on the data volume as `dataset` before it spawns the job (the Go server sends it for every job that uses one of its datasets). The sidecar generates the job's ID, spawns the job and answers with a [`TrainingJob`](https://gligabao.mintlify.site/training/jobs#the-trainingjob-struct). `400` with `{"error": ...}` for a `model_size` that is not `4b`, `12b` or `31b`, for a `num_train_steps` that is not a positive integer, for a missing `dataset`, for a `dataset` that is not a plain `*.jsonl` file name, for a `dataset_content` that is not valid base64, is empty or has no `dataset`, and for a `dataset` without `dataset_content` that is not on the data volume. It returns as soon as the job is spawned. ```sh theme={null} curl -X POST $URL/training/job_start -H "Content-Type: application/json" \ -d '{"num_train_steps": 500, "model_size": "12b", "dataset": "my.jsonl"}' ``` ## Progress events The training run appends one JSON object per event to the job's **event log**: entries in a Modal Dict named `-events`, which drops each entry 7 days after its last use (the Go database is the permanent record). There is no volume, commit or reload. An event is one request to the Dict, tried again if it fails; a `progress` or `log` event that still fails is skipped, a `status` event that fails raises. An event without a `type` is a progress event. Status events, `{"type": "status", "status": ..., "timestamp": ...}`: | `status` | When | Extra fields | | - | - | - | | `running` | The training container started. | `model_size`, `num_train_steps` | | `training` | The trainer began. | `max_steps` | | `completed` | Training finished and the adapter was saved. | `final_loss`, `model` (the saved weights, see [Trained models](https://gligabao.mintlify.site/training/trained-models.md)) | | `failed` | Anything raised (a bad dataset, out of memory, ...). | `error` | | `cancelled` | The job was asked to stop (see `POST /training/job_cancel` below). | `error` (where it stopped) | `completed`, `failed` and `cancelled` end the job. Log events, `{"type": "log", "message": ...}`, say what the job is doing: importing the libraries, the dataset loaded, the model loaded, the trainer set up, the adapters saved. Progress events, one per trainer step (`logging_steps=1`): | Field | Meaning | | - | - | | `step` | The global training step. | | `loss` | The training loss (`null` for logs without one, and for a loss that is not finite: NaN is not valid JSON). | | `learning_rate` | The current learning rate. | | `epoch` | The epoch, fractional. | | `timestamp` | Unix time of the entry. | | `elapsed_seconds` | Seconds since training began. | ## `POST /training/job_cancel` Body `{"job_id": ...}`. Asks the job to stop: the web container sets the job's stop flag in the Dict, and the training container looks at it between steps (at most every 2 seconds) and between the stages of its setup. It then ends the job with a `cancelled` event, without saving the adapters. So the stop takes effect at the next step, or, during the model load, when the load is done; it is not instant. Answers `200 {"job_id": ..., "status": "stopping"}`, `400` for a bad `job_id`, `404` for an unknown job (one started through `job_start` is known at once, before its container runs) and `409` for one that is over already. The web container writes no events here. ## `GET /training/job_events` `?job_id=[&from=]` (`400` without a valid `job_id`). A Server-Sent Events stream of the job's events as they are written, from the event with index `n` (default 0): ``` id: 4 event: progress data: {"type":"progress","step":20,"loss":1.234,"learning_rate":0.0002,"epoch":0.3,"timestamp":1792169586.9,"elapsed_seconds":42.1} ``` `id` is the event's index in the event log, `event` its type. While nothing new arrives, `: keepalive` comment lines are sent. The web container looks for new events every second. The stream ends after the `completed`, `failed` or `cancelled` event, or after 5 minutes whatever happened; a reader that wants more reconnects with `from=`, and gets every event it missed, however long it was away (within the 7 days). If the job has no events yet (it is still waiting for a GPU) the stream just waits. **Watchdog.** A container that is killed (out of memory, a timeout) cannot report its end. While a stream is idle, every 15 seconds the web container asks Modal about the job's call (its id is stored by `job_start`); if the call is over and the log has no final event, it writes a `failed` event with Modal's error, and the stream delivers it. The Go monitor therefore learns of such a failure at once, not after its 3 hour limit. The Go client reads it with `StreamTrainingJobEvents`, see [Training progress](https://gligabao.mintlify.site/training/jobs.md#training-progress). ## `GET /training/job_progress` `?job_id=` (`400` without a valid one): all the events of that job's event log so far, as `{"progress": [...]}` (an empty list when there are none). Until 0.44.0 it also answered without `job_id` with the newest job. Unlike the stream it answers once with everything so far. There is no Go client method for it; the Go server uses the stream. ## `POST /infer/query` Body `{prompt, max_tokens?, temperature?, model_size?, model_path?}` (defaults `1024`, `0.7` and `4b`): runs a fine-tuned adapter and answers with the result of the inference function. `model_path` (`trained/`, the `path` of a [trained model](https://gligabao.mintlify.site/training/trained-models.md)) picks the weights; `model_size` must be the size they were trained for. Without `model_path` the adapter that versions before 0.38.0 saved per size (`/models/finetuned_model_`) is used. `400` for an invalid `model_size` or `model_path`; an adapter that is not on the volume answers `{"error": ...}`. The models volume is reloaded first so a newly trained adapter is seen. The answer also has `cold_start` (`true` when this request had to load the model), `load_seconds` (how long that took; the container start is not included) and `input_tokens` / `output_tokens`: the container keeps the model loaded and stays up for `SIDECAR_INFER_IDLE_TIMEOUT_S` (default 60 seconds) after its last request, so only the first request pays the load (see [Warm inference containers](https://gligabao.mintlify.site/training/modal.md#warm-inference-containers); what a cold request does in the container is in [Cold and warm starts](https://gligabao.mintlify.site/training/modal.md#cold-and-warm-starts)). ## `GET /models/file` `?model_path=trained/&file=` answers the raw bytes (`application/octet-stream`) of one file of a trained model: `file` is a `path` of the model's `files` (relative to its directory, nested paths allowed). It exists so the Go server can fetch a model's weights to its own disk for a [deployment](https://gligabao.mintlify.site/training/deploying.md#fetching-the-weights), since the web container mounts neither the models volume nor the HF cache. `model_path` must have the shape `trained/` and `file` must be a relative path of plain segments (no `..`, no absolute path): else `400`. A volume or file that does not exist is `404`; a backend without such storage is `404` too; a failure reading it is `502` (`{"error"}`). The Modal backend (`backends/modal_backend.py`, `model_file`) runs `python -m providers.modal.read_model_file ` (`ai_train/providers/modal/read_model_file.py`, read only, like the volume listing) in a subprocess and streams its stdout, so a multi-gigabyte file is never held in memory. The first chunk is read before the answer starts, so a missing file is a clean `404`; a failure after that aborts the response, which the Go side sees as a short body. In Go, `sidecar.Client.DownloadModelFile(ctx, modelPath, file, w)` streams it into a writer, and `internal/training.FetchModelFiles` downloads every file of a model into a directory through a temporary name, checks the size and the sha256 recorded in `gb_trained_models.files`, and renames it into place; a file already there with the right size and sha256 is not downloaded again. ## Eval examples `POST /evals/add_to_dataset` takes `{prompt, response, notes?}` (`400` if the prompt or response is missing), appends it with a timestamp to `eval_dataset.jsonl` on the eval volume and commits the volume. `GET /evals/get_dataset_count` answers `{"count": n}`, the number of lines of that file. These examples are collected for later evaluation; they are not training data. ## Tests `sidecar/ai_train/test_ai_train_core.py` tests the API with FastAPI's test client and injected fakes (no Modal, GPU or ML libraries): `cd sidecar/ai_train && python -m unittest test_ai_train_core -v`. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.