> ## Documentation Index > Fetch the complete documentation index at: https://gligabao.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Training > How gligabao fine-tunes Gemma-4 on a cloud GPU: the pieces, how a job flows from the training API to the provider, and where each part is documented. `internal/sys/sidecar/`, `internal/api/training/`, `admin_ui/src/training.ts`, `internal/db/relational/`, `sidecar/ai_train/` gligabao can fine-tune Gemma-4 (4B, 12B and 31B, with LoRA adapters) on a cloud GPU and serve the result. The training code is Python and runs on Modal, behind the [Python sidecar](https://gligabao.mintlify.site/operations/sidecar.md): the Go server only ever talks to the sidecar, and the sidecar proxies every request to Modal. The Go server starts jobs through the sidecar, follows their progress (status and loss) and keeps a record of every job it started. The [admin UI](https://gligabao.mintlify.site/getting-started/configuration.md#admin-ui) has a **training** view to run jobs, pick their dataset and watch them. ## The pieces | Piece | Where | Does | | - | - | - | | Admin UI, training view | `admin_ui/src/training.ts` | The datasets (list, inspect, upload), a `Run` button (model size, dataset, steps), the list of jobs, the list of trained models (with a form to run inference on one, and an **Eval** button that has an LLM judge the model against its training dataset: [Evals of trained models](https://gligabao.mintlify.site/evals/training.md)), and for the selected job a live event feed and a loss graph. | | Training endpoints | Go, `internal/api/training/` | `POST` / `GET /v1/gb/training/jobs` (start a job and record it; list jobs), `GET /v1/gb/training/models` and `POST /v1/gb/training/models/{id}/infer` (the trained models, and running a prompt through one), `POST` / `GET /v1/gb/training/datasets` (upload and list the training datasets, found in the training datasets directory by their `dataset.toml` manifests: the Go server alone manages them) and the SSE feed `GET /v1/gb/training/events`. Served whether or not the admin UI is on. | | Training monitor | Go, `internal/api/training/monitor.go` | Follows the sidecar's progress stream of every job, stores the events, keeps the job's row current and forwards the events to the admin UI. | | Go client | Go, `internal/sys/sidecar/` | The only code that knows the sidecar's HTTP API (the Go server never calls Modal itself): `StartTrainingJob`, `StreamTrainingJobEvents`. | | `gb_training_job_runs`, `gb_training_job_events`, `gb_training_datasets`, `gb_trained_models` | Database (SQLite and PostgreSQL) | One row per job started, kept current as it runs; one row per event the sidecar reported for it; one row per uploaded training dataset (the file is kept in the training dataset directory); one row per trained model, the weights a completed job saved. | | Sidecar | Python, `sidecar/` (`routes.py`, `backends/`) | The REST API the Go client calls: `/training/*`, `/infer/*`, `/evals/*`; it has no dataset endpoints. It does no training work: it forwards each request to its backend (`modal`) and streams the answer back. | | Modal web API | Python, `sidecar/ai_train/api/` | The web API of the deployed Modal app, which the sidecar's `modal` backend forwards to. Only the sidecar calls it. | | Training code | Python, `sidecar/ai_train/ai_train_core/` | Runs inside the GPU container: storing and syncing datasets (internal only), the 4B, 12B and 31B pipelines, progress, inference. | | Provider | Python, `sidecar/ai_train/providers/` | Everything cloud-specific (Modal today) behind one interface. | ## How a job starts ``` client ── POST /v1/gb/training/jobs ──► Go server (internal/api/training) │ sidecar.Client.StartTrainingJob ▼ sidecar REST API: POST /training/job_start │ proxied by the modal backend ▼ Modal web API: POST /training/job_start │ spawns the job on the compute provider (Modal) ▼ the TrainingJob travels back the same way │ Go server ◄────────────────────────────────────────────────┘ │ INSERT INTO gb_training_job_runs ▼ 201 with the stored row ``` 1. The endpoint validates the request and, for a `dataset`, loads that dataset's file from the Go server's own storage (an unknown dataset is a `400`). It calls the Go client. 2. The client calls `POST /training/job_start` on the sidecar, the only address the Go server knows. The request carries the dataset file as `dataset_content`. The sidecar forwards the request unchanged to the Modal web API, which stores the dataset on its data volume (internally, there is no dataset endpoint) and spawns the job on the compute provider **without waiting for it** and reports back with a `TrainingJob`; the sidecar returns that answer as it is. A request the Modal web API rejects (for example a bad `dataset_content`) is a `400` that reaches the Go server unchanged. If Modal cannot be reached, the sidecar answers `502`. 3. The server stores the `TrainingJob` as a new row of `gb_training_job_runs`, answers with that row, and hands the job to the training monitor. Details: [Training jobs](https://gligabao.mintlify.site/training/jobs.md). ## How progress comes back ``` training container ── appends events ──► event log of the job (a Modal Dict, kept 7 days) │ Modal web API ◄── GET /training/job_events?from=N (reads the log) ─┘ │ SSE ▼ sidecar (proxies the stream, as it arrives) │ SSE ▼ Go server (training monitor): stores each event, updates the job's row │ SSE: GET /v1/gb/training/events ▼ clients, admin UI: job list, event feed, loss graph ``` The events are status changes (`running`, `training`, `completed` with the final loss, `failed` with the error) and one progress event per trainer log (step, loss, learning rate, epoch). Because the Go server stores them, the loss graph of a finished job is still there after the sidecar's log is gone, and the Go server resumes following unfinished jobs after a restart. The log keeps every event of a job, so a Go server or sidecar that was offline while the job trained gets the events it missed when it reconnects (`from=`). See [Training progress](https://gligabao.mintlify.site/training/jobs.md#training-progress). ## A typical workflow 1. Set up the [sidecar](https://gligabao.mintlify.site/operations/sidecar.md) with your Modal tokens once, and deploy the Modal app: with the **Deploy** button of the admin UI's training view ([Modal app lifecycle](https://gligabao.mintlify.site/training/modal.md)) or `modal deploy app.py` ([Running on Modal](https://gligabao.mintlify.site/training/running.md)). 2. [Add](https://gligabao.mintlify.site/training/datasets.md#adding-a-dataset) one or more datasets: a subdirectory of the training datasets directory (`build/datasets_training/`) with a `dataset.toml` manifest and the JSONL file (or upload a file through `POST /v1/gb/training/datasets` or the admin UI's training view). The server finds them by their manifests, and sends one to the sidecar with the job that uses it. A job needs a dataset; there is no default one. Format and manifest: [Training datasets](https://gligabao.mintlify.site/training/datasets.md). 3. Pick a size ([Model sizes](https://gligabao.mintlify.site/training/models.md)) and a dataset; start with `4b` and 100 to 200 steps to check that the loss falls. 4. Start the job: the `Run` button of the admin UI, the [endpoint](https://gligabao.mintlify.site/training/jobs.md#starting-a-job) it calls, or `modal run app.py --action train` directly (that one bypasses the sidecar and is not recorded by the server). 5. Watch it in the **training** view of the admin UI (feed and loss graph), or through [`GET /v1/gb/training/events`](https://gligabao.mintlify.site/training/jobs#training-progress); the sidecar's own views are `GET /training/job_events` and `GET /training/job_progress` ([sidecar API](https://gligabao.mintlify.site/training/sidecar-api.md)). 6. Run inference on the trained model: pick it in the admin UI's **Trained models** table, or `POST /v1/gb/training/models/{id}/infer` ([Trained models](https://gligabao.mintlify.site/training/trained-models.md)). Every completed job is a new model, so you can train again with other data, steps or size and keep all of them. ## Configuration | Key | Default | Meaning | | - | - | - | | `sidecar_url` (env `SIDECAR_URL`) | `http://127.0.0.1:8787` | Base URL of the sidecar's REST API, which the training endpoints call. It is the only service address the Go server is given: there is no Modal setting on the Go side. A key in the file overrides the environment. | The address is used as given. On the docker compose network that is `http://sidecar:8787` (see [With docker compose](https://gligabao.mintlify.site/operations/sidecar.md#with-docker-compose)). For a job to start, the sidecar at that URL must itself reach the Modal web API: it needs `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`, and the Modal app deployed (from the admin UI, see [Modal app lifecycle](https://gligabao.mintlify.site/training/modal.md), or with `modal deploy app.py`; or `SIDECAR_MODAL_WEB_URL` set, for `modal serve`); see [the sidecar's configuration](https://gligabao.mintlify.site/operations/sidecar.md#configuration). The Go server pings the sidecar's `/health` once on startup; if that fails it runs without a sidecar and the training endpoints answer `503`. The sidecar has no authentication, so keep it on a private network. The settings of the sidecar itself (`SIDECAR_BACKEND`, `SIDECAR_APP_NAME`, the Modal tokens) are in [Python sidecar](https://gligabao.mintlify.site/operations/sidecar.md#configuration); the settings of the Modal app (`SIDECAR_PROVIDER`, `SIDECAR_GPU`, the volumes, ...) are in [Running on Modal](https://gligabao.mintlify.site/training/running.md#configuration). ## In this section * [Training jobs](https://gligabao.mintlify.site/training/jobs.md): the `TrainingJob` struct, the training endpoints, the `gb_training_job_runs` table and the Go client. * [Training datasets](https://gligabao.mintlify.site/training/datasets.md): the dataset format, the training datasets directory and its manifests, listing and uploading on the Go server, how a dataset reaches the sidecar, `TrainingDataset`. * [Model sizes](https://gligabao.mintlify.site/training/models.md): 4B, 12B and 31B, backends, GPUs, cost, LoRA settings, troubleshooting. * [Trained models](https://gligabao.mintlify.site/training/trained-models.md): where a job's weights are saved, the `gb_trained_models` table, how inference picks a model, and how to add an inference endpoint. * [Model deploying](https://gligabao.mintlify.site/training/deploying.md): the deployer hook that receives a trained model's weights as local files, how they are fetched and verified, and the `gb_trained_model_deployments` record. * [Evals of trained models](https://gligabao.mintlify.site/evals/training.md): the **Eval** button: a model is asked its training dataset's questions, one per prompt, and an LLM judges the answers in batches. * [Sidecar training API](https://gligabao.mintlify.site/training/sidecar-api.md): the REST endpoints the sidecar serves and the Modal web API behind them. * [Running on Modal](https://gligabao.mintlify.site/training/running.md): the code layout, the `modal` commands, the sidecar's configuration, tests. * [Modal app lifecycle](https://gligabao.mintlify.site/training/modal.md): deploy, watch (live log), cancel and stop the Modal app and list its volumes from the sidecar and the admin UI, without the CLI. * [Providers and extending](https://gligabao.mintlify.site/training/providers.md): how the provider abstraction works, adding a provider, a model size or a function. ## Not tracked yet * A job's end is only known through the events the training container writes. If the container dies without writing `failed` (killed, out of memory, an image that fails to build), nothing reports it: the Go server marks the job `failed` 3 hours after it started. * Jobs started directly with `modal run` do not go through the sidecar and are not in the database; only jobs started through the training endpoint (or the UI) are. * There is no endpoint to cancel a job. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.