Model deploying

How a trained model is handed to your code: the deployer hook, how gligabao fetches and verifies the weights, the gb_trained_model_deployments record, the API and the limits.

internal/training/deploy.go, internal/training/db/, internal/api/training/models.go, internal/sys/sidecar/, gligabao.go

Deploying a trained model means handing its weights to code of yours. gligabao does the part that is the same for everybody, getting the weights from where they were trained onto the local disk and checking them, and calls a hook with the result: a deployer, a Go callback registered by the program that embeds gligabao. What the deployer does with the files (upload them to a service, copy them to a bucket, convert them to another format, …) is its own business; gligabao has no built-in destination.

This page describes how deploying works inside and the hook’s contract. The hook’s entry in the list of all hooks is Hooks; the trained models themselves (the gb_trained_models table, where the weights live) are in Trained models.

Deploying is part of the training API: it is there only when the server runs in training mode (--training) and, to fetch the weights, with an active sidecar.

The flow

POST /v1/gb/training/models/{id}/deployments   {"deployer": "<name>", "options": {...}}
  1. look up the trained model {id}                          404 if unknown
  2. find the deployer by name, check the options            400 if none is registered, the name is unknown or the options are bad
  3. record the deployment in gb_trained_model_deployments           status "running"
  4. fetch the model's files from the sidecar into Dir       each file verified (size, sha256)
  5. call Deploy(ctx, DeployTrainedModelRequest)             your code; the request waits for it
  6. update the record                                       status "completed" (ref, details) or "failed" (error)
  7. emit the system event training.model_deploy.completed / .failed
  8. answer 200 with the stored deployment                       502 if step 4 or 5 failed

The whole thing is synchronous: the HTTP request is open while the files are downloaded and while Deploy runs, which can take minutes for a large model. Callers must not use a short timeout. If the client disconnects, the request’s ctx is cancelled, which Deploy should notice; the deployment is then recorded as failed.

Registering a deployer

Deployers are passed in ServeOptions.TrainedModelDeployers (there is no command-line or configuration-file way to register one, since a deployer is code):

import "github.com/deepblue-phoenix/gligabao"

err := gligabao.Serve(ctx, cfg, gligabao.ServeOptions{
    TrainedModelDeployers: []gligabao.TrainedModelDeployer{{
        Name: "bucket",
        Description: "Uploads the model's files to the releases bucket.",
        Deploy: func(ctx context.Context, req gligabao.DeployTrainedModelRequest) (gligabao.DeployTrainedModelResult, error) {
            dest := fmt.Sprintf("models/%s/deploy-%d", req.Model.JobID, req.DeployID)
            for _, f := range req.Model.Files {
                // f.Path is relative to req.Dir; the file is complete and verified
                if err := upload(ctx, filepath.Join(req.Dir, filepath.FromSlash(f.Path)), dest+"/"+f.Path); err != nil {
                    return gligabao.DeployTrainedModelResult{}, err
                }
            }
            return gligabao.DeployTrainedModelResult{
                Ref:     dest,
                Details: map[string]any{"files": len(req.Model.Files)},
            }, nil
        },
    }},
})

Type	Fields
gligabao.TrainedModelDeployer	Name, Description (optional, shown in the admin UI), Deploy
gligabao.DeployTrainedModelFunc	func(ctx context.Context, req DeployTrainedModelRequest) (DeployTrainedModelResult, error), the type of Deploy
gligabao.DeployTrainedModelRequest	DeployID, Model, Dir, Options
gligabao.DeployTrainedModelResult	Ref, Details

What Deploy receives

Field	Meaning
DeployID	the ID of this deployment (gb_trained_model_deployments.id): unique to each call, so use it when the deployer needs a unique name for what it creates
Model	the trained model (gligabao.TrainedModel): ID, Name, JobID, BaseModel, ModelSize, Dataset, NumExamples, NumTrainSteps, FinalLoss, Params, Path, and Files (gligabao.TrainedModelFile: Path, SizeBytes, SHA256), …
Dir	the absolute path of a local directory that holds the model’s files, at the paths Model.Files lists, relative to it
Options	the options object of the request, unchanged, as a map[string]any (nil if the request gave none)

What Deploy returns

• Ref: identifies what the deployment made, in the deployer’s own terms (the name of a model created elsewhere, an object-store path, …). A string, stored and returned with the deployment.
• Details: anything else worth keeping; a map[string]any that must be JSON-encodable (a deployer that returns details that are not is treated as failed). May be nil.
• A non-nil error, or a panic (it is caught), makes the deployment failed; the error text is stored and returned.

Rules of the hook

• Read only. Dir is shared with later deployments of the same model. Deploy must not modify or delete the files in it; copy, archive or convert into a place of its own.
• Concurrent. Deploy requests are handled concurrently, also several for the same model, so Deploy must be safe for concurrent use. The files of one model are fetched once even then (see below).
• Honour ctx. It is the request’s context: it is cancelled if the client goes away. The fetch has its own limit of 30 minutes; Deploy has none besides ctx.
• No retries. gligabao runs Deploy once per request. A failed deployment stays failed; ask for a new one to try again (the files are already on the disk by then).
• No sidecar. The deployer never talks to the sidecar; it finds the weights on the local disk.

Validation at startup

Serve checks the registered deployers and returns an error (invalid trained model deployer: ...) for any of these, before the server starts:

• more than 20 deployers;
• a Name that is empty, longer than 64 characters, or not letters, digits, _, - and . starting with a letter or digit;
• two deployers with the same Name;
• a deployer with no Deploy function.

Deployers are sorted by name; each one registered is logged at startup. With no deployer registered, every deployment request is answered 400.

Fetching the weights

The weights of a trained model live on the sidecar’s storage (Modal: the models volume), not on the server. Before it calls the deployer, gligabao copies them to the server’s disk. This is the part of deploying that is the same for every deployer.

• Where. <trained model deploy dir>/<job id>, an absolute path (Dir). The deployment directory is serve --trained-model-deploy-dir (ServeOptions.TrainedModelDeployDir); the default is a trained_models directory next to the database file (build/trained_models). There is one subdirectory per training job, so models never mix. A job ID that is not usable as a directory name is refused (400).
• What. Every file in the model’s files (the list recorded when the job completed), at its recorded relative path; nested paths are allowed. A path that is absolute, contains .. or uses a backslash is refused.
• How. One GET /models/file?model_path=trained/<job id>&file=<path> request per file to the sidecar, which streams the raw bytes out of the volume (see Sidecar training API); the file is never held in memory.
• Verified. Each file must have exactly the recorded size and the recorded sha256 (a model with no sha256 recorded for a file is checked by size only). A mismatch fails the deployment; nothing is deployed from unverified bytes.
• Atomic. A file is downloaded under a temporary name (.fetch-*, in its own directory) and renamed to its real name only when it is complete and verified, so the real names never hold half a file, even if the server dies halfway.
• Reused. A file that is already in Dir with the right size and sha256 is kept and not downloaded again; only missing or damaged files are fetched. Deploying the same model a second time therefore costs only the verification.
• Serialized. Fetches into one directory are serialized: two simultaneous deployments of the same model do not download it twice; the second waits and then finds the files in place.
• Never deleted. gligabao does not clean Dir. The files stay until you remove them (rm -r build/trained_models/<job id>); the next deploy fetches them again. Model weights are large, so plan for the disk space.
• No recorded files. A model with no recorded files (for instance one saved by a version before files were listed) cannot be deployed: the deployment fails with “the trained model has no recorded files”.

The record of a deployment

Every deployment that gets past the checks of steps 1 and 2 is stored in gb_trained_model_deployments, when it starts and again when it ends, so each can be reviewed later, failures included:

Column	Meaning
id	the deployment’s ID (DeployID in the hook)
trained_model_id	the model (gb_trained_models.id; deleting the model deletes its deployments)
deployer	the name of the deployer asked
options	the request’s options, as a JSON object ({} if none)
status	running (fetching, or the deployer is working), completed or failed
dir	the local directory the files were fetched to; empty until they are
ref, details	the deployer’s result; empty and {} until completed
error	why it failed; empty otherwise
duration_ms	how long the whole deploy took (0 while running)
created_at, completed_at	when it started and when it ended (null while running)

The table exists on SQLite and PostgreSQL and is created automatically. A row that stays running long after it started was interrupted, for example by a restart of the server: nothing resumes it, and nothing marks it failed.

HTTP API

Endpoint
POST /v1/gb/training/models/{id}/deployments	Body {"deployer": "<name>", "options": {...}}. deployer is required (the error lists the registered names); options is a free-form JSON object of at most 64 KB that reaches the hook unchanged. Fetches the files, calls the deployer, waits, and answers 200 with the stored deployment (id, trained_model_id, deployer, options, status completed, dir, ref, details, error, duration_ms, created_at, completed_at).
GET /v1/gb/training/deployers	{"deployers": [{"name": "...", "description": "..."}]}: the registered deployers, sorted by name (empty if none). The admin UI’s Deployers block shows them.
GET /v1/gb/training/models/{id}/deployments[?limit=]	{"deployments": [...]}: the stored deployments of the model, newest first, failed ones too (default 100, at most 500).

$ curl -X POST localhost:7878/v1/gb/training/models/3/deployments \
       -d '{"deployer": "bucket", "options": {"prefix": "release"}}'
{"id":7,"trained_model_id":3,"deployer":"bucket","options":{"prefix":"release"},"status":"completed",
 "dir":"/srv/gligabao/build/trained_models/fc4a...","ref":"models/fc4a.../deploy-7","details":{"files":3},
 "error":"","duration_ms":41873,"created_at":"...","completed_at":"..."}

Status	When
200	the deployment completed
400	invalid JSON body; no deployer registered; deployer missing or unknown; options too large or not a JSON object; a job ID unusable as a directory name
404	unknown model
500	the deployment could not be recorded or read back
502	fetching the files failed, or the deployer returned an error or panicked; the failed deployment is stored (error) and has an ID, listed by the GET endpoint
503	there is no active sidecar

Admin UI

In the admin UI’s Training view, the action box of a trained model (click its row) has a Deploy button, after Audit: it opens a dialog listing the registered deployers with their descriptions, each with its own Deploy button; the one pressed calls POST .../deployments with that deployer, and its outcome (status, ref, duration, or the error) is shown in its row. The Deployers block at the bottom of the view lists the registered deployers; with none registered, Deploy is disabled.

System events

Each deployment that reaches the fetch emits a system event to the SystemEvent hook:

Event type	Fields
training.model_deploy.completed	model_id, deploy_id, deployer, duration_ms (int64)
training.model_deploy.failed	the same, and error

In the code

internal/training/deploy.go (the whole flow), internal/training/db/ (RecordTrainedModelDeployment, FinishTrainedModelDeployment, GetTrainedModelDeployment, ListTrainedModelDeployments; the table for SQLite and PostgreSQL), internal/api/training/models.go (DeployOf, ListDeploymentsOf), internal/sys/sidecar/ (Client.DownloadModelFile), gligabao.go (the public types and ServeOptions).

• training.DeployModel(ctx, db, fetcher, deployers, dir, model, DeployRequest{Deployer, Options}) runs steps 2 to 7 and returns the stored deployment. Its errors are *training.Error with a kind that the handler maps to a status (invalid: 400, upstream: 502, internal: 500).
• training.FetchModelFiles(ctx, fetcher, model, dir) is the fetch alone; fetcher is a training.ModelFileFetcher, one method, DownloadModelFile(ctx, modelPath, file, w), which *sidecar.Client implements. training.ModelDir(root, model) gives the directory of a model.
• training.ResolveDeployers(list) is the startup validation.

Limits and not included

• Deployers are code in the embedding program; none is built in. There is no endpoint to register one at run time.
• A deployment is one synchronous request. There is no background deploy, no progress, no cancellation besides the client leaving, and no resuming.
• gligabao does not merge LoRA adapters into the base model, and does not convert formats: the deployer gets the files as they were saved (for the LoRA fine-tuning: adapter_config.json, adapter_model.safetensors and the tokenizer files, not the base model; adapter_config.json names the base model).
• Deployments are never deleted, and neither are the fetched files.