Modal app lifecycle

List the active Modal apps of this system; deploy, watch, cancel and stop each one, and list their volumes, from the sidecar and the admin UI, without the modal CLI.

sidecar/backends/modal_backend.py, sidecar/backends/actions.py, sidecar/ai_train/deploy_modal.py, sidecar/ai_train/stop_modal.py, sidecar/ai_train/list_volumes.py, sidecar/ai_train/list_apps.py, sidecar/ai_train/inference_containers.py, internal/api/admin/sidecar.go, admin_ui/src/actions.ts

The training code runs on Modal as an app (gligabao-train by default). The sidecar is what talks to Modal, and since gligabao 0.34.0 it also manages the app’s lifecycle: you can see the active Modal apps of this system, deploy and redeploy them, watch the build live, cancel it, stop the app, and list the volumes it uses, all from the training view of the admin UI or over HTTP. None of this needs the modal CLI, a login or a ~/.modal.toml: the sidecar uses the Modal SDK with the tokens in its environment. The CLI commands in Running on Modal still work and do the same thing; the sidecar is the way to do it from the admin panel, or on a machine that has no CLI (the sidecar container).

The Go server stays out of it: it only passes the admin UI’s requests on to the sidecar (see the API).

What you need

• The sidecar running with MODAL_TOKEN_ID and MODAL_TOKEN_SECRET (create them with modal token new; see the sidecar’s configuration). The same variables are what the deploy, stop and volume scripts use: the sidecar hands them to the subprocesses it starts.
• SIDECAR_APP_NAME set the same for the sidecar and the app (the default gligabao-train is fine). The sidecar passes its value to the scripts, so a deploy from the admin UI always deploys the app the sidecar looks up.
• The server running with the admin UI and a reachable sidecar (sidecar_url).
• For fine-tuning, a Modal secret with HF_TOKEN named by SIDECAR_HF_SECRET, created in Modal beforehand (see Running on Modal). A deploy fails, and says so in its log, if the secret the app names does not exist.

Where the state lives

• In Modal: the deployed app, its images, its volumes. This is the state that matters: it survives restarts of the sidecar, of its container, and of the machine, and modal on another computer sees the same app.
• On the sidecar’s disk: nothing. With the tokens in the environment the sidecar never reads or writes ~/.modal.toml (which the CLI creates with modal token set), so a container with no home directory or a read-only filesystem works.
• In the sidecar’s memory: the state and log of the last run of each action (below). A sidecar restart forgets them and ends a deploy that is running; the next status check simply shows the app as deployed or not.

What the admin UI shows

In the training view, the Backends list shows, for the modal backend:

Badge	Meaning
healthy / unhealthy	Modal answers with the sidecar’s tokens. unhealthy has the reason as a tooltip (for example Token not found, a network error, or a timeout after 10 seconds).
app deployed	The app’s web function (serve_api) is found in Modal: training jobs can be started.
app not deployed	Modal answers but has no such app: deploy it.
deployment unknown	The lookup did not give a clear answer.
deploy: <stage>…	A deploy is running (the stage is its current step).
last deploy failed / cancelled	The last run of the action ended that way; View log shows why.

The deploy-related badges (app deployed, deploy: <stage>…, last deploy failed) now appear on the app’s row in the apps list below, not on the backend line.

The apps list

Under the backend line is the list of the active Modal apps of this system (since gligabao 0.35.0), one row per app:

• The app’s name, its state (deployed, ephemeral, initializing, or not deployed; the tooltip has the app id and creation time) and how many tasks it has running.
• Redeploy (Deploy when the app is not deployed), Last log (View log while an action runs; the log of the last action run on that app), Stop inference (red; only while the app is deployed), Stop (red; only while the app is deployed) and Volumes.
• A badge N inference container(s) warm while GPU inference containers are up (the tooltip lists each one: function, model, requests, last use; see Warm inference containers).
• Badges for an action that is running (deploy: building images…) or whose last run failed / was cancelled.

“Of this system” means the app named by SIDECAR_APP_NAME and apps named SIDECAR_APP_NAME-<suffix> (for example a -dev copy). Other apps of the Modal workspace are neither shown nor touchable. The configured app is always listed, as not deployed when it is not up, so that it can be deployed; an app that is stopped (history in Modal) is not shown, unless an action ran on it since the sidecar started.

The list comes from Modal (sidecar/ai_train/list_apps.py, the equivalent of modal app list, read only) and takes a few seconds; the previous list stays on screen while it loads. It refreshes every 5 seconds while an action runs, and when you press Refresh. Every button below acts on its own app: each app has its own deploy and stop actions and its own log, and deploy and stop exclude each other per app (two apps can deploy at the same time).

Deploy

Deploy asks for a confirmation, starts the run and opens the live log. What happens:

1. The sidecar starts sidecar/ai_train/deploy_modal.py in a subprocess, with sidecar/ai_train/ as its working directory.
2. It imports app.py (the app definition) and calls the Modal SDK’s app.deploy(), with Modal’s build output switched on. This is what modal deploy app.py does.
3. Modal uploads the ai_train/ directory (mounted into the containers at /pkg), builds the images that are not cached, creates the app’s functions and the web endpoint, and creates any of the volumes that do not exist yet.
4. The script exits with 0 on success; the run is then succeeded and the badge turns to app deployed. Any error ends the script with a traceback and a final Deploy failed: <error> line in the log, and the run is failed.

The first deploy builds the images (the Unsloth training image for both model sizes, and the lighter inference image without Unsloth), which takes several minutes; later deploys reuse Modal’s image cache, and because the code is mounted last, a code change does not rebuild the dependency layers. Redeploying updates the app in place under the same name and web URL.

Stages

The log dialog shows four stages as chips: starting (the process runs), building images, creating functions, deployed. The sidecar reads them from Modal’s output (the lines about building images and creating objects) and they only move forward. They are a guide, not a protocol: if Modal changes its wording a stage may be skipped, so the log below is the source of truth.

The live log

The dialog (admin_ui/src/actions.ts) has the state badge, the stage chips, the elapsed time, and the output of the command, one line each with its time since the start (+12.3s). Lines with error, failed, traceback or exception are red. Follow keeps the view at the end; scrolling up turns it off, ticking it scrolls down again. The dialog keeps the last 5000 lines.

Closing the dialog does not stop the run: it only stops watching. View log reopens it, also after a page reload, because the log lives in the sidecar. Behind it is a Server-Sent Events stream, so it is live and needs no polling:

• log events carry lines, a list of {i, t, line} (i the line’s index, t its time in Unix seconds), with the SSE id of the last one.
• state events carry the state of the run (see the API).
• The stream ends after the run is over. A client that loses the connection and reconnects gets the log again from the start (or ?from=<i>) and skips the indices it already has.
• The output of the command is stdout and stderr merged; \r (progress bars) and \n both end a line, and colour codes are removed.

Cancel

While a run is going the dialog has a Cancel run button. The sidecar sends the command SIGINT (so the SDK can clean up; the traceback of the interruption shows in the log) and kills it if it is still there 10 seconds later. The run ends as cancelled and can be started again. Cancelling before the deploy has finished normally leaves the previous deployment in place. What Modal had already built or cached on its side is not rolled back.

Stop

Stop asks for a confirmation that names the consequences, then runs sidecar/ai_train/stop_modal.py, which does what modal app stop does: it looks the app up by name and tells Modal to stop it.

Stopping the app terminates its running containers, including training jobs and inference that are running, and takes the web API offline: training and inference calls through the sidecar fail until you deploy again. The volumes (datasets, adapters, the Hugging Face cache) and the app’s history are kept, and a stopped job’s row stays in the database, where it is marked failed after the usual timeout (see Not tracked yet).

The Stop button is not offered while the app is not deployed, and it never runs at the same time as a deploy (409; cancel the deploy first). The SDK has no public App.stop yet, so the script makes the same two internal calls as the CLI (look up the app id by name, AppStop); a future modal 1.x release could change them, which would show as a failed stop with the error in the log. Starting the app again is a deploy. To free only the GPU containers that hold inference models, use Stop inference instead (see Warm inference containers).

Warm inference containers

An inference request (POST /infer/query) runs in a GPU container of the infer_4b / infer_12b / infer_31b function. Until gligabao 0.39.0 every request loaded the base model and the adapter again, which was most of the latency of a request. Since 0.39.0 the container keeps the loaded model in memory and stays up for SIDECAR_INFER_IDLE_TIMEOUT_S (default 60 seconds) after its last request, so the requests that follow skip the load. When no request arrives within that time Modal shuts the container down by itself; the next request starts a container and loads the model again. The answer of /infer/query says which it was: cold_start and load_seconds.

• One model per container. A request for another trained model (or size) replaces the loaded one. A model retrained in place (the legacy finetuned_model_<size>) is noticed by its file time and loaded again.
• Concurrency. A container answers one request at a time, as before; Modal starts more containers when requests queue up. Each loads the model and each is billed while it is up.
• Redeploy after changing it. The timeout is part of the function’s definition: Redeploy the app after changing SIDECAR_INFER_IDLE_TIMEOUT_S, or after upgrading to 0.39.0. Modal accepts 2 to 1200 seconds.
• Stop inference. The button on the app’s row (action stop_inference, script sidecar/ai_train/inference_containers.py) ends all the warm inference containers at once, without waiting for the timeout. A request being answered by one of them fails; the app, its web API and the training jobs are not touched, and the next request starts a container again. It asks for a confirmation, is only offered while the app is deployed, and can run at the same time as a deploy or a stop.
• How the containers are found. Modal does not say which function a container runs, so each inference container records itself (task id, function, model, request count, last use) in a Modal Dict named <app name>-inference. The listing and the stop script only trust a record while Modal still lists that task, and remove the records of containers that are gone.

Cold and warm starts

A request is cold when no container of the function holds the requested model: the first request after a deploy, the first one after the container was idle for SIDECAR_INFER_IDLE_TIMEOUT_S, one for another trained model or size than the one loaded, and one that Modal sends to an extra container because the others are busy. It is warm when a container that is already up holds that model: it only generates. This is the GPU inference container (infer_4b / infer_12b / infer_31b). The training container is another function: it loads the model with Unsloth for each job and is not kept warm.

What happens in the GPU container on a cold request, in order (app.py, inference.py, model_load_infer.py):

1. Containers start. The request reaches the web API and the CPU router infer (each of them can be cold too), which call the GPU function. Modal starts a container on the GPU of that size (A10G for 4b, A100 for 12b, A100 80 GB for 31b) from the inference image, mounts the models and Hugging Face cache volumes and imports the app. This is Modal’s time: it is not in load_seconds.
2. The models volume is reloaded, so an adapter trained a moment ago is visible (a failure here is only logged).
3. The cache is checked. The container holds at most one model, identified by its adapter directory and the file time of adapter_config.json. It is empty (or holds another model), so the loaded one is dropped and its GPU memory freed. From here until the model is ready is what load_seconds measures.
4. The adapter is looked up. resolve_model_dir turns trained/<job_id> into a directory of the models volume; without an adapter_config.json the request fails with “No trained model at …” (there is no silent fallback to the un-tuned base model).
5. The libraries are imported (torch, peft, transformers), once per container.
6. The base model is loaded 4-bit. from_pretrained reads the base model (google/gemma-4-E4B-it, google/gemma-4-12B-it or google/gemma-4-31B-it) from the HF cache volume and quantizes it to nf4 (bfloat16 compute, double quantization) while placing it on GPU 0, the quantization the adapter was trained with. If the cache volume does not have the model yet (the first use ever, or a size that was never trained or served), it is downloaded from Hugging Face first (needs HF_TOKEN, see SIDECAR_HF_SECRET), which makes that one request much slower; the volume keeps it for every later container. This step is most of load_seconds.
7. The adapter is attached with peft (not trainable, eval() mode) and the tokenizer is read from the adapter directory (the base model’s if it is not there).
8. The model is cached in the container’s memory, for the requests that follow.
9. The prompt is generated: tokenized, sampled up to max_tokens, decoded. When the answer is ready the container records itself in the <app name>-inference Dict, so the admin UI can list it (see above).

A warm request does steps 2 and 9 only, so its time is the generation. For a short reply that is seconds. A reply that does not end early (the model stops at its end-of-sequence token or at max_tokens) can take much longer, because 4-bit generation with transformers is not fast: read output_tokens next to the time.

Every request reports this, and the admin UI shows it:

• The answer of POST /infer/query has cold_start (true when the request loaded the model), load_seconds (steps 3 to 8; 0 when warm), input_tokens and output_tokens. POST /v1/gb/training/models/{id}/infer passes them on.
• The Go server stores cold_start and load_seconds with the other facts of the request (gb_trained_model_inferences; NULL for a failed request, a request made before gligabao 0.46.0, or a sidecar that does not report them).
• In the trained-model dialog of the admin UI, each request of the history has a line such as cold start (model load 27.5s) · 3.2s without the model load · 12 tokens in / 34 out, or warm (model already loaded) · …. The time without the model load is the total minus load_seconds, so it includes the container start of step 1 and the generation. The status line under the form shows the same after a request, and a dot above the prompt field shows whether a warm container holds this model.

How to read it: a long load_seconds is the model load (steps 3 to 8); a cold request with a short load_seconds but a long total spent its time starting containers; a slow warm request has a large output_tokens (a long reply that did not end) or waited behind another request of the same container.

Ways to get fewer cold requests (gligabao does none of them): a longer SIDECAR_INFER_IDLE_TIMEOUT_S (up to 1200 seconds) at the price of GPU time while idle; a request every few minutes to keep a container up; or, in app.py, a minimum number of warm containers for infer_4b, which Modal bills all the time.

Volumes

Volumes (on an app’s row) opens a table of the Modal volumes the app uses, read from the app’s configuration (config.py, so the SIDECAR_VOL_* settings are respected):

Column	Meaning
Mounted at	Where the container sees it: /data, /models, /eval, /root/.cache/huggingface.
Volume	Its name in Modal (gligabao-train-data, …).
In Modal	found, or not created yet (the first deploy creates it), or error with the reason as a tooltip.
Entries	How many files and directories its top level holds.
Created (UTC)	When Modal created it.

The /models volume holds the trained models, one directory per completed job (trained/<job_id>, see Trained models); its Entries count is the number of top-level entries (trained, and finetuned_model_<size> if an older version saved one), not the number of models. /root/.cache/huggingface holds the downloaded base models.

It is read only: nothing is created, changed or deleted, and it works whether or not the app is deployed. It takes a few seconds because it runs sidecar/ai_train/list_volumes.py in a subprocess (60 second limit). A volume that cannot be queried gets an error of its own and the others are still listed. Deleting or emptying volumes is not offered; use modal volume for that.

The API

All routes are answered by the sidecar itself (they are not forwarded to the Modal app). {name} is the backend (modal), {app} an app of this system and {action} is deploy, stop or stop_inference. The Go server passes them on under /v1/gb/admin/api/sidecar/backends/… (the same paths after /backends), flushing the stream as it arrives; it adds nothing and keeps no state. They are for the admin UI, are only registered when the admin UI is on, and have no authentication of their own.

Sidecar route	Meaning
GET /backends	The backends with their live status: name, configured, healthy, and for modal also deployed (true, false, or null if unknown), app, error (when unhealthy) and actions (each action’s state; stop and stop_inference are left out while the app is not deployed). The server’s GET /v1/gb/admin/api/sidecar/backends answers {"backends": [...]}, or an empty list with an error text when the sidecar is down.
POST /backends/{name}/actions/{action}	Start the action: 202 with its state; 409 if it (or the action it excludes) is running; 404 for an unknown backend or action.
POST /backends/{name}/actions/{action}/cancel	Cancel it: 202, or 409 if it is not running.
GET /backends/{name}/actions/{action}[?from=<i>]	The state and the kept log lines with index >= from.
GET /backends/{name}/actions/{action}/events[?from=<i>]	The SSE stream above.
GET /backends/{name}/apps	{"apps": [{"name", "app_id", "state", "n_running_tasks", "created_at", "inference", "active", "configured", "actions"}, ...]}: the active apps of this system (inference is the list of its warm inference containers, each {task_id, function, model_size, adapter, loaded_at, last_used, requests}; actions is the state of each action; stop and stop_inference are left out for an app that is not active). 404 if the backend has none, 502 if they cannot be listed.
…/backends/{name}/apps/{app}/actions/{action}[/events|/cancel]	The same routes as the actions above, for one app (404 for a name that is not the configured app or <it>-<suffix>). The routes without apps/{app} act on the configured app.
GET /backends/{name}/volumes	{"volumes": [{"mount_path", "name", "exists", "created_at"?, "entries"?, "error"?}, ...]}; 404 if the backend has none, 502 if they cannot be listed.

The state of an action:

{
  "action": "deploy", "label": "Deploy the Modal app", "confirm": "", "danger": false,
  "run_id": 3, "state": "running",
  "stage": "building images", "stages": ["starting", "building images", "creating functions", "deployed"],
  "started_at": 1792188549.8, "finished_at": null, "exit_code": null, "error": "", "lines": 412
}

state is idle (never run since the sidecar started), running, succeeded, failed (exit_code and error say how) or cancelled. run_id grows with every run, which is how a client tells a new run from the old one. A run’s log is kept until the next run of the same action starts. Only one run of an action goes at a time, and deploy and stop exclude each other.

curl -X POST   http://127.0.0.1:8787/backends/modal/actions/deploy           # start
curl -N        http://127.0.0.1:8787/backends/modal/actions/deploy/events    # follow it
curl           http://127.0.0.1:8787/backends/modal/actions/deploy           # state and log
curl -X POST   http://127.0.0.1:8787/backends/modal/actions/deploy/cancel    # cancel
curl           http://127.0.0.1:8787/backends/modal/volumes
curl           http://127.0.0.1:8787/backends/modal/apps                                   # the active apps
curl -X POST   http://127.0.0.1:8787/backends/modal/apps/gligabao-train-dev/actions/deploy  # deploy one of them

In the container

Dockerfile.sidecar copies sidecar/ai_train/ to /app/ai_train, next to the sidecar, because that is the code a deploy uploads (and where the four scripts live); it is still not imported by the sidecar process. The scripts run as the image’s non-root sidecar user, which has no home directory and needs none. Only modal from requirements.txt is needed to deploy: the heavy ML packages are installed in Modal’s images, not in the sidecar’s.

Troubleshooting

What you see	Cause and fix
app not deployed	Nothing is deployed under SIDECAR_APP_NAME: press Deploy. A training job started now fails with 502 saying the web URL of the app cannot be found.
unhealthy, tooltip Token not found / Token ID is malformed	The sidecar’s MODAL_TOKEN_ID / MODAL_TOKEN_SECRET are wrong or revoked: create new ones with modal token new and restart the sidecar.
Deploy ends failed with AuthError in the log	The same: the tokens are not accepted.
Deploy ends failed about a missing secret	SIDECAR_HF_SECRET names a Modal secret that does not exist: create it (modal secret create) or unset the variable.
Deploy stays at starting for long	Modal is slow to answer or the upload is large; read the log. If it is really stuck, Cancel run and try again.
Stop or Stop inference is missing	The app is not deployed (nothing to stop).
Stop inference says no container is running	The containers already shut down (the idle timeout passed) or none was started; nothing to do.
Every inference request is slow	Each one finds no warm container: requests are more than SIDECAR_INFER_IDLE_TIMEOUT_S apart, the app was not redeployed since 0.39.0, or the requests alternate between models (one model per container). Check cold_start in the answers. The request history of the model in the admin UI shows it for each request (cold start, warm), with the model load time and the token counts: see Cold and warm starts.
A warm inference request is still slow	Read output_tokens: a reply that does not end until max_tokens takes a long time with 4-bit generation. If replies regularly run to the limit, the adapter was probably trained before gligabao 0.46.1 (wrong end-of-turn marker, so the model never learned to stop): retrain it. Otherwise it waited behind another request of the same container.
An app I deployed is not in the list	Its name is not SIDECAR_APP_NAME or SIDECAR_APP_NAME-<suffix>, or it is stopped. Only active apps of this system are listed.
Could not list the modal apps	Modal did not answer within 30 seconds, or rejected the tokens; the message has the reason. The rest of the view still works.
409 on Deploy or Stop	A run of it, or of the action it excludes, is going: open View log, or cancel it.
Volumes shows not created yet	Normal before the first deploy.
The log is gone after a restart	By design: the log is in the sidecar’s memory. The app itself is still in Modal.

Not included

• No authentication: the sidecar and the admin panel trust their network, so keep the sidecar on localhost or the private network and the admin UI off the public internet. Anyone who can reach these routes can deploy or stop the apps.
• No scheduling or automatic redeploy: nothing deploys unless someone presses the button or calls the route.
• No volume management (create, empty, delete) and no log history across restarts.
• Only the modal backend has actions. Another backend gets them by overriding Backend.actions() (and apps(), app_actions(), volumes()); the admin UI draws whatever buttons the backend reports, using each action’s label, confirm and danger. See Python sidecar.