AIR CLI Integration: Batch Merge Phase 3 - #6267
Open
riddhibhagwat-db wants to merge 30 commits into
Open
Conversation
- Remove a hardcoded email from render_test.go (use user@example.com). - list_tui.go: use the existing cmdio.IsPromptSupported instead of the air-added IsPagerSupported; drop IsPagerSupported from libs/cmdio/io.go since it is no longer used. - format.go: standardize on termenv.Hyperlink and drop the hand-rolled osc8Link helper (and its test). - Centralize EnvMatrix.DATABRICKS_BUNDLE_ENGINE = [] into a shared acceptance/experimental/air/test.toml; remove the per-dir duplication (deleting the test.toml files left with nothing else). Co-authored-by: Isaac
## Changes Ports `--override KEY=VALUE` from the Python CLI. Overrides apply to the parsed YAML map before re-decode + validate, so path existence, type coercion, and semantic validate() rules all run. A reflection-based path check names the exact --override key and lists available fields on error. ## Why --override allows users to tweak a training configuration at launch time without editing the file and this allows for ease of use with sweep hyperparameters or scalable compute without having to maintain a forked configuration for each run. ## Tests unit tests: - TestParseOverrides: parsing KEY=VALUE - TestValidateOverridePaths: dotted-path validation against the schema - TestLoadRunConfigWithOverrides: end-to-end through the loader: scalar coercion, multiple overrides, free-form env-var-as-string, auto-created intermediate maps, unknown-path rejection, semantic re-validation after override, type mismatch, malformed override. - TestSubmitWorkload: the harness pattern being extended to prove overrides reach the actual POST body sent to /api/2.2/jobs/runs/submit, verified against the in-process testserver that records the request. - TestSubmitWorkloadHonorsOverride: proves a --override actually changes what gets sent to the Jobs API on a real submit, not just during dry-run validation. acceptance tests: - successful override that logs the change and then validates - an unknown field override that errors with an actionable message (see screenshots below) - override that passes type checking but fails schema validation (so we know that validate() still runs and works properly) Manual verification: <img width="1894" height="888" alt="Screenshot 2026-07-14 at 10 41 22 AM" src="https://github.com/user-attachments/assets/69187c62-ad60-4144-9744-79e52787447c" />
…fig (#5968) Post-merge cleanup for the experimental AIR CLI (follow-up to #5847, which squash-merged `air-cli` into `main`). This commit was made after that merge, so it is not yet in `main`. - Remove a hardcoded email from `render_test.go` (use `user@example.com`). - `list_tui.go`: use the existing `cmdio.IsPromptSupported` instead of the air-added `IsPagerSupported`; drop `IsPagerSupported` from `libs/cmdio/io.go` since it is no longer used. - `format.go`: standardize on `termenv.Hyperlink` and drop the hand-rolled `osc8Link` helper (and its test). - Centralize `EnvMatrix.DATABRICKS_BUNDLE_ENGINE = []` into a shared `acceptance/experimental/air/test.toml`; remove the per-dir duplication. Stacked below the `air logs` port branch (`air-logs-m4`), which depends on the centralized `test.toml` introduced here. This pull request and its description were written by Isaac. --------- Signed-off-by: Lennart Kats <lennart.kats@databricks.com> Co-authored-by: radakam <55745584+radakam@users.noreply.github.com> Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com> Co-authored-by: Jan N Rose <janniklas.rose@gmail.com> Co-authored-by: Grigory Panov <grigory.panov@databricks.com> Co-authored-by: Andrew Nester <andrew.nester.dev@gmail.com> Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
## Changes Implements the air logs JOB_RUN_ID command (previously a notImplemented stub) for the experimental AIR CLI. It fetches a run's training logs with a Bricklens-first, MLflow-fallback strategy: - Bricklens (primary): streams logs from the AiTraining log endpoint following an active run to completion, or tailing a completed one. - MLflow (fallback): when Bricklens is unavailable & gated off by a backend flag (FEATURE_DISABLED), not deployed (ENDPOINT_NOT_FOUND/404), or persistently failing. The command falls back to reading the run's MLflow log artifacts (chunk discovery + credential-vended download). The flag is evaluated server-side; the CLI only reads the response error code. Flags: - `--minutes` restricts the fetch to the last N minutes (Bricklens time window). - `--lines <N>` is the tail the last N lines of a completed run. Mutually exclusive with --minutes. - `--node`, `--retry` is used to select a node / retry attempt. A past retry of a still-active run renders its (immutable) logs once instead of following the run. - `--download-to` / `--review` are rejected with a clear "not implemented" error (next PR follow up). ## Why `air logs` was the last unimplemented read command in the AIR CLI port. Bricklens is the primary log source, but it's behind a backend feature flag and isn't universally deployed, so the command must degrade gracefully to MLflow artifacts rather than fail. Bricklens is time-indexed (hence --minutes), while MLflow stores fixed log chunks (hence line-based --lines). the two flags map to what each backend can actually do. ## Tests - Unit: classifyLogError fallback classification, --minutes/--lines window math, run-status projection, bounded dedup set, page draining + tail ordering, MLflow chunk listing/sorting and log-path discovery, flag validation, and an end-to-end Bricklens MLflow fallback through a mock server. - Acceptance: acceptance/experimental/air/logs/ (text + JSON streaming, --minutes, --lines, --retry, mutual-exclusion error, invalid ID, --download-to rejection) and logs-mlflow-fallback/ (Bricklens FEATURE_DISABLED, MLflow fallback, no-logs, text + JSON). <img width="1628" height="860" alt="Screenshot 2026-07-24 at 4 14 21 PM" src="https://github.com/user-attachments/assets/1f70943a-8a5d-47f2-8797-ec6f3e91a8fe" />
…6080) ## Changes & Why After submitting a workload, --watch follows the run's logs to completion and exits with the run's outcome, reusing the same Bricklens-with-MLflow-fallback pipeline as `air logs`. - Text mode: prints "Submitted run", the dashboard link, and "Monitoring run and streaming logs...", then streams the logs. - JSON mode: emits a SUBMITTED event with the run id, a STATUS event on each lifecycle transition, the streamed LOG/ALERT events, and a closing terminal-status envelope (SUCCESS/FAILED/CANCELED) — matching the Python CLI's --watch JSONL contract. - Without --watch, the plain submit path now prints a tip about --watch. - STATUS events are watch-scoped (opt-in via logRequest.onStatusChange), so the merged `air logs` output is unchanged. --dry-run still takes precedence over --watch (nothing is submitted or streamed). ## Tests Unit tests (experimental/air/cmd/) - logbricklens_test.go: Bricklens client query/path serialization + time_unix_nano parsing - logstream_test.go: fallback classification, status projection, --minutes/tail math, page dedup/ordering, retry-then-fallback, JSONL/ALERT emit, Ctrl-C exit - logmlflow_test.go: MLflow chunk discovery/listing, attempt-prefix layout, no-logs exit-code parity - logs_test.go: air logs command: flag validation, completed-run tail, Bricklens→MLflow fallback, past-retry static view - run_watch_test.go: air run --watch: text stream, JSON SUBMITTED→STATUS→LOG→terminal envelope, failed-run exit code, dry-run precedence Acceptance tests (acceptance/experimental/air/) - logs/ : text/JSON streaming, --minutes, --lines, --lines 0, --retry, mutual-exclusion errors, invalid id, negative node, --download-to - logs-mlflow-fallback/ : Bricklens FEATURE_DISABLED → MLflow fallback → no-logs (text & JSON) - run/ : dry-run, --override, config validation, --watch ignored under --dry-run - run-submit/ : real submit payload + --watch tip line - help/ : air --help, air logs --help command-tree pins Manual verification: Properly monitors and outputs logs from runs on manual test: <img width="1166" height="112" alt="Screenshot 2026-07-27 at 3 00 15 PM" src="https://github.com/user-attachments/assets/f4522965-f200-410c-8ddc-307092c6b731" />
Brings the air-cli feature branch current with main. air-cli had drifted 110 commits behind, spanning Grigory Panov's localenv/environments redesign (cmd/localenv renamed to cmd/environments, new JobTaskEnvironment API, --cluster-name/--job-task flags, uv provisioning) and other infra work. Conflict resolution rule: - Infra (libs/localenv, cmd/environments, libs/filer, bundle/config/validate) and their acceptance goldens: take main. air-cli only carried an older snapshot of this shared code (via the #5968 squash); it had no AIR-specific edits there, so main is authoritative. - AIR (experimental/air/**, acceptance/experimental/air/**): keep air-cli. This is Riddhi's AIR CLI work and the reason the branch exists. The per-dir air test.toml deletions are honored (engine matrix centralized in the shared acceptance/experimental/air/test.toml). - libs/localenv/target_test.go aligned to main to match main's target.go API. Verified: go build ./... clean; 123 packages pass across libs/, cmd/, bundle/config/ (0 failures); localenv acceptance green. Co-authored-by: Isaac
The main catch-up merge brought in a new acceptance validator that rejects EnvMatrix.DATABRICKS_BUNDLE_ENGINE = [] (it would run on both direct and terraform CI runners). The shared acceptance/experimental/air/test.toml still had [], which broke every air acceptance test on air-cli post-merge. Pin to ["direct"] as the validator directs: no air command deploys a bundle, so running once on a single engine is the intent (an empty list is now disallowed). Co-authored-by: Isaac
…ommand The main catch-up merge renamed the local-env command to `environments setup-local` (--job-task <id>.<key> model). Three air-cli-only tests (job-ambiguous-compute, job-multicluster-mismatch, job-serverless-version-mismatch) still invoked the removed `local-env python sync --job ... --check` surface, so they broke post-merge with `unknown command "local-env"`. Delete them: they targeted the pre-redesign CLI, and main's newer localenv suite already covers these cases (cluster-name-ambiguous, job-task-missing-key, job-task-jobcluster, serverless-check, etc.) via the new command surface. Co-authored-by: Isaac
Follow-up to the earlier shared test.toml fix ([] -> ["direct"]): the derived out.test.toml for the logs and logs-mlflow-fallback dirs still recorded the old [] value, so CI's "changed files" check (git diff --exit-code after regenerating out.test.toml) failed. Regenerate them to match. Co-authored-by: Isaac
Package the code_source working tree into a tarball and upload it
through DABs' artifact-upload plumbing (libraries.ReplaceWithRemotePath
+ libraries.Upload over a minimal in-memory bundle), rewriting
ai_runtime_task.code_source_path to the uploaded remote path. The
packaging + upload orchestration is CLI-owned (experimental/air/cmd,
OWNERS = us); it only reuses DABs' uploader so we don't reimplement
workspace/volume upload.
snapshot_dabs.go: build the plain-tar tarball (createPlainTarball),
carry it as a file-valued code_source_path on a minimal bundle, and
drive the DABs upload. runsubmit.go swaps the old raw-filer snapshot
upload for this. Removes the retired raw-filer upload path (snapshot.go
uploader, snapshot_test.go).
Tar snapshotting only; git pinning follows in the next PR (its git
helpers are removed here and reintroduced there).
Co-authored-by: Isaac
## Changes
<!-- Brief summary of your changes that is easy to understand -->
## Why
<!-- Why are these changes needed? Provide the context that the reviewer
might be missing.
For example, were there any decisions behind the change that are not
reflected in the code itself? -->
## Testing
### Unit + acceptance
`experimental/air/cmd/...` and `acceptance/experimental/air/run-submit`
— working-tree,
git-pinned, and remote-Volume submits each assert the tarball lands
under `.internal/`
and the rewritten `code_source_path` rides the submitted
`ai_runtime_task`; plus the tar
builders (`.gitignore`, `.git` exclusion, `include_paths`) and the
no-`code_source`
nil-guard. All green.
### Live E2E — staging `dbc-04ac0685-8857` (GPU_1xA10)
5/5 runs SUCCESS, one per packaging mode. All runs are `CAN_VIEW` for
the workspace
`users` group, so every link below is openable by anyone in the
workspace.
**Setup** — a tiny project with a gitignored file (`debug.log`) to prove
exclusion:
```bash
mkdir -p /tmp/air-demo/proj/src/pkg && cd /tmp/air-demo/proj
printf 'import os\nprint("train ran; cwd:", os.getcwd(), "CODE_SOURCE_PATH:", os.environ.get("CODE_SOURCE_PATH"))\n' > src/train.py
echo 'def helper(): return 1' > src/pkg/util.py
echo '*.log' > src/.gitignore
echo 'noise' > src/debug.log # gitignored — must never be uploaded
cat > wt.yaml <<'YAML'
experiment_name: vchen_demo_wt
command: cd $CODE_SOURCE_PATH && python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
environment: {version: "4", dependencies: []}
code_source: {type: snapshot, snapshot: {root_path: src}}
YAML
```
**1. Working-tree tarball** — plain tar of the working tree, honoring
`.gitignore`.
Run:
[994765091508414](https://dbc-04ac0685-8857.staging.cloud.databricks.com/jobs/runs/994765091508414)
```bash
dbcli experimental air run -f wt.yaml -p dbc-04ac0685-8857
# Uploading src.tar.gz... → Submitted run 994765091508414
dbcli workspace export /Workspace/Users/v.chen@databricks.com/.air/repo_snapshots/.internal/src.tar.gz --file /tmp/wt.tar.gz -p dbc-04ac0685-8857
tar tzf /tmp/wt.tar.gz
# → src/train.py, src/pkg/util.py, src/.gitignore (NO debug.log ✓ gitignore honored)
```
**2. Git-pinned commit** — `git archive` of a pinned SHA. An uncommitted
file is created
*after* the commit to prove the archive captures the commit, not the
dirty working tree.
Run:
[304463075281818](https://dbc-04ac0685-8857.staging.cloud.databricks.com/jobs/runs/304463075281818)
```bash
git init -q && git add -A && git commit -qm init
SHA=$(git rev-parse HEAD)
echo "print('uncommitted')" > src/uncommitted.py # created AFTER the commit
cat > git.yaml <<YAML
experiment_name: vchen_demo_git
command: cd \$CODE_SOURCE_PATH && python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
environment: {version: "4", dependencies: []}
code_source: {type: snapshot, snapshot: {root_path: src, git: {commit: $SHA}}}
YAML
dbcli experimental air run -f git.yaml -p dbc-04ac0685-8857
# Uploading src.tar.gz... → Submitted run 304463075281818
dbcli workspace export /Workspace/Users/v.chen@databricks.com/.air/repo_snapshots/.internal/src.tar.gz --file /tmp/git.tar.gz -p dbc-04ac0685-8857
tar tzf /tmp/git.tar.gz
# → src/train.py, src/pkg/util.py, src/.gitignore
# (NO uncommitted.py, NO debug.log ✓ archived the commit, not the working tree)
```
**3. UC Volume destination** — `remote_volume` routes the upload to a UC
Volume via the
Files API (`/api/2.0/fs/files/...`), natively, with no special-casing in
the CLI.
Run:
[438683652713410](https://dbc-04ac0685-8857.staging.cloud.databricks.com/jobs/runs/438683652713410)
```bash
dbcli volumes create main default vchen_demo MANAGED -p dbc-04ac0685-8857
cat > vol.yaml <<'YAML'
experiment_name: vchen_demo_vol
command: cd $CODE_SOURCE_PATH && python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
environment: {version: "4", dependencies: []}
code_source: {type: snapshot, snapshot: {root_path: src, remote_volume: /Volumes/main/default/vchen_demo}}
YAML
dbcli experimental air run -f vol.yaml -p dbc-04ac0685-8857 --debug 2>&1 | grep -iE "submitted|code_source_path|/api/2.0/fs/files"
# → "code_source_path": "/Volumes/main/default/vchen_demo/.internal/src.tar.gz"
# → Submitted run 438683652713410
dbcli fs ls dbfs:/Volumes/main/default/vchen_demo/.internal -p dbc-04ac0685-8857
# → src.tar.gz (uploaded to the Volume ✓)
```
**4. `include_paths` subset** — only the listed paths are packaged.
Run:
[261250771126835](https://dbc-04ac0685-8857.staging.cloud.databricks.com/jobs/runs/261250771126835)
```bash
cat > inc.yaml <<'YAML'
experiment_name: vchen_demo_inc
command: cd $CODE_SOURCE_PATH && python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
environment: {version: "4", dependencies: []}
code_source: {type: snapshot, snapshot: {root_path: src, include_paths: [pkg]}}
YAML
dbcli experimental air run -f inc.yaml -p dbc-04ac0685-8857
# Uploading src.tar.gz... → Submitted run 261250771126835
dbcli workspace export /Workspace/Users/v.chen@databricks.com/.air/repo_snapshots/.internal/src.tar.gz --file /tmp/inc.tar.gz -p dbc-04ac0685-8857
tar tzf /tmp/inc.tar.gz
# → src/pkg/util.py ONLY (train.py / .gitignore excluded ✓)
```
**5. No `code_source`** — nothing is uploaded; `code_source_path` is
left empty (nil-guard).
Run:
[138286541552104](https://dbc-04ac0685-8857.staging.cloud.databricks.com/jobs/runs/138286541552104)
```bash
cat > none.yaml <<'YAML'
experiment_name: vchen_demo_none
command: echo hello
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
environment: {version: "4", dependencies: []}
YAML
dbcli experimental air run -f none.yaml -p dbc-04ac0685-8857
# → Submitted run 138286541552104 (no "Uploading" line; code_source_path empty)
```
The main->air-cli catch-up merge bumped the SDK to v0.165 and pulled in updated generated pydabs models, but the checked-in python/databricks/bundles/** was formatted by an older pinned ruff. The current ruff pin reformats them (e.g. "Self" -> 'Self', import wrapping), so `task generate-check` / the validate-generated CI job drifts on every PR into air-cli (#6102, #6090). Regenerated with `task pydabs-codegen` so the checked-in models match the generators + pinned ruff. Generated-only change (python/databricks/bundles/**). Co-authored-by: Isaac
This reverts commit b632473.
## Changes <!-- Brief summary of your changes that is easy to understand --> ## Why <!-- Why are these changes needed? Provide the context that the reviewer might be missing. For example, were there any decisions behind the change that are not reflected in the code itself? --> ## Tests <!-- How have you tested the changes? --> <!-- If your PR needs to be included in the release notes for next release, add a changelog fragment: create .nextchanges/<section>/<name>.md with a one-line description (e.g. .nextchanges/cli/quickstart.md). See .nextchanges/README.md. -->
Reverts #6121 ("Air cli drop requirements yaml"). #6121 moved `air run` dependencies onto `environments[].spec.dependencies` and dropped the co-located `requirements.yaml` upload. Reverting so the equivalent change can land via #6077, which additionally: - resolves the **version declared inside a file-form `requirements.yaml`**. In #6121 `requirementsDoc.Version` is decoded but never used, so `dependencies: ./reqs.yaml` with `version: 5` inside silently falls back to the default runtime image (`cfg.runtimeVersion()` returns `ok=false` for the file form). - rejects `-r`/`--requirement` includes in a requirements file, which reference a second file that is never uploaded with the run and so cannot resolve on the node. - adds acceptance coverage (`acceptance/experimental/air/run-submit-deps`) asserting the deps on the wire, the file-form version, and that no requirements file is uploaded. ## Tests Verified on this branch after the revert: `go build ./experimental/air/...`, the `experimental/air/cmd` unit suite, and the air acceptance suite (`TestAccept/experimental/air`) all pass. This pull request and its description were written by Isaac.
…6077) ## Changes & Why `air run` now carries the user's declared dependencies (which may be an inline list, or read from a requirements.yaml file) on the submission's environments[].spec.dependencies, and no longer uploads a requirements.yaml artifact at all. This is the follow up PR to https://github.com/databricks-eng/universe/pull/2178617?timeline_per_page=5 (implementing this method in the python CLI) and https://github.com/databricks-eng/universe/pull/2297011?timeline_per_page=5 (follow up backend changes to unblock the new path; installs the inline deps via --deps-config and treats a missing co-located requirements.yaml as "no requirements"). This removes the vestigial empty requirements.yaml that a no-dependency run used to upload just to satisfy the launcher's derived path. When no dependencies are declared, spec.dependencies is omitted and the payload is unchanged. A -r/--requirement include in a requirements file is rejected, since the referenced file is never uploaded with the run. ## Tests Unit tests: - Upload side: TestBuildArtifacts_CommandAndConfig, TestBuildArtifacts_ParametersButNoRequirements, TestBuildArtifacts_RequirementsFileNotUploaded, TestBuildArtifacts_EnvVarsAndSecrets, TestBuildArtifacts_OversizeConfigRejected - Submit side: TestBuildSubmitPayloadInlineDependencies, TestEnvironmentDependencies, TestReadRequirementsDependencies, TestEnvironmentDependencies_MissingRequirementsFile - End-to-end (unit): TestSubmitWorkload / TestSubmitWorkloadWithCodeSource �� Acceptance tests: `acceptance/experimental/air/run-submit-deps/` -> submits with inline deps, golden asserts: - `spec.dependencies: [numpy, torch==2.3.0]` on the runs/submit wire �� - Only `command.sh` + `training_config.yaml` uploaded no requirements.yaml � Can verify tests using: ``` go test ./experimental/air/cmd/ â�� pass go test ./acceptance -run TestAccept/experimental/air â�� pass gofmt clean ``` Manual verification: Instantiates a run succesfully with/without req.yaml dependencies declared: <img width="1673" height="1033" alt="Screenshot 2026-07-27 at 1 45 20 PM" src="https://github.com/user-attachments/assets/61ff9e24-225f-4cd2-87af-9b63030980b3" /> <img width="1166" height="354" alt="Screenshot 2026-07-27 at 1 46 08 PM" src="https://github.com/user-attachments/assets/97a0ffc9-a1e6-4e65-809f-a583bce6e083" />
## Changes Ports the `dcs register-image` capability (image registration) from the Python `ai-compute/cli` into the Go CLI as `air register-image`, under `experimental/air/cmd`. Mirrors a Docker image into the workspace registry. - `air register-image IMAGE_URL` registers an image and waits for it to become AVAILABLE, reporting the manifest digest (text or `-o json` envelope). - Registration always re-checks the source registry for the latest digest. - Credentials for private images are discovered from the local Docker config (`docker login` → `~/.docker/config.json`: credHelpers → credsStore → inline auth) and auto-stored in a per-user Databricks secret (creator-only ACL). If stored credentials are rejected, it retries once anonymously in case the image is public. ## Why Brings image registration to the Go `air` CLI so users on the Go binary can register private and public images. The credential-flag removal narrows the surface to a single, secure path so that creds are read from an existing `docker login` and stored per-user (never workspace-readable), so a registry PAT is never passed on the command line or leaked to other workspace members. ## Tests - Unit tests: URL normalization, status parsing, credential resolution order (incl. a credential-helper subprocess stub), secret scope/key storage + quota, error classification, and the anonymous-retry fallback. - Acceptance test (`acceptance/experimental/air/register-image/`) covers the registration flow, credential discovery (asserting the secret reference reaches the POST while the raw PAT never appears in output), and flag validation. - Manual Verification:
## Changes & Why usage_policy_id was also validated and then silently dropped: nothing wired it into the runs/submit payload. Both paths now populate budget_policy_id, the field the AI Runtime backend reads (matching the Python CLI). The resolver pages GET /api/2.0/serverless-policies with the partial, case-insensitive filter_by.policy_name filter, then re-applies an exact case-insensitive match locally. Not-found errors list candidate names; an ambiguous match refuses to guess rather than pick the wrong policy. Resolution happens before any artifact upload so a bad name fails fast. Also ports the UUID-shape check on usage_policy_id, so a policy name pasted into the id field gets an error pointing at usage_policy_name. ## Tests Unit tests: usagepolicy_test.go (new) - Wire format: filter_by.policy_name arrives as a flattened dotted key (not a nested map), page_size=1000 - Pagination: follows next_page_token; terminates on self-repeat and on A→B→A cycles; dedupes the same policy_id across pages - Resolution: exact match; case-insensitive exact wins over a partial sibling; not-found with candidate suggestions; not-found with no candidates omits the hint; suggestions capped at 10 with ...; ambiguous match refuses to guess; match missing policy_id; blank name rejected with zero API calls Unit tests: runsubmit_test.go / runconfig_test.go (modified) - TestSubmitWorkloadSendsUsagePolicy: id reaches BudgetPolicyId on the wire via both a literal id and a resolved name (asserted against the captured jobs.SubmitRun) - Unresolvable name fails before any workspace write — asserted by recording served paths - Empty payload case: no policy configured leaves BudgetPolicyId empty - Validation: non-UUID id rejected, a name pasted into the id field gets pointed at usage_policy_name, valid UUID accepted - Replaced the old usage_policy_name is not yet supported guard test Acceptance tests - go test ./acceptance -run 'TestAccept/experimental/air' passes with no golden-file changes needed - No new acceptance test added: the feature needs a workspace API response, which the unit tests cover via testserver
PR #6166 touched libs/cmdio/io.go only to delete IsPagerSupported, which routes the PR through the /libs/cmdio/ maintainer-approval gate. The helper already exists on main (added by #5847), so restoring it removes the shared-code diff entirely and lets the PR land as air-only. `air list` goes back to IsPagerSupported, which is the correct check: the inline navigable table writes rows to stdout, so stdout must be a TTY. IsPromptSupported only checks stderr+stdin, so with stdout redirected it would still start bubbletea and write escape sequences to the file. Co-authored-by: Isaac
These three fragments were consumed by the v1.9.0 release (2026-07-22) and deleted from main; all three entries are already published in CHANGELOG.md. They reappeared on this branch via the main catch-up merge (2f317bb), so re-adding them would duplicate the entries in the next release and routes PR #6166 through the general-files maintainer-approval gate. Co-authored-by: Isaac
## Changes Adds databricks experimental air run -h config.<field> — a help path that documents any field of the run YAML config from the command line. - `-h` config lists the top-level fields; `-h config.compute lists` a section's fields; `-h config.compute.accelerator_type` shows one field's type, required-ness, and description - Unknown fields produce a "did you mean" suggestion plus the valid siblings; free-form maps (parameters, env_variables, secrets) report that their keys are user-defined rather than implying a typo ## Why `air run` takes a YAML config with 35 fields across 9 nested structs, and until now the only way to learn a field was to read runconfig.go or trip its validation error. This surfaces the schema in the CLI itself. ## Tests - Unit: path resolution incl. bare paths, nested, slices, free-form maps and the 3 polymorphic unions; error cases incl. typo suggestion, distant-name (no suggestion), free-form key, scalar sub-field; renderer output for leaf vs container; command-help wiring (fallback, field help without --file, error-to-stderr with clean exit). - A guard test (TestConfigFieldsAllDocumented) fails if any schema field lacks a help: tag, so a new field can't merge undocumented. Verified it fails by removing a tag, then restored - Acceptance (acceptance/experimental/air/config-help/): 9 traced invocations pinning the rendered output Manual verification: <img width="1028" height="773" alt="Screenshot 2026-08-11 at 2 11 38 PM" src="https://github.com/user-attachments/assets/a6ba7236-e160-4ce0-8d01-826fbc12a8c7" /> Informational error outputs for unknown fields: <img width="1517" height="255" alt="Screenshot 2026-08-11 at 2 13 11 PM" src="https://github.com/user-attachments/assets/9e6ad1a5-ca85-4db7-a5b3-cfba47c7fc5d" />
#6244) ## Summary `air get`'s **Environment** cell always showed `N/A` for AI Runtime (serverless) runs — the kind `air run` submits. The environment version those runs use lives on the run's `environments[].spec.environment_version` (keyed by the task's `environment_key`), but the **typed SDK `jobs.Run` struct has no `environments` field**, so `w.Jobs.GetRun` silently drops it and the cell fell back to `N/A`. The cell is now resolved from that field via a raw `GetRun` request, matching the entry keyed `"default"` (`aiRuntimeEnvironmentKey`). The previous `gen_ai_compute` runtime-image source (`DlRuntimeImage`) is **dropped rather than special-cased** — that path is deprecated, so `gen_ai_compute` runs now show `N/A` for the environment. ## Changes - `experimental/air/cmd/render.go`: new `aiRuntimeEnvironmentVersion` helper — a raw GET to `/api/2.2/jobs/runs/get` that decodes only the `environments[]` array the typed SDK drops (same raw-call pattern as `usagepolicy.go` / `aitraining.go`). Best-effort: logs and returns `""` on any error or when the run declares no environment. `renderRunText` sets the cell from it (empty → stays `N/A`). - `experimental/air/cmd/get.go`: `buildGetData` no longer seeds the cell from the gen_ai `DlRuntimeImage`; environment is resolved at render time (it needs an extra API call). - `experimental/air/cmd/format.go`: removed the now-unused `environment(run)` helper. ## Testing - `experimental/air/get-ai-runtime` acceptance test: mock `runs/get` now returns an `environments` block; golden asserts `Environment 4`. - `experimental/air/get` (gen_ai) acceptance test: golden updated to `Environment N/A` (deprecated path). - `render_test.go` updated; `go test ./experimental/air/...` and `go test ./acceptance -run TestAccept/experimental/air` pass; package lints clean. - Verified live: `air get 796253027111946` shows `Environment 4`. Manual verification: <img width="1160" height="437" alt="Screenshot 2026-08-11 at 5 09 28 PM" src="https://github.com/user-attachments/assets/c36a4828-71cb-4b52-8beb-4c4b935ed901" />
…6241) ## Changes & Why Aligns `databricks experimental air run`'s submit UX with the Python `air` CLI: - Prints a "Submitting experiment: <name>" line before uploading (text mode). - Success line is now green "Submitted workload with Job Run ID: <id>", and the run URL is a terminal hyperlink ("View job run at:"). - Adds hyperlinked "View MLflow run at:" / "View MLflow experiment at:" links. The MLflow IDs are assigned only once the task run starts, so a short best-effort poll (bounded, ctx-cancellable) resolves them; the confirmation and job-run link print first so a bare submit is never blocked on links that usually aren't ready yet, and the two MLflow links are omitted on timeout. - Shows stderr spinners for the upload and snapshot-packaging phases (text mode, interactive terminals only). - The non-watch JSON envelope now reports status "PENDING" (matching Python), distinct from the --watch JSONL "SUBMITTED" event type. Colors, hyperlinks, and spinners degrade to plain output on non-TTY / NO_COLOR and are suppressed in JSON mode, so piped output and the JSON envelope stream stay clean. ## Tests Unit & acceptance tests all pass. Manual verification: [spinners are present] Standard air run outputs: <img width="1003" height="130" alt="Screenshot 2026-08-11 at 2 39 45 PM" src="https://github.com/user-attachments/assets/d3a35617-a8b4-40dd-99ee-9dc0cf325539" /> JSON style outputs: <img width="1003" height="158" alt="Screenshot 2026-08-11 at 2 40 47 PM" src="https://github.com/user-attachments/assets/b148ef5a-4929-4ed8-9c5b-8607493183a1" /> <img width="1160" height="827" alt="Screenshot 2026-08-11 at 2 44 05 PM" src="https://github.com/user-attachments/assets/3019fe12-95ad-4a97-9ade-21cb7d4b703a" />
## Summary **#6153 ("AIR CLI Migration: `--download-to` flag for logs") was lost from `air-cli`.** It merged on 2026-08-06, but `air-cli` was later rewound to `1fcb3c09a` before #6239 merged (08-12), and the rebuilt line (#6239 → #6244 → #6241) bypassed #6153. As a result `air-cli` today still carries the **pre-#6153 stub**: - `logs.go`: `--download-to is not implemented yet` (the flag is rejected) - no `logdownload.go` / `logdownload_test.go` - no `acceptance/experimental/air/logs-download/` test dir This PR restores #6153's change set onto the current `air-cli` tip. ## How Cherry-pick of #6153's original squash commit (`60cd876910cd`) onto `air-cli`. Verified equivalence to the original: - Every file except `logstream.go` is **byte-identical** to what #6153 landed. - `logstream.go` is re-merged against #6241's later edits to that file (git auto-merged it cleanly; both changes coexist). ## Testing - `go build ./experimental/air/...` — ok - `go test ./experimental/air/...` — 548 pass - `go test ./acceptance -run TestAccept/experimental/air` — 29 pass (incl. `logs` and the restored `logs-download`) - Package lints clean - Confirmed the `--download-to is not implemented` stub is gone and `logs.go` now wires the real implementation This pull request and its description were written by Isaac.
Render the Job Run and MLflow submit links via link() (blue, underlined, OSC 8 clickable), matching the air get view, so they read as hyperlinks instead of plain-looking URLs. Degrades to plain text on non-rich terminals. Co-authored-by: Isaac
…gs for a terminal run (#6245) ## Changes `air logs <run>` printed "No logs available for run <id>. Run terminated in state SUCCESS" and exited 0 for runs whose logs were fully retrievable: `air logs --download-to DIR` on the same run returned the complete log. This happens when Bricklens is enabled for the workspace but never ingested the run so it answers every request successfully with zero records, yet the logs are present in MLflow. The streaming (print) path only fell back to MLflow on errBricklensFeatureDisabled (gated off / not deployed / persistent failure); an empty-but-successful Bricklens response was treated as the final answer. The download path already reads from MLflow, which is why it worked. This is the Go version of the Python fix: https://github.com/databricks-eng/universe/pull/2384942 ## Tests 1. TestStreamBricklensEmptyFallsBackToMLflow: - terminal run — a TERMINATED/SUCCESS run whose Bricklens /logs returns {"log_records": []} must make streamBricklensLogs return errBricklensFeatureDisabled and emit nothing (so the caller falls back to MLflow rather than printing "No logs available"). - static view of a past retry — same assertion for the staticView: true path. 2. TestStreamBricklensTerminalWithRecordsDoesNotFallBack: the guard against over-eager fallback: a terminal run whose Bricklens stream does have a record prints it ("line":"hello"), returns true, no error — confirming the fallback only fires on a genuinely empty stream. 3. TestFetchLogsFallsBackToMLflowWhenBricklensEmpty: the end-to-end reproduction of your bug: one fake server serving empty Bricklens /logs plus a populated MLflow path (runs/get → get-output → artifacts/list → credentials-for-read → presigned chunk download). Asserts fetchLogs returns success and the two MLflow log lines (line 1, line 2) reach stdout — i.e. the print path now behaves like --download-to
…d i/L in `air list` (#6260) Enriches the interactive `air list` picker toward the Python AIR CLI. ## Changes 1. **Run ID and Experiment cells are OSC-8 hyperlinks** — Run ID → the job-run page, Experiment → the MLflow experiment page — alongside the existing MLflow link. Underlined only when a link is actually present; piped / `NO_COLOR` output stays plain (no escapes). 2. **MLflow column shows the MLflow run name** (with a `…<id8>` fallback) instead of a truncated `…/runs/<id>` URL. Reuses the existing `fetchMLflowRunName` / `mlflowRunLabel`; the resolved label is cached for terminal runs. 3. **`i` and `L`/`l` open an in-TUI scrollable pane** — `i` shows run details (the same styled view as `air get`), `L` shows a logs snapshot (`bubbles/viewport`). `esc` returns to the list with the cursor preserved. `enter` still opens MLflow in the browser. ## Notes - The logs snapshot is a one-shot tail (`staticView`), so viewing an **active** run’s logs can’t hang the pane waiting on a live stream. - The detail/logs panes render captured (`renderRunText` / `fetchLogs`) output; since the capture target isn’t a TTY they render in the ASCII profile (text and box borders intact, no color). A forced-color pane is a possible follow-up. - Reuses air-cli’s existing `mlflowExperimentURL` for the experiment link (no `?o=`, consistent with the other ML URLs) rather than adding a parallel helper. ## Tests - Unit: hyperlink rendering (asserts the OSC-8 escape is present with links, absent without), `i` → detail-mode transition, detail-pane content + `esc` back, run-name label. - Acceptance (`air list`): the MLflow column now shows the run name; golden regenerated. Full air unit + acceptance suite, `go vet`, `gofmt`, and `golangci-lint` all pass. This pull request and its description were written by Isaac. <img width="1631" height="471" alt="Screenshot 2026-08-13 at 9 38 35 PM" src="https://github.com/user-attachments/assets/7bca4471-0548-411d-a278-6c6bd67b030c" />
…before submit (#6205) ## What & why `air run` now pre-flights the config against the backend `ValidateConfig` RPC before uploading anything, so a bad config fails fast with the server's field-level errors instead of after the code snapshot is packaged and uploaded. The same rules back the submit gate, so the pre-flight can't disagree with what submission enforces. (See 1DD: https://docs.google.com/document/d/1xWKHisVk9YbsnmWyHE1J5OOTZTIHx2DSiyrkuTNM0NA/edit?tab=t.0#heading=h.culzh2kyug09 ) ## How do you know it works? `validateconfig_test.go` covers valid→pass, errors→blocking message (each pointing at its config field), fail-open on both `FEATURE_DISABLED` and 404, and the request-shape mapping (incl. omitting unset options). The submit acceptance tests (`run-submit`, `run-submit-deps`) exercise the pre-flight end-to-end: it fires, is served, and submission proceeds. `go test ./experimental/air/...` and the air acceptance tests pass. Manual verification (tested with the corresponding backend in liteswap pod): <img width="1446" height="963" alt="Screenshot 2026-08-11 at 12 00 38 PM" src="https://github.com/user-attachments/assets/f09dafed-1181-430c-afa9-e664d04e85dd" /> ## How to review - `validateconfig.go`: `preflightValidate` builds the `{task, run_options}` body from `runConfig`, POSTs to `/api/2.0/ai-training/config:validate` (raw `client.Do`, since the SDK doesn't model AiTrainingService, matching `aitraining.go`), and renders any `FieldError`s. - the endpoint is behind a SAFE flag (default off) and older workspaces lack it, so `FEATURE_DISABLED` / 404 / 501 skip the check and let submission proceed and only a populated error list blocks. - `runsubmit.go` the call is the first thing `submitWorkload` does, before token/policy resolution and any upload. - `--dry-run` is unchanged since it stays local-only and needs no workspace
riddhibhagwat-db
requested review from
maggiewang-db and
vinchenzo-db
and removed request for
maggiewang-db
August 14, 2026 06:56
Contributor
Approval status: pending
|
## Changes & Why environment.dependencies now accepts only an inline list of packages. The string form (a path to a requirements.yaml file) is rejected at config load, now that inline deps are fully supported (#6077). Removes the file-reading paths (readRequirementsDependencies, requirementsDoc, requirementsFile) and the file-vs-inline branching in run submit and convert-to-dabs; the dependencies union collapses to a plain list. ## Tests - TestLoadRunConfig_PolymorphicFields: requirements.yaml fails at load with must be a list of packages - Acceptance rejection case: end-to-end proof the CLI rejects the file-path form with the actionable error message, captured verbatim in output.txt. - TestRunConfigDependencies: the inline list still decodes and is returned by inlineDependencies(); unset returns false. - TestLoadRunConfig_FullFeatured: a full config with an inline dependencies: list parses into Dependencies.{set, list} correctly. - TestBuildSubmitPayloadInlineDependencies: inline deps are carried on environments[].spec.dependencies in the submit payload (empty/nil omit the key). - Acceptance inline happy path: a real submit uploads only command.sh + training_config.yaml (no requirements.yaml) and the recorded runs/submit body carries the deps inline. - TestBuildArtifacts_ParametersButNoRequirements: inline deps are not written as an uploaded artifact. - TestEnvironmentConfigValidate: environment.version is valid only alongside inline deps (the old file-deps branch is gone). - Full-featured convert test: the convert-to-dabs path folds inline deps + version into the bundle environments[].spec, proving both consumers stay in lockstep on the inline-only path. Manual Verification: Rejected: <img width="1243" height="173" alt="Screenshot 2026-08-12 at 9 51 21 AM" src="https://github.com/user-attachments/assets/30ede5b3-846f-47c8-8340-c63eb8ebf416" /> Accepted: <img width="1243" height="272" alt="Screenshot 2026-08-12 at 9 51 29 AM" src="https://github.com/user-attachments/assets/8e9bab5a-a20d-4859-826c-986d2838c762" />
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
UI Enhancements, bug fixes (from bug bash), ensured parity with Python CLI, completed validation port to backend. See individual PRs for more information on the specific changes & tests.