AIR CLI: send dependencies inline, stop uploading requirements.yaml

[riddhibhagwat-db]

Changes & Why

air run now carries the user's declared dependencies (which may be an inline list, or read from a requirements.yaml file) on the submission's environments[].spec.dependencies, and no longer uploads a requirements.yaml artifact at all. This is the follow up PR to https://github.com/databricks-eng/universe/pull/2178617?timeline_per_page=5 (implementing this method in the python CLI) and https://github.com/databricks-eng/universe/pull/2297011?timeline_per_page=5 (follow up backend changes to unblock the new path; installs the inline deps via --deps-config and treats a missing co-located requirements.yaml as "no requirements").

This removes the vestigial empty requirements.yaml that a no-dependency run used to upload just to satisfy the launcher's derived path. When no dependencies are declared, spec.dependencies is omitted and the payload is unchanged. A -r/--requirement include in a requirements file is rejected, since the referenced file is never uploaded with the run.

Tests

Unit tests:

• Upload side: TestBuildArtifacts_CommandAndConfig, TestBuildArtifacts_ParametersButNoRequirements, TestBuildArtifacts_RequirementsFileNotUploaded, TestBuildArtifacts_EnvVarsAndSecrets, TestBuildArtifacts_OversizeConfigRejected
• Submit side: TestBuildSubmitPayloadInlineDependencies, TestEnvironmentDependencies, TestReadRequirementsDependencies, TestEnvironmentDependencies_MissingRequirementsFile
• End-to-end (unit): TestSubmitWorkload / TestSubmitWorkloadWithCodeSource
  ��
  Acceptance tests:
  acceptance/experimental/air/run-submit-deps/ -> submits with inline deps, golden asserts:
• spec.dependencies: [numpy, torch==2.3.0] on the runs/submit wire ��
• Only command.sh + training_config.yaml uploaded no requirements.yaml �

Can verify tests using:

go test ./experimental/air/cmd/ � pass
go test ./acceptance -run TestAccept/experimental/air � pass
gofmt clean

Manual verification:
Instantiates a run succesfully with/without req.yaml dependencies declared:

[eng-dev-ecosystem-bot]

Integration test report

Commit: e87da09

Run: 30386921385

	Env	❌​FAIL	🟨​KNOWN	🔄​flaky	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
❌​	aws-cli-is linux	8	1		3	4	320	1036	7:33
❌​	aws-cli-is windows	8	1		3	4	322	1034	20:51
❌​	azure-cli-is linux	8	1		3	4	320	1035	7:27
❌​	azure-cli-is windows	8	1		3	4	322	1033	13:27
❌​	gcp-cli-is linux	8	4	2		4	317	1037	11:07
❌​	gcp-cli-is windows	8	4			4	321	1035	16:44

18 interesting tests: 8 FAIL, 4 KNOWN, 4 SKIP, 2 flaky

	Test Name	aws-cli-is linux	aws-cli-is windows	azure-cli-is linux	azure-cli-is windows	gcp-cli-is linux	gcp-cli-is windows
🟨​	TestAccept	🟨​K	🟨​K	🟨​K	🟨​K	🟨​K	🟨​K
❌​	TestAccept/bundle/deploy/mlops-stacks	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/generate/auto-bind	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/generate/auto-bind/DATABRICKS_BUNDLE_ENGINE=terraform	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
❌​	TestAccept/bundle/resources/grants/volumes	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/resources/grants/volumes/DATABRICKS_BUNDLE_ENGINE=direct	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/resources/grants/volumes/DATABRICKS_BUNDLE_ENGINE=terraform	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🔄​	TestFilerWorkspaceNotebook	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
🔄​	TestFilerWorkspaceNotebook/scalaNb.scala	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
🟨​	TestFetchRepositoryInfoAPI_FromRepo	💚​R	💚​R	💚​R	💚​R	🟨​K	🟨​K
🟨​	TestFetchRepositoryInfoAPI_FromRepo/root	💚​R	💚​R	💚​R	💚​R	🟨​K	🟨​K
🟨​	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚​R	💚​R	💚​R	💚​R	🟨​K	🟨​K

[maggiewang-db]

Can you do the validation before uploadArtifacts?

[pardis-beikzadeh-db]

I think mentioning "Deps are no longer uploaded as a requirements.yaml" is confusing for anyone who reads this later without the historical context — it references impl that no longer exists. Suggest just describing what the code does now (deps ride on spec.dependencies), dropping the "no longer" framing.

[pardis-beikzadeh-db]

File-form deps lose their version. readRequirementsDependencies returns only doc.Dependencies and drops doc.Version; the wire version comes solely from top-level environment.version — which validation rejects for file-form ("set the version inside that file"). So dependencies: ./reqs.yaml with version: 5 inside silently gets the default image, and the error message points users at the field we ignore. Python reads it (_parse_requirements_yaml), so this also diverges from the Python CLI.

Fix: return the file's version (the requirementsDoc.Version you already decode — currently dead) and use it when top-level version is unset. Worth an acceptance case: file-form with in-file version: → assert environment_version on the wire.

[pardis-beikzadeh-db]

Minor: the -r/--requirement rejection only runs on file-form; inline dependencies: ["-r foo.txt"] passes straight to spec.dependencies. Python has the same gap so it's not a divergence — flagging in case we close it in both. Non-blocking.