Fail environment validation when standalone pyspark collides with databricks-connect

[rugpanov]

Stacked on #6276 (base branch dbconnect/pyspark-merge-warning). Review that one first.

Problem

The validate phase read only the Python and databricks-connect versions, so a venv
with a standalone pyspark installed on top of databricks-connect passed validation
and setup-local reported "environment ready" — even though the two share the
pyspark namespace, overwrite each other, and the environment then cannot start a
session. Users hit this later as an opaque Java or protobuf gencode error with no link
back to setup.

What

• PackageManager.Validate now also returns the standalone pyspark version. It reads
  distribution metadata: databricks-connect vendors pyspark without registering a
  pyspark distribution, so a non-empty value means a separate pyspark was installed
  on top of it — the collision.
• The pipeline validate phase fails with E_VALIDATE, and actionable guidance, when
  both databricks-connect and a standalone pyspark are present. Keyed on both being
  installed in the venv, not on the run mode.
• uvManager.Validate probes pyspark alongside databricks-connect; the interface
  and mocks are updated to the new signature.

Compatibility note

This is a behaviour change: environments that today report "ready" with both packages
installed will now fail validation. That is the intended correction — the previous
"ready" was misleading — but it is worth a conscious sign-off.

Testing

• go test ./libs/localenv/ (new TestPipelineValidateRejectsStandalonePyspark plus
  the existing suite).
• go vet ./libs/localenv/.

This pull request and its description were written by Isaac.

[github-actions]

Waiting for approval

Could not determine reviewers from git history.
Round-robin suggestion: @misha-db

Eligible reviewers: @anton-107, @misha-db, @rclarey

_{Suggestions based on git history. See OWNERS for ownership rules.}

[eng-dev-ecosystem-bot]

Integration test report

Commit: 550e991

Run: 31831053412

	Env	🟨​KNOWN	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
🟨​	aws linux	3	1	4	286	1146	7:38
🟨​	aws windows	3	1	4	288	1144	7:03
🟨​	azure linux	3	1	4	285	1146	7:48
🟨​	azure windows	3	1	4	287	1144	8:08
💚​	gcp linux		1	5	286	1146	6:07
💚​	gcp windows		1	5	288	1144	5:45

8 interesting tests: 4 SKIP, 3 KNOWN, 1 RECOVERED

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🟨​	TestFetchRepositoryInfoAPI_FromRepo	🟨​K	🟨​K	🟨​K	🟨​K	🙈​S	🙈​S
🟨​	TestFetchRepositoryInfoAPI_FromRepo/root	🟨​K	🟨​K	🟨​K	🟨​K
🟨​	TestFetchRepositoryInfoAPI_FromRepo/subdir	🟨​K	🟨​K	🟨​K	🟨​K

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
5:41	azure windows	TestAccept
5:37	aws windows	TestAccept
5:37	gcp windows	TestAccept
3:13	azure linux	TestAccept
3:07	aws linux	TestAccept
3:01	gcp linux	TestAccept

[anton-107]

Reviewed as part of the three-PR stack (#6276, this, and databricks/databricks-vscode#2121). Not approving yet — one finding below is, I think, worth settling before this merges, since this is the PR that converts a warning into a hard error.

What I verified

The mechanism is sound. databricks-connect ships the pyspark package tree but registers no pyspark distribution, so reading distribution metadata is the right probe. Confirmed on 13.3.12, 14.3.19, 17.3.8 and 18.0.6 — stable across four majors, so the probe is not resting on a quirk of one release.

Against a genuinely collided venv the probe reports exactly what the PR claims:

databricks-connect -> 18.0.6
pyspark            -> 4.2.0

and that venv is genuinely broken (ImportError: cannot import name 'PythonUDFEnvironment' at from databricks.connect import ...). Failing validation rather than reporting "ready" is the right instinct for that case.

Finding 1 — this hard-fails a working environment

The load-bearing claim is "the two cannot coexist." That is install-order dependent, and I have a counterexample.

Installing standalone pyspark first, then databricks-connect over it:

probe:  databricks-connect -> 18.0.6 ,  pyspark -> 4.2.0     ← this PR hard-fails
actual: import OK, pyspark.__version__ == 4.1.0              ← env is FUNCTIONAL

databricks-connect's files won the overwrite, so the venv works — but the orphaned pyspark-4.2.0.dist-info is left behind, the probe reads it, and E_VALIDATE fails an environment that runs fine.

The version mismatch is the tell: importlib.metadata.version("pyspark") returns 4.2.0 while the importable pyspark.__version__ is 4.1.0 (dbconnect's vendored tree). The probe is reading a stale dist-info, not the live module. That also makes the error message wrong in its specifics — it prints standalone pyspark 4.2.0, a version that is not actually importable.

uv sync resolution order isn't something a user controls, so this isn't hypothetical.

Worth considering: confirm the collision is live before failing — compare importlib.metadata.version("pyspark") against pyspark.__version__, or attempt the databricks.connect import inside the probe. Either distinguishes a real collision from an orphaned dist-info and keeps the hard fail truthful. The PR explicitly asks for a conscious sign-off on the behaviour change, and I think this is the case that deserves it: "the previous ready was misleading" holds for a genuine collision, but not for this one, where the previous "ready" was correct.

Finding 2 — gating disagrees with #6276

This fails on venv contents (dbcVer != "" && pysparkVer != ""); #6276's warning gates on the mode (c.DatabricksConnect != ""). The mismatch is reachable, because mergeDatabricksConnect is a no-op on an empty value rather than a deletion (merge.go:351) — so in constraints-only mode a project that already pins databricks-connect itself keeps that pin and uv installs it:

constraints-only + project's own dbconnect pin + standalone pyspark
  → merge:    warning SKIPPED   (#6276, gated on mode)
  → validate: HARD FAIL E_VALIDATE  (here, gated on venv contents)

No warning at merge, then an unexplained hard failure at validate — and since dbcPin == "" on that path the user isn't told which databricks-connect is involved. The comment here ("Keyed on both packages actually being present in the venv, not on the mode") is the better reasoning of the two; I've suggested on #6276 that its warning match it. Whichever way it's settled, the two should agree.

Testing

TestPipelineValidateRejectsStandalonePyspark asserts code, phase, and message content — good. Two gaps:

• no test for reverse install order (Finding 1 — the case that makes this fail a working env);
• no test for constraints-only + the user's own dbconnect pin (Finding 2).

The interface change to PackageManager.Validate and the mock updates are mechanical and look complete.

I could not run the Go suites here: the repo requires Go 1.26, the local toolchain is 1.24, and the module proxy is blocked in my environment. I verified the logic by reading it and by reproducing the package behaviour directly in Python, not by executing the tests. Taking the reported go test ./libs/localenv/ pass at face value.

Note on sequencing

This is stacked on #6276, which I've approved. If #6276's gating changes per Finding 2, this will need a rebase.

[anton-107]

Follow-up: I found a working PyPI mirror (pypi-proxy.cloud.databricks.com) and re-ran Finding 1 with real uv instead of the hand-layered venv I used before. This closes the caveat I noted earlier and, I think, strengthens the finding rather than weakening it.

Both cases reproduced through genuine uv resolution

Case A — declared together in one pyproject.toml, single uv sync:

dependencies = ["databricks-connect==18.0.6", "pyspark==4.2.0"]

probe:  databricks-connect -> 18.0.6 ,  pyspark -> 4.2.0
live:   pyspark.__version__ == 4.2.0        ← agrees with metadata
import: ImportError: cannot import name 'PythonUDFEnvironment'

Genuinely broken. This PR fails it, correctly.

Case B — standalone pyspark already present, databricks-connect added later:

uv pip install pyspark==4.2.0           (user's pre-existing project)
uv pip install databricks-connect==18.0.6   (setup-local, later)

probe:  databricks-connect -> 18.0.6 ,  pyspark -> 4.2.0   ← this PR HARD-FAILS
live:   pyspark.__version__ == 4.1.0                        ← dbconnect's vendored tree won
import: OK — environment is FUNCTIONAL

The part that matters most

I then wrote Case B as a pyproject.toml and ran the actual uv sync that setup-local performs:

[project]
dependencies = ["pyspark==4.2.0"]
[dependency-groups]
dev = ["databricks-connect==18.0.6"]

$ uv sync
Resolved 31 packages in 3.17s
Audited 30 packages in 0.04ms      ← no reinstall; functional state preserved

probe:  databricks-connect -> 18.0.6 , pyspark -> 4.2.0
live:   pyspark.__version__ == 4.1.0
import: OK — still FUNCTIONAL

uv sync audits rather than reinstalls, so it does not normalize the layout into the broken Case-A state. That means setup-local genuinely reaches this configuration and would fail an environment that works. My earlier report left open the possibility that this was an artifact of hand-copying files into a venv — it isn't.

The version mismatch is the tell in both directions: metadata says 4.2.0, the importable module is 4.1.0. The probe is reading a dist-info that no longer describes the live pyspark.

Both proposed discriminators verified

Tested against both venvs:

discriminator	Case A (broken)	Case B (functional)
metadata.version("pyspark") vs pyspark.__version__	4.2.0 == 4.2.0 → fail	4.2.0 != 4.1.0 → don't fail
attempt from databricks.connect import DatabricksSession	ImportError → fail	OK → don't fail

Either one separates the cases cleanly. The import probe is the more direct test of the property you actually care about ("can this environment start a session"), and it needs no assumption about how the shadowing manifests; the version comparison is cheaper and avoids importing Spark during validate.

One caveat I still can't close: I couldn't confirm Case B builds a live SparkSession end-to-end, since that needs workspace connectivity this sandbox doesn't have. So the strict claim is "imports cleanly and is not the failure mode this PR describes" — the import is where the collision actually surfaces, per the traceback above. If dbconnect turns out to fail later at session time in Case B too, that would argue for keeping the fail, and the version comparison would then be the wrong discriminator while the message would still need fixing (it reports 4.2.0, which is not the importable version).

Everything else in my original review stands, including the Finding 2 gating mismatch with #6276.

[rugpanov]

Thanks for the thorough review — especially reproducing Finding 1 through a real uv sync. Both findings are addressed in the latest push.

Finding 1 — hard-failing a working environment. You're right: the metadata-only probe read an orphaned pyspark-*.dist-info and failed an environment that imports fine. Switched to your more direct discriminator — the validate probe now attempts import databricks.connect, and the hard fail requires all three: databricks-connect present, a standalone pyspark present, and the import actually failing. A stale dist-info (import succeeds) is no longer rejected.

• Verified against a real functional databricks-connect venv → imports clean → no fail; and against the reproduced collision → ImportError → fail.
• Validate now returns a VenvInfo struct (python minor, databricks-connect version, pyspark version, databricks-connect import error) instead of a growing tuple.
• The message now reports the import error plus the colliding versions, so it's truthful in the case it fires.

Finding 2 — gating disagreement with #6276. Aligned on the venv-based reasoning: #6276's warning now fires whenever databricks-connect is present by any route (mode or a user-pinned databricks-connect kept in constraints-only mode), matching this PR's databricks-connect present && pyspark present && import fails.

Testing gaps. Added TestPipelineValidateAllowsStalePysparkDistInfo (functional env with a leftover pyspark dist-info → must not fail — the reverse-install-order case) here, and TestStandalonePysparkFiresInConstraintsOnlyWhenUserPinsDBConnect on #6276.

Rebased onto the updated #6276.