Add Cluster Policy support to bundle resources - #6225
Open
Sankalp-Mittal wants to merge 29 commits into
Open
Conversation
Collaborator
Integration test reportCommit: 1ce85e0
8 interesting tests: 5 RECOVERED, 3 SKIP
Top 8 slowest tests (at least 2 minutes):
|
Sankalp-Mittal
commented
Aug 11, 2026
| "vector_search_indexes", | ||
| "volumes", | ||
| }, | ||
| require.Equal( |
Collaborator
Author
There was a problem hiding this comment.
This change is due to the linter
Sankalp-Mittal
force-pushed
the
sankalp-mittal/cluster-policy-resource
branch
2 times, most recently
from
August 13, 2026 12:19
c67e925 to
01b7a51
Compare
The cluster_policies resource was added without regenerating derived files, failing validate-generated and the refschema acceptance test. Co-authored-by: Isaac
Adding cluster_policies with a workspace URL pattern extended the list of openable resource types, but the workspace_open command tests hardcoded the old list. Add cluster_policies to the expected completion, help text, and unknown-type error assertions. Co-authored-by: Isaac
The workspace open command's supported-resource-type list now includes cluster_policies; regenerate the golden output. Co-authored-by: Isaac
TestInvariantConfigsCoverage requires every resource type to have an invariant config. Add a cluster_policy config and register it in the invariant matrix. Exclude it from the migrate suite (terraform-seeded; the resource is direct-only) and continue_293 (unsupported on the old CLI). Regenerate the affected out.test.toml snapshots. Co-authored-by: Isaac
Restructure the single cluster_policies acceptance test into a group:
- move the existing test unchanged into basic/
- add job_ref/: a job task referencing the policy via
${resources.cluster_policies.pol.id}, asserting the direct engine
orders policy create before job create (and job delete before policy
delete on destroy) and resolves the policy id into the job body
- add definition_multiline/: a block-scalar JSON definition, asserting
it is preserved as a newline-escaped string end to end
No production code change.
Co-authored-by: Isaac
The cluster policy `definition` was a plain JSON string. Add a top-level `Definition any` field that shadows the embedded compute.CreatePolicy string so the definition can also be written as native YAML. ConfigureClusterPolicyDefinition normalizes an inline map/sequence to a JSON string at the dyn layer (same approach as genie serialized_space), avoiding int/float structdiff drift; PrepareState copies the normalized string into state. A string definition passes through unchanged. Co-authored-by: Isaac
- unit test for ConfigureClusterPolicyDefinition (string passthrough, map/sequence -> JSON string, invalid-kind diagnostic, absent field) - unit test for ResourceClusterPolicy.PrepareState copying the normalized string into state - acceptance test definition_yaml/: a native YAML mapping serializes to the same compact JSON string the API receives as the basic test Co-authored-by: Isaac
The inline-YAML definition feature added an 'any'-typed definition input field but did not regenerate out.fields.txt, failing validate-generated. Co-authored-by: Isaac
Add a direct-only acceptance test asserting that deploying a cluster_policies resource with DATABRICKS_BUNDLE_ENGINE=terraform fails with an actionable error, mirroring secrets/direct-only. Co-authored-by: Isaac
Add an acceptance test where a job new_cluster references an undeclared cluster_policies resource; bundle plan fails with a config-time dependency-resolution error. Co-authored-by: Isaac
One bundle creates a cluster policy; a second, separate bundle consumes it by hardcoding the generated policy_id (captured from the first bundle's state) in a job's new_cluster. Asserts the consumer job's create request carries the producer's policy id. Co-authored-by: Isaac
Make the testserver's /api/2.0/policies/clusters/list stateful so it
returns policies created via the create handler, seeding the two legacy
names the variable-lookup tests rely on. Add an acceptance test where a
consumer bundle resolves a policy created by a separate producer bundle
through a variable lookup {cluster_policy: <name>}, then uses the
resolved id in a job.
Co-authored-by: Isaac
Sankalp-Mittal
force-pushed
the
sankalp-mittal/cluster-policy-resource
branch
from
August 13, 2026 13:40
08272eb to
a93ad20
Compare
Instead of inline JSON everywhere, spread the three authoring forms across the non-targeted tests for incidental coverage: job_ref and cross_bundle_lookup use a multiline JSON block scalar, direct-only uses native YAML, cross_bundle_id keeps inline JSON. basic and the two targeted definition tests are unchanged. Co-authored-by: Isaac
Sankalp-Mittal
marked this pull request as ready for review
August 13, 2026 13:58
Contributor
Approval status: pending
|
| case dyn.KindInvalid, dyn.KindNil, dyn.KindString: | ||
| // KindInvalid means definition is absent; leave it for backend validation. | ||
| return v, nil | ||
| case dyn.KindMap, dyn.KindSequence: |
Contributor
There was a problem hiding this comment.
why do we allow KindSequence? yes, it's valid JSON but it's not valid policy language (or is it?)
| @@ -0,0 +1,2 @@ | |||
| Error: invalid dependency "${resources.cluster_policies.missing.id}", no such node "resources.cluster_policies.missing" | |||
Contributor
There was a problem hiding this comment.
this test case is unnecessary for cluster policy specifically
| @@ -0,0 +1,4 @@ | |||
| title "Deploy should fail in terraform mode" | |||
| trace $CLI bundle deploy 2>&1 | contains.py \ | |||
| "Cluster Policy resources are only supported with direct deployment mode" \ | |||
Contributor
There was a problem hiding this comment.
no need for a resource-specific test for this - we already encode resources not supported in TF elsewhere
| cluster_policies: | ||
| pol: | ||
| name: my_policy | ||
| definition: |
Contributor
There was a problem hiding this comment.
this test would be more powerful if you also added a non-string field as part of the definition to make sure yml -> json mapping is done correctly
|
|
||
| # catalogs: no api field behaviors | ||
|
|
||
| # cluster_policies: no api field behaviors |
Contributor
There was a problem hiding this comment.
would be great to add hashed_in_state behaviour for the policy field after #6105 lands & cover with acceptance test for a large policy doc
| >>> [CLI] bundle validate -o json | ||
| { | ||
| "pol": { | ||
| "definition": "{\n \"spark_version\": {\n \"type\": \"fixed\",\n \"value\": \"13.3.x-scala2.12\"\n }\n}", |
Contributor
There was a problem hiding this comment.
I'd be interested to see if the backend normalises/formats this json
| @@ -0,0 +1,17 @@ | |||
| title "Bundle A creates the cluster policy" | |||
Contributor
There was a problem hiding this comment.
not sure about the value of the cross_bundle_ tests - this is not really resource specific and standard DABs behaviuor
| @@ -0,0 +1,6 @@ | |||
| Cloud = false | |||
Contributor
There was a problem hiding this comment.
please enable cloud for some of the tests once you have run them and verified they pass
|
|
||
| trace $CLI bundle summary | ||
|
|
||
| title "Update the cluster policy name" |
Contributor
There was a problem hiding this comment.
can you add a new test case (not in basic/) for an out-of-band edit to the policy?
Comment on lines
+211
to
+213
| // Reads (dynamic): resources.cluster_policies.*.definition | ||
| // Updates (dynamic): resources.cluster_policies.*.definition (inline YAML -> JSON string) | ||
| ConfigureClusterPolicyDefinition(), |
Contributor
There was a problem hiding this comment.
Seems there's definition and policy_family_definition_overrides, both are JSON policies
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Add support for a new
cluster_policiesresource type in DABs. Cluster policies are admin-defined guardrails that constrain how clusters can be configured (allowed node types, fixed autotermination, required tags, cost limits). This lets users manage them declaratively alongside their other bundle resources. This includes:resources.ClusterPolicyconfig type, wrappingcompute.CreatePolicy.ResourceClusterPolicy) covering create/read/update/delete plusPrepareState/RemapState.all.go), dev-mode name prefixing (presets), and the workspace URL pattern.definitionas native inline YAML, in addition to a JSON string or block scalar (see "Key points" below)./api/2.0/policies/clusters/*endpoints.Cluster policies are only supported in direct deployment mode — they have no Terraform provider mapping here, so they are excluded from Terraform-mode conversion.
Why
Cluster policies were previously only referenceable from bundles via a variable
lookup(resolving an existing policy name to itspolicy_id). Users want to define and own the policies themselves as code, instead of provisioning them out-of-band through the UI.Example
The
definitioncan be an inline JSON string:…or native YAML, which is normalized to the same JSON string at deploy time:
Key points to note
Update uses
Edit, and it is a full replace. The SDK's update call isClusterPolicies.Edit(notUpdate);DoUpdatesends the whole desired policy along with the existingpolicy_id.Output-only fields are excluded from state.
policy_id,created_at_timestamp,creator_user_name, andis_defaultare returned on read but never written back, soRemapStatecopies only the fields shared withCreatePolicy.Dev-mode name prefixing. The policy
nameis a user-facing display name (unique, 1–100 chars), not the API identity (policy_id), so it is prefixed in development mode to avoid collisions between developers without changing identity.Inline-YAML
definitionreplicates the Genieserialized_spacepattern. A top-levelDefinition anyfield shadows the embeddedcompute.CreatePolicy.Definition string, and a per-resourcedyn-layer mutator (ConfigureClusterPolicyDefinition, registered right afterConfigureGenieSpaceSerializedSpace()) normalizes it before deploy: a string passes through unchanged, an inline map/sequence is marshaled to a compact JSON string (json.Marshal, sorted keys), and any other kind is rejected with a diagnostic.PrepareStatethen copies the normalized string into state; the state type stayscompute.CreatePolicy, so there is no state-compatibility break.The conversion is done at the config layer on purpose: authoring structured content inline in YAML decodes small integers as Go
int, while the state JSON round-trip decodes them asfloat64, sostructdiffreports false drift on every plan. Normalizing to a canonical JSON string makes config-side and state-side hold an identical string — the same reason Genie does it. The dashboardserialized_dashboardmutator is deliberately not the model here: it only reads afile_pathinto an opaque string and does no YAML→JSON conversion.Tests
acceptance/bundle/resources/cluster_policies/:basic/— single-policy validate/summary/deploy plus create/edit/delete, pinned to the direct engine.job_ref/— a job task referencing the policy via${resources.cluster_policies.pol.id}; asserts the direct engine orders policycreatebefore jobcreate(and job delete before policy delete on destroy) and resolves the policy id into the job body.definition_multiline/— a block-scalar JSON definition preserved as a\n-escaped string end to end.definition_yaml/— a native YAML mapping that serializes to the same compact JSON string the API receives asbasic/.direct-only/— deploying in terraform mode fails with the actionable "only supported with direct deployment mode" error.missing_ref/— a job referencing an undeclaredcluster_policies.*resource fails to plan with a dependency-resolution error.cross_bundle_id/— one bundle creates a policy; a separate bundle consumes it by hardcoding the generatedpolicy_id(captured from the producer's state) in a job.cross_bundle_lookup/— a separate bundle resolves the policy created by the producer via a variablelookup: {cluster_policy: <name>}and uses the resolved id in a job.ConfigureClusterPolicyDefinition(string passthrough, map/sequence → JSON, invalid-kind diagnostic, absent field) and forResourceClusterPolicy.PrepareState.GET /api/2.0/policies/clusters/liststateful (returning created policies, seeded with the two legacy names the variable-lookup tests rely on) so the lookup-by-name flow is exercisable.cluster_policiescoverage to shared suites: bundle permissions, presets/target-mode prefixing, run-as, state-load round-tripping, and the Terraform lifecycle exclusion list.This pull request and its description were written by Isaac.