Skip to content

[BigQueryIO] Parallelize schema update integration tests - #39622

Open
bvolpato wants to merge 1 commit into
apache:masterfrom
bvolpato:bvolpato/parallelize-schema-update-it
Open

[BigQueryIO] Parallelize schema update integration tests#39622
bvolpato wants to merge 1 commit into
apache:masterfrom
bvolpato:bvolpato/parallelize-schema-update-it

Conversation

@bvolpato

@bvolpato bvolpato commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

StorageApiSinkSchemaUpdateIT currently runs all 32 parameterized cases serially inside one test class. Gradle already gives GCP integration tests four forks, but it can only distribute work between classes.

This splits the matrix by useInputSchema into two concrete test classes backed by the same test implementation. Each class owns a separate BigQuery dataset, so parallel setup and cleanup cannot interfere. Early-rollout coverage includes both classes explicitly.

No test cases, assertions, row counts, stream counts, schema triggers, polling intervals, or soak delays change.

Live result

Java_GCP_IO_Direct passed on this PR. Its integrationTest task completed in 1h24m52s.

Four nearby successful runs without this split took 1h43m07s to 1h47m26s, with a 1h44m37s median: 31012265163, 31014377562, 31028738723, and 31042299715. This PR reduced that median by 19m45s, or 19%.

The full Gradle command completed in 1h46m53s versus a 2h10m41s baseline median, an 18% reduction. Full-command timing also includes compilation and cache variance, so integrationTest is the cleaner comparison.

Historical timing model

Scheduled master run 30486556613 recorded all 32 schema-update cases passing in 81m49s. Grouping those measured cases by the new class boundary gives:

Execution unit Historical test time
Current single class 81m49s
Without input schema 52m10s
With input schema 29m39s
Expected split critical path 52m10s

This predicts a 29m39s, or 36%, reduction for this critical shard when both classes receive existing Gradle workers. It does not increase maxParallelForks or add another workflow job.

Two classes are intentional. The longest individual parameter took 49m11s, only 2m59s below the two-class estimate. Splitting all four parameters would therefore add more concurrent Storage Write API tests for little additional critical-path gain.

Testing

  • ./gradlew :sdks:java:io:google-cloud-platform:compileTestJava -PdisableSpotlessCheck=true
  • ./gradlew :sdks:java:io:google-cloud-platform:spotlessJavaCheck
  • integrationTest --test-dry-run: 32 cases discovered across two classes, no base-class tests
  • bigQueryEarlyRolloutIntegrationTest --test-dry-run: same 32 cases discovered
  • Java_GCP_IO_Direct: passed against live GCP services

  • No existing issue to link.
  • No CHANGES.md entry; test scheduling only.
  • Small contribution; ICLA not applicable.

@bvolpato
bvolpato marked this pull request as ready for review August 5, 2026 22:53
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @ahmedabu98 for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@ahmedabu98 ahmedabu98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nice optimization

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants