feat(pruning): expose pruning predicate IN-list rewrite size cap as a config option - #24074
Open
zhuqi-lucas wants to merge 8 commits into
Open
feat(pruning): expose pruning predicate IN-list rewrite size cap as a config option#24074zhuqi-lucas wants to merge 8 commits into
zhuqi-lucas wants to merge 8 commits into
Conversation
Issue: apache#24059 `PruningPredicate` rewrites `col IN (v1..vn)` into a chain of per-value min/max checks (via `build_predicate_expression`), but only when `n` is below a hardcoded `MAX_LIST_VALUE_SIZE_REWRITE = 20`. Beyond that, the IN branch falls through to `unhandled_hook`, which by default returns `TRUE` — so row-group / file-range statistics pruning does not fire at all for IN lists longer than 20. This is problematic for query patterns that pass a batch of identifiers as `col IN (...)` (REST endpoints filtering by a page of ~25-100 values, ORM-generated `WHERE id IN (25 items)` queries, batched crawlers). On a table sorted by `col`, the reader is forced to materialize the filter column across every row group instead of skipping row groups whose stats disagree with the IN set. ## Changes - Add `datafusion.execution.parquet.pruning_max_in_list_size: usize` (default `20`, preserving existing behaviour) to `TableParquetOptions` next to `max_predicate_cache_size`. - Add `PredicateRewriter::with_max_in_list_size(usize) -> Self` builder, mirroring the existing `with_unhandled_hook`. - Add `PruningPredicate::try_new_with_max_in_list_size` variant. - Add `build_pruning_predicate_with_max_in_list_size` variant of the public helper. - Make `MAX_LIST_VALUE_SIZE_REWRITE` `pub const` so callers can reference the historical default explicitly. - Wire the value through `datasource-parquet`: - `ParquetSource::pruning_max_in_list_size()` reads from `TableParquetOptions.global`. - `ParquetMorselizer` / `PreparedParquetOpen` / `RowGroupPruner` carry the value alongside `max_predicate_cache_size`. - `build_pruning_predicates` (opener) accepts the size and forwards to `build_pruning_predicate_with_max_in_list_size`. ## Backward compatibility - Public `PruningPredicate::try_new` and `build_pruning_predicate` are preserved as thin wrappers passing the historical default. - Internal `build_predicate_expression` takes a new `usize` parameter (crate-private). - Default value of the config option is `20`, so behaviour is unchanged unless the option is set explicitly. ## Tests - `row_group_predicate_in_list_rewritten_at_raised_cap`: `PredicateRewriter::with_max_in_list_size(32)` rewrites a 25-item IN into per-value min/max checks instead of falling through to `true`. - `row_group_predicate_in_list_disabled_at_zero_cap`: cap = 0 skips the IN rewrite even for small lists (opt-out path). - Existing `row_group_predicate_in_list_to_many_values` continues to pass, guarding the default-20 behaviour.
github-actions
Bot
added
common
Contributor
There was a problem hiding this comment.
Pull request overview
This PR makes the IN (...)-list rewrite cap used by PruningPredicate configurable (instead of a hardcoded 20), and plumbs that setting through the parquet datasource so row-group / file-range stats pruning can remain effective for larger IN lists when users opt in.
Changes:
- Add
datafusion.execution.parquet.pruning_max_in_list_size(default20) to parquet execution config and thread it throughParquetSource→ opener/morselizer → row-group pruner. - Expose the historical default as
pub const MAX_LIST_VALUE_SIZE_REWRITEand add API variants/builders to pass an explicit cap (with_max_in_list_size,try_new_with_max_in_list_size,build_pruning_predicate_with_max_in_list_size). - Add unit tests covering raised-cap rewrite behavior and cap=0 opt-out behavior.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| datafusion/pruning/src/pruning_predicate.rs | Adds configurable IN-list rewrite cap to predicate rewriting/pruning APIs and tests it. |
| datafusion/pruning/src/lib.rs | Re-exports the new const and helper function as part of the public pruning API. |
| datafusion/datasource-parquet/src/source.rs | Reads the new config from TableParquetOptions.global and propagates it into pruning predicate construction. |
| datafusion/datasource-parquet/src/push_decoder.rs | Stores and applies the cap when (re)building pruning predicates in RowGroupPruner. |
| datafusion/datasource-parquet/src/opener/mod.rs | Threads the cap through ParquetMorselizer/PreparedParquetOpen and uses the new helper to build pruning predicates. |
| datafusion/common/src/file_options/parquet_writer.rs | Updates writer options destructuring to account for the newly added parquet option field. |
| datafusion/common/src/config.rs | Introduces the pruning_max_in_list_size parquet execution config option with documentation. |
Suppressed comments (3)
datafusion/pruning/src/pruning_predicate.rs:493
- Docs reference
datafusion.execution.pruning_max_in_list_size, but the actual config option isdatafusion.execution.parquet.pruning_max_in_list_size. Update this reference so callers can find the right setting.
/// Same as [`PruningPredicate::try_new`] but with an explicit cap on the
/// size of `IN (...)` lists rewritten into per-value statistics checks.
/// Query engines typically pass
/// `datafusion.execution.pruning_max_in_list_size` here.
datafusion/pruning/src/pruning_predicate.rs:1404
- This doc comment points to
datafusion.execution.pruning_max_in_list_size, but the new option is namespaced underparquet(datafusion.execution.parquet.pruning_max_in_list_size). Fixing the key avoids confusion for users trying to set the default cap explicitly.
/// Default maximum number of entries in an `IN (...)` list that will be
/// rewritten into a chain of per-value min/max checks by
/// [`build_predicate_expression`]. Callers threading a [`PredicateRewriter`]
/// can override this via [`PredicateRewriter::with_max_in_list_size`], and
/// query engines can wire it from the
/// `datafusion.execution.pruning_max_in_list_size` config option.
pub const MAX_LIST_VALUE_SIZE_REWRITE: usize = 20;
datafusion/pruning/src/pruning_predicate.rs:1445
- This builder method's docs reference
datafusion.execution.pruning_max_in_list_size, but the config key isdatafusion.execution.parquet.pruning_max_in_list_size. Update the docs to match the actual option name.
/// The default (see [`MAX_LIST_VALUE_SIZE_REWRITE`]) preserves the
/// historical behaviour. Callers wiring config through can override via
/// `datafusion.execution.pruning_max_in_list_size`.
pub fn with_max_in_list_size(mut self, max_in_list_size: usize) -> Self {
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+399
to
+402
| /// Same as [`build_pruning_predicate`] but with an explicit cap on the size | ||
| /// of `IN (...)` lists rewritten into per-value statistics checks. Query | ||
| /// engines typically pass `datafusion.execution.pruning_max_in_list_size` | ||
| /// here. |
Contributor
Author
There was a problem hiding this comment.
Fixed by renaming to datafusion.execution.parquet.max_in_list_size (per alamb below) — docs now match the actual config key.
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
github-actions
Bot
added
auto detected api change
The new ParquetOptions::pruning_max_in_list_size field was not carried through the proto layer, so every explicit ParquetOptions initializer in datafusion-proto-common and datafusion-proto failed to compile (E0063), which also cascaded into the MSRV job. - Add uint64 pruning_max_in_list_size = 38 to the ParquetOptions proto message and regenerate proto-common (prost.rs, pbjson.rs) and proto-models (datafusion_proto_common.rs). - Map the field in proto-common from_proto/to_proto and in the proto crate's file_formats TryFromProto/IntoProto for TableParquetOptions. - Reorder the datafusion_pruning import (cargo fmt) and drop the now unused build_pruning_predicate import.
zhuqi-lucas
force-pushed
the
qizhu/config-pruning-in-list-rewrite-size
branch
from
August 4, 2026 06:32
038b3ee to
5f4cfa1
Compare
- cargo doc: the public MAX_LIST_VALUE_SIZE_REWRITE doc linked the private build_predicate_expression via an intra-doc link; demote it to a code span and correct the config path to datafusion.execution.parquet.pruning_max_in_list_size. - Regenerate configs.md for the new pruning_max_in_list_size option. - Add the two pruning_max_in_list_size rows to information_schema.slt (SHOW ALL and the df_settings description listing).
github-actions
Bot
added
documentation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24074 +/- ##
========================================
Coverage 80.98% 80.98%
========================================
Files 1104 1104
Lines 378826 378968 +142
Branches 378826 378968 +142
========================================
+ Hits 306797 306912 +115
- Misses 53813 53832 +19
- Partials 18216 18224 +8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
alamb
reviewed
Aug 4, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
Looks good @zhuqi-lucas . I had some suggestions on API design and naming, but the overall idea makes a lot of sense
| enable_row_group_stats_pruning: false, | ||
| coerce_int96: None, | ||
| max_predicate_cache_size: None, | ||
| pruning_max_in_list_size: MAX_LIST_VALUE_SIZE_REWRITE, |
Contributor
There was a problem hiding this comment.
it is strange to me that these names are not the same -- I would expect something like
pruning_max_in_list_size: PRUNING_MAX_IN_LIST_SIZE,
Contributor
Author
There was a problem hiding this comment.
Fixed — renamed the field to max_in_list_size and the const to MAX_IN_LIST_SIZE so the shapes match.
| predicate, | ||
| file_schema, | ||
| predicate_creation_errors, | ||
| MAX_LIST_VALUE_SIZE_REWRITE, |
Contributor
There was a problem hiding this comment.
Why not just add the parameter to build_pruning_predicate ?
If we are going to introduce a new API, perhaps we can make one that is more future proof, like a builder
let pruning_predicate = PruningPredicaateBuilder::new()
.with_file_schema(file_schema)
.with_error_counter(predicate_creation_errors)
.build(predicate)?;That way if we add new parameters we have a place to put them
Contributor
Author
There was a problem hiding this comment.
Great suggestion — implemented as PruningPredicateBuilder with .with_file_schema(...), .with_error_counter(...), .with_max_in_list_size(...), and .build(predicate) returning Option<Arc<PruningPredicate>> for the parquet scan path, plus .try_build(predicate) returning Result<PruningPredicate> for callers that want to surface errors themselves. The standalone build_pruning_predicate_with_max_in_list_size and PruningPredicate::try_new_with_max_in_list_size are gone.
| /// before calling this method to make sure the expressions can be used for pruning. | ||
| pub fn try_new(mut expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) -> Result<Self> { | ||
| pub fn try_new(expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) -> Result<Self> { | ||
| Self::try_new_with_max_in_list_size(expr, schema, MAX_LIST_VALUE_SIZE_REWRITE) |
Contributor
There was a problem hiding this comment.
Same comment above related to simplifying this API via a builder rather than more methods
Contributor
Author
There was a problem hiding this comment.
Removed the try_new_with_max_in_list_size variant. PruningPredicate::try_new keeps its historical signature; the new PruningPredicateBuilder is the entry point for callers that want to override max_in_list_size (or supply an error counter).
| /// container. Set to 0 to disable the rewrite path entirely. | ||
| /// | ||
| /// The default of 20 preserves the previous hardcoded behaviour. | ||
| pub pruning_max_in_list_size: usize, default = 20 |
Contributor
There was a problem hiding this comment.
Also I suggest changing this to be something more conisstent with the others names like max_predicate_cache_size
Perhaps something likemax_in_list_size or max_in_list_pruning_size
Contributor
Author
There was a problem hiding this comment.
Renamed to max_in_list_size (matches the max_predicate_cache_size neighbour). Section is already execution.parquet.* so the pruning context is inferable from the key path.
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Contributor
Author
|
Thanks for the review and good suggestions @alamb, addressed comments now. |
- Introduce PruningPredicateBuilder per alamb's suggestion, replacing the ad-hoc try_new_with_max_in_list_size / build_pruning_predicate_with_max_in_list_size helpers with a single builder that also carries the error counter. - Rename config option pruning_max_in_list_size -> max_in_list_size to match the max_predicate_cache_size naming style (per alamb). - Rename const MAX_LIST_VALUE_SIZE_REWRITE -> MAX_IN_LIST_SIZE so the field name and the const name are consistent (per alamb's opener/mod.rs comment). - Trim the config-option doc (removed accidentally-duplicated 'of' word, tightened wording). - Regenerate docs/source/user-guide/configs.md and datafusion/sqllogictest/test_files/information_schema.slt. - Add PruningPredicateBuilder unit test verifying max_in_list_size is threaded end-to-end (default -> 'true'; raised cap -> real statistics predicate).
zhuqi-lucas
force-pushed
the
qizhu/config-pruning-in-list-rewrite-size
branch
from
August 5, 2026 12:07
92b7e69 to
ab65428
Compare
alamb
approved these changes
Aug 5, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
Look great to me -- thank you @zhuqi-lucas
| file_schema, | ||
| predicate_creation_errors, | ||
| ) | ||
| PruningPredicateBuilder::new() |
| .build(predicate) | ||
| } | ||
|
|
||
| /// Builder for a [`PruningPredicate`]. Groups optional configuration — |
| /// It is recommended that you pass the expressions through [`PhysicalExprSimplifier`] | ||
| /// before calling this method to make sure the expressions can be used for pruning. | ||
| pub fn try_new(mut expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) -> Result<Self> { | ||
| pub fn try_new(expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) -> Result<Self> { |
Contributor
There was a problem hiding this comment.
Maybe as a follow on PR we want to direct people to PruningPredicateBuilder 🤔 in comments / deprecate the try_new and move this construction into the builder
alamb
changed the title
Contributor
|
I update an output file in 8ad8ad8 and merged up from main |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale
PruningPredicaterewritescol IN (v1..vn)into a chain of per-value min/max checks (viabuild_predicate_expression), but only whenn <= MAX_LIST_VALUE_SIZE_REWRITE— currently a hardcoded20. Beyond that, the IN branch falls through tounhandled_hook, which by default returnsTRUE, so row-group and file-range statistics pruning does not fire at all for IN lists longer than 20.This is problematic for query patterns that pass a batch of identifiers as
col IN (...)— REST endpoints filtering by a page of ~25-100 values, ORM-generatedWHERE id IN (25 items)queries, batched crawlers. On a table sorted bycol, the reader is forced to materialize the filter column across every row group instead of skipping row groups whose stats disagree with the IN set.Full context in #24059.
What changes are included in this PR?
datafusion.execution.parquet.pruning_max_in_list_size: usize(default20, preserving existing behaviour), placed next tomax_predicate_cache_sizeonTableParquetOptions.global.MAX_LIST_VALUE_SIZE_REWRITEpromoted topub constso callers can reference the historical default explicitly.PredicateRewriter::with_max_in_list_size(usize) -> Selfbuilder, mirroring the existingwith_unhandled_hook.PruningPredicate::try_new_with_max_in_list_sizevariant.build_pruning_predicate_with_max_in_list_sizevariant of the public helper.datasource-parquet:ParquetSource::pruning_max_in_list_size()reads fromTableParquetOptions.global, propagates throughParquetMorselizer→PreparedParquetOpen→RowGroupPruner, then flows intobuild_pruning_predicatesat the opener andbuild_pruning_predicate_with_max_in_list_sizeinside the dynamic row-group pruner.Internal
build_predicate_expressiongains a newusizeparameter (crate-private).Backward compatibility
PruningPredicate::try_newandbuild_pruning_predicateare preserved as thin wrappers that pass the historicalMAX_LIST_VALUE_SIZE_REWRITEdefault. All existing callers continue to work with unchanged behaviour.20, so behaviour is unchanged unless the option is set explicitly.Are these changes tested?
Two new unit tests in
datafusion-pruning:row_group_predicate_in_list_rewritten_at_raised_cap:PredicateRewriter::with_max_in_list_size(32)rewrites a 25-item IN into per-value min/max checks OR'd together, instead of falling through totrue.row_group_predicate_in_list_disabled_at_zero_cap:cap = 0skips the IN rewrite even for small lists (opt-out path).The existing
row_group_predicate_in_list_to_many_valuescontinues to pass, guarding the default-20 behaviour.Are there any user-facing changes?
Yes — one new config option (
datafusion.execution.parquet.pruning_max_in_list_size, default20). Users who want row-group / file-range pruning for IN lists longer than 20 items can raise it (e.g.,SET datafusion.execution.parquet.pruning_max_in_list_size = 128).New public API on
datafusion-pruning:MAX_LIST_VALUE_SIZE_REWRITE: usize(re-exported)PredicateRewriter::with_max_in_list_size(usize) -> SelfPruningPredicate::try_new_with_max_in_list_size(expr, schema, size)build_pruning_predicate_with_max_in_list_size(predicate, schema, errors, size)