You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#24090 teaches the Parquet projection path to prune leaves when a nested column is consumed through a narrowing cast — the shape DefaultPhysicalExprAdapter produces whenever a table's declared nested schema is narrower than the physical file (Comet, delta-rs, Iceberg integrations all hit this). PushdownChecker collects a CastColumnAccess for CAST(col AS narrower_nested_type), and clip_for_cast walks the physical and target type trees together to compute exactly which Parquet leaves the cast consumes.
That collection is deliberately opt-in and enabled only for projection analysis:
/// Enable collection of whole-column casts to narrower nested types.pub(crate)fnwith_cast_collection(mutself) -> Self{ ...}
Filter pushdown leaves it off, so a predicate over a narrowed nested column still reads every physical leaf of that column.
What this issue asks for
Extend the same clipping to the row-filter path (datafusion/datasource-parquet/src/row_filter.rs), so that with pushdown_filters = true a predicate like WHERE s['x'] = 200 against a table whose declared schema narrows s reads only the leaves the predicate needs, rather than the whole column.
The projection side already computes this; the filter side needs the equivalent two outputs kept in agreement:
the ProjectionMask handed to the reader for the filter, and
the projected Arrow schema the row filter evaluates against.
This cannot land as a pure optimization until #24109 is resolved. Today PushdownChecker::f_down only recognises get_field whose first argument is a bare Column. When the expression adapter interposes a cast — which is always the case on exactly the tables this feature targets — the visitor descends past the CastExpr, sees a bare struct Column, sets non_primitive_columns = true, and pushdown_columns returns None. The conjunct is then silently dropped from the row filter while planning has already removed FilterExec on the strength of an "exactly handled" claim made against the table schema.
So filter pushdown over a narrowed nested column is currently unsound, not merely unoptimised. Teaching PushdownChecker to look through the cast would address both #24109 and this issue, but only if the post-decode fallback proposed in #24109 exists to catch whatever the row-filter construction still cannot handle.
The clip is total by design: any shape it does not understand keeps all leaves, so the worst case is today's full read. That property should be preserved on the filter side.
Part of #24119.
Background
#24090 teaches the Parquet projection path to prune leaves when a nested column is consumed through a narrowing cast — the shape
DefaultPhysicalExprAdapterproduces whenever a table's declared nested schema is narrower than the physical file (Comet, delta-rs, Iceberg integrations all hit this).PushdownCheckercollects aCastColumnAccessforCAST(col AS narrower_nested_type), andclip_for_castwalks the physical and target type trees together to compute exactly which Parquet leaves the cast consumes.That collection is deliberately opt-in and enabled only for projection analysis:
Filter pushdown leaves it off, so a predicate over a narrowed nested column still reads every physical leaf of that column.
What this issue asks for
Extend the same clipping to the row-filter path (
datafusion/datasource-parquet/src/row_filter.rs), so that withpushdown_filters = truea predicate likeWHERE s['x'] = 200against a table whose declared schema narrowssreads only the leaves the predicate needs, rather than the whole column.The projection side already computes this; the filter side needs the equivalent two outputs kept in agreement:
ProjectionMaskhanded to the reader for the filter, andPrerequisite: #24109
This cannot land as a pure optimization until #24109 is resolved. Today
PushdownChecker::f_downonly recognisesget_fieldwhose first argument is a bareColumn. When the expression adapter interposes a cast — which is always the case on exactly the tables this feature targets — the visitor descends past theCastExpr, sees a bare structColumn, setsnon_primitive_columns = true, andpushdown_columnsreturnsNone. The conjunct is then silently dropped from the row filter while planning has already removedFilterExecon the strength of an "exactly handled" claim made against the table schema.So filter pushdown over a narrowed nested column is currently unsound, not merely unoptimised. Teaching
PushdownCheckerto look through the cast would address both #24109 and this issue, but only if the post-decode fallback proposed in #24109 exists to catch whatever the row-filter construction still cannot handle.Suggested order
get_fieldpredicate when the file needs schema adaptation (wrong results) #24109 (post-decode filter fallback, so a failed row-filter build costs performance rather than correctness).row_filter.rs— see Refactor: Build a reusable struct-access path tree for Parquet row-filter planning #23156, which proposes a shared access tree for exactly the "mask and projected schema must describe the same pruned shape" invariant.Notes
ReadSchema).