Skip to content

Extend Parquet nested schema pruning to filter pushdown #24120

Description

@adriangb

Part of #24119.

Background

#24090 teaches the Parquet projection path to prune leaves when a nested column is consumed through a narrowing cast — the shape DefaultPhysicalExprAdapter produces whenever a table's declared nested schema is narrower than the physical file (Comet, delta-rs, Iceberg integrations all hit this). PushdownChecker collects a CastColumnAccess for CAST(col AS narrower_nested_type), and clip_for_cast walks the physical and target type trees together to compute exactly which Parquet leaves the cast consumes.

That collection is deliberately opt-in and enabled only for projection analysis:

/// Enable collection of whole-column casts to narrower nested types.
pub(crate) fn with_cast_collection(mut self) -> Self { ... }

Filter pushdown leaves it off, so a predicate over a narrowed nested column still reads every physical leaf of that column.

What this issue asks for

Extend the same clipping to the row-filter path (datafusion/datasource-parquet/src/row_filter.rs), so that with pushdown_filters = true a predicate like WHERE s['x'] = 200 against a table whose declared schema narrows s reads only the leaves the predicate needs, rather than the whole column.

The projection side already computes this; the filter side needs the equivalent two outputs kept in agreement:

  1. the ProjectionMask handed to the reader for the filter, and
  2. the projected Arrow schema the row filter evaluates against.

Prerequisite: #24109

This cannot land as a pure optimization until #24109 is resolved. Today PushdownChecker::f_down only recognises get_field whose first argument is a bare Column. When the expression adapter interposes a cast — which is always the case on exactly the tables this feature targets — the visitor descends past the CastExpr, sees a bare struct Column, sets non_primitive_columns = true, and pushdown_columns returns None. The conjunct is then silently dropped from the row filter while planning has already removed FilterExec on the strength of an "exactly handled" claim made against the table schema.

So filter pushdown over a narrowed nested column is currently unsound, not merely unoptimised. Teaching PushdownChecker to look through the cast would address both #24109 and this issue, but only if the post-decode fallback proposed in #24109 exists to catch whatever the row-filter construction still cannot handle.

Suggested order

  1. Fix Parquet filter pushdown silently drops a get_field predicate when the file needs schema adaptation (wrong results) #24109 (post-decode filter fallback, so a failed row-filter build costs performance rather than correctness).
  2. Enable cast collection in the filter path and route clipped leaves through row-filter planning.
  3. Reconcile with the struct-access path handling in row_filter.rs — see Refactor: Build a reusable struct-access path tree for Parquet row-filter planning #23156, which proposes a shared access tree for exactly the "mask and projected schema must describe the same pruned shape" invariant.

Notes

Metadata

Metadata

Assignees

No one assigned

    Labels

    datasourceChanges to the datasource crateenhancementNew feature or requestperformanceMake DataFusion faster

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions