Repository navigation
Conversation
Switch the default Parquet data page version from V1 to V2 across C++ and Python (PyArrow). V2 data pages enable per-page compression selectivity (via min_compression_savings) and include explicit page-level null count metadata (num_nulls) in the page header. Closes apache#55272.
Awesome06
requested review from
AlenkaF,
HuaHuaY,
pitrou,
raulcd,
rok and
wgtmac
as code owners
November 9, 2026 12:38
|
|
… masking and synchronize V2 page defaults
Reorder MaybeReplaceValidity before update_stats in WriteArrowDictionary so writeable_indices reflects parent nulls prior to computing page stats, and guard dictionary reuse with chunk_indices->null_count() == 0.
Update TestSerialize for BooleanType in Parquet C++ to account for V2 selective compression keeping incompressible 7-byte boolean pages uncompressed.
Update test_parquet_metadata_api expected boolean encodings to {'RLE'} for V2 defaults.
Synchronize parquet.rst Sphinx doctest expected row group byte size for V2 headers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Parquet V2 data pages are widely supported across the ecosystem. They provide explicit page-level null counts (
num_nulls) in page headers and enable per-page selective compression (min_compression_savings), allowing pages that do not benefit from compression to remain uncompressed.Benchmarks across TPC-H, SSB, and TPC-DS (discussed on GH-55272) showed that V2 data pages yield comparable file sizes (0.0% to 1.2% delta depending on schema sparsity) while improving read throughput by 12% to 35% on scanned columns.
What changes are included in this PR?
properties.h):data_page_versioninWriterProperties::BuilderfromParquetDataPageVersion::V1toParquetDataPageVersion::V2.V2default.properties_test.cc,column_writer_test.cc):TestWriterProperties.Basicsto expectParquetDataPageVersion::V2.MinCompressionSavingsRequiresDataPageV2to verify defaultV2allowsmin_compression_savingswithout throwing, while explicitly configuredV1raises an exception.WriteLargeDictEncodedPage, updated expected page type toPageType::DATA_PAGE_V2.parquet.rst):data_page_versiondefaults to V2, with V1 retained for older reader compatibility.core.py,_dataset_parquet.pyx,_parquet.pyx):data_page_versionfrom'1.0'to'2.0'inParquetWriter,write_table, and dataset write options._parquet.pyxvalidation somin_compression_savings > 0.01only rejects explicitdata_page_version="1.0".test_parquet_writer.py):test_min_compression_savings_rejects_v1to explicitly specifydata_page_version="1.0".test_min_compression_savings_default_v2to verify default write paths acceptmin_compression_savingswithout error.Are these changes tested?
Yes. Updated existing C++ properties and writer unit tests, and added Python test coverage for default V2 writer behavior.
Are there any user-facing changes?
Yes. Files written with default writer properties in C++ and PyArrow will now use Parquet Data Page V2 instead of V1. Users requiring V1 for legacy compatibility can pass
data_page_version="1.0"(Python) or.data_page_version(ParquetDataPageVersion::V1)(C++).Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: