| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
There was a problem hiding this comment.
I generally like the approach @yadavay-amzn, here's why I would not use cache alone as I mentioned in the issue. It's better than the current approach of checking it RGxCC times
The cleaner fix would be at the caller side in ParquetMetadataConverter.fromParquetMetadata(): compute shouldIgnoreStatistics once before the row group loop using the file-level createdBy, then pass the pre-computed boolean through buildColumnChunkMetaData → fromParquetStatisticsInternal. This eliminates all per-column overhead — no hash, no lookup, just a boolean flowing through the call chain.
This eliminates the need to check shouldIgnoreStatistics for every RGxCC. Both approaches could coexist (cache helps other callers), but the caller-side fix is where we see real improvements
Sorry, something went wrong.
|
@asifsmohammed Thanks for reviewing! I am an agreement with your suggestions. Removed the global static cache entirely. Now computes shouldIgnoreStatistics once per file in fromParquetMetadata before the row-group loop and passes the pre-computed flag through buildColumnChunkMetaData to fromParquetStatisticsInternal. cc @wgtmac |
Sorry, something went wrong.
There was a problem hiding this comment.
Optimizes footer metadata conversion by avoiding repeated CorruptStatistics.shouldIgnoreStatistics(created_by, ...) evaluation during row-group/column iteration, by precomputing a boolean once and threading it through column-chunk metadata/statistics conversion.
Changes:
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Sorry, something went wrong.
| boolean shouldIgnoreCorruptStats = | ||
| CorruptStatistics.shouldIgnoreStatistics(createdBy, PrimitiveTypeName.BINARY); | ||
| return buildColumnChunkMetaData(metaData, columnPath, type, createdBy, shouldIgnoreCorruptStats); |
| // Compute once per file: the result is the same for BINARY and FIXED_LEN_BYTE_ARRAY | ||
| // (the only types affected by PARQUET-251), and always false for other types. | ||
| boolean shouldIgnoreCorruptStats = | ||
| CorruptStatistics.shouldIgnoreStatistics(parquetMetadata.getCreated_by(), PrimitiveTypeName.BINARY); |
There was a problem hiding this comment.
This is calling shouldIgnoreStatistics with PrimitiveTypeName.BINARY hardcoded which is incorrect.
Instead we can refactor shouldIgnoreStatistics by adding public methods.
public static boolean shouldIgnoreStatistics(String createdBy, PrimitiveTypeName columnType) {
if (!isCorruptStatisticsColumnType(columnType)) {
// the bug only applies to binary columns
return false;
}
return fileHasCorruptStatistics(createdBy);
}
public static boolean isCorruptStatisticsColumnType(PrimitiveTypeName columnType) {
return columnType == PrimitiveTypeName.BINARY || columnType == PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY;
}
public static boolean fileHasCorruptStatistics(String createdBy) {
// rest of the logic from shouldIgnoreStatistics
}
Sorry, something went wrong.
There was a problem hiding this comment.
Done. Added isCorruptStatisticsColumnType(columnType) and fileHasCorruptStatistics(createdBy), with shouldIgnoreStatistics delegating to both. The file-level flag is computed via fileHasCorruptStatistics (no hardcoded BINARY), and the per-column BINARY/FIXED_LEN_BYTE_ARRAY check is applied where stats are converted.
Sorry, something went wrong.
| // Compute once per file: the result is the same for BINARY and FIXED_LEN_BYTE_ARRAY | ||
| // (the only types affected by PARQUET-251), and always false for other types. |
| } | ||
|
|
||
| // Overload that uses a pre-computed shouldIgnoreCorruptStats flag to avoid redundant parsing | ||
| private org.apache.parquet.column.statistics.Statistics fromParquetStatisticsInternal( |
There was a problem hiding this comment.
Instead of duplicating the entire fromParquetStatisticsInternal body, the existing method can simply delegate to a new overload and this eliminates duplicate code
static org.apache.parquet.column.statistics.Statistics fromParquetStatisticsInternal(
String createdBy, Statistics formatStats, PrimitiveType type, SortOrder typeSortOrder) {
return fromParquetStatisticsInternal(
formatStats,
type,
typeSortOrder,
CorruptStatistics.fileHasCorruptStatistics(createdBy) // This is a new method in CorruptStatistics
);
}
// overloaded method
static org.apache.parquet.column.statistics.Statistics fromParquetStatisticsInternal(
Statistics formatStats, PrimitiveType type, SortOrder typeSortOrder, boolean fileHasCorruptStats) {
Sorry, something went wrong.
There was a problem hiding this comment.
Done. Removed the duplicated body; the createdBy overload now delegates to the boolean overload via CorruptStatistics.fileHasCorruptStatistics(createdBy).
Sorry, something went wrong.
| // Compute once per file: the result is the same for BINARY and FIXED_LEN_BYTE_ARRAY | ||
| // (the only types affected by PARQUET-251), and always false for other types. | ||
| boolean shouldIgnoreCorruptStats = | ||
| CorruptStatistics.shouldIgnoreStatistics(parquetMetadata.getCreated_by(), PrimitiveTypeName.BINARY); |
There was a problem hiding this comment.
This is calling shouldIgnoreStatistics with PrimitiveTypeName.BINARY hardcoded which is incorrect.
Instead we can refactor shouldIgnoreStatistics by adding public methods.
public static boolean shouldIgnoreStatistics(String createdBy, PrimitiveTypeName columnType) {
if (!isCorruptStatisticsColumnType(columnType)) {
// the bug only applies to binary columns
return false;
}
return fileHasCorruptStatistics(createdBy);
}
public static boolean isCorruptStatisticsColumnType(PrimitiveTypeName columnType) {
return columnType == PrimitiveTypeName.BINARY || columnType == PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY;
}
public static boolean fileHasCorruptStatistics(String createdBy) {
// rest of the logic from shouldIgnoreStatistics
}
Sorry, something went wrong.
| } | ||
| } | ||
|
|
||
| String createdBy = parquetMetadata.getCreated_by(); |
There was a problem hiding this comment.
We no longer need to pass createdBy downstream
Sorry, something went wrong.
There was a problem hiding this comment.
Done. The precomputed boolean is threaded instead; createdBy is no longer passed downstream.
Sorry, something went wrong.
| return buildColumnChunkMetaData(metaData, columnPath, type, createdBy, shouldIgnoreCorruptStats); | ||
| } | ||
|
|
||
| ColumnChunkMetaData buildColumnChunkMetaData( |
There was a problem hiding this comment.
buildColumnChunkMetaData can delegate to a package-private overload that takes the boolean similar to what you have done but with few changes,
public ColumnChunkMetaData buildColumnChunkMetaData(
ColumnMetaData metaData, ColumnPath columnPath, PrimitiveType type, String createdBy) {
return buildColumnChunkMetaData(
metaData, columnPath, type, CorruptStatistics.fileHasCorruptStatistics(createdBy));
}
ColumnChunkMetaData buildColumnChunkMetaData(
ColumnMetaData metaData, ColumnPath columnPath, PrimitiveType type, boolean fileHasCorruptStats) {
SortOrder expectedOrder = overrideSortOrderToSigned(type) ? SortOrder.SIGNED : sortOrder(type);
return ColumnChunkMetaData.get(...,
fromParquetStatisticsInternal(metaData.statistics, type, expectedOrder, fileHasCorruptStats), ...);
}
No need to pass createdBy downstream, the boolean is all the internal overload needs. SortOrder computation moves here since we bypass fromParquetStatistics to avoid re-parsing createdBy as you have already done by replacing fromParquetStatisticsInternal with fromParquetStatistics.
Also notice how the new public methods we extracted in CorruptStatistics are being used in each delegate method here
Sorry, something went wrong.
There was a problem hiding this comment.
Done. buildColumnChunkMetaData now delegates to a package-private overload taking the boolean, with the SortOrder computation moved into it; the public (..., String createdBy) overload is preserved for back-compat.
Sorry, something went wrong.
| boolean ignoreForThisColumn = shouldIgnoreCorruptStats | ||
| && (primitiveTypeName == PrimitiveTypeName.BINARY | ||
| || primitiveTypeName == PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY); | ||
| if (!ignoreForThisColumn && (sortOrdersMatch || maxEqualsMin)) { |
There was a problem hiding this comment.
This check in shouldIgnoreStatistics is dead code with current changes as we always pass BINARY
if (columnType != PrimitiveTypeName.BINARY && columnType !=PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY)
We can utilize the new methods here. Please refer to below comments for context.
if (!(fileHasCorruptStats && CorruptStatistics.isCorruptStatisticsColumnType(type.getPrimitiveTypeName()))
Instead of calling or moving the PrimitiveTypeName checks to ParquetMetadataConverter and leave it as the responsibility of CorruptStatistics
if (!CorruptStatistics.shouldIgnoreStatistics(createdBy, type.getPrimitiveTypeName())
Sorry, something went wrong.
There was a problem hiding this comment.
Done. Removed the now-dead column-type check; the gate is expressed via CorruptStatistics.isCorruptStatisticsColumnType in fromParquetStatisticsInternal.
Sorry, something went wrong.
|
@asif-moh @wgtmac Addressed the feedback: factored CorruptStatistics into fileHasCorruptStatistics + isCorruptStatisticsColumnType (with shouldIgnoreStatistics delegating), precompute the file-level flag once in fromParquetMetadata (schema-gated so binary-free files do no created_by parsing), thread a boolean through buildColumnChunkMetaData/fromParquetStatisticsInternal, drop the duplicated body and dead code, and fix the per-column gate so non-binary columns are not ignored. Updated the title/description to match. PTAL. |
Sorry, something went wrong.
There was a problem hiding this comment.
LGTM! Thanks for making the changes
Sorry, something went wrong.
| // the bug only applies to binary columns | ||
| return false; | ||
| } | ||
| public static boolean isCorruptStatisticsColumnType(PrimitiveTypeName columnType) { |
There was a problem hiding this comment.
IMHO, we need to be cautious when adding a new public method because we have to maintain it forever. Can we remove it from here and just let downstream where calls it to use a private method instead?
Sorry, something went wrong.
There was a problem hiding this comment.
We need fileHasCorruptStatistics (or mayHaveCorruptStatistics after rename) to remain public, otherwise we'd have to duplicate the version-parsing logic in ParquetMetadataConverter. We can make isCorruptStatisticsColumnType private and inline the column-type check downstream.
Or we can overload shouldIgnoreStatistics(String createdBy) but either way we will have 1 new public method.
@yadavay-amzn @wgtmac wdyt?
Sorry, something went wrong.
There was a problem hiding this comment.
Went with your suggestion - kept mayHaveCorruptStatistics public (ParquetMetadataConverter needs it) and made isCorruptStatisticsColumnType private, inlining the column-type check (BINARY || FIXED_LEN_BYTE_ARRAY) at its call sites. Net new public API is just mayHaveCorruptStatistics(String). Behavior unchanged; parquet-column + parquet-hadoop tests pass.
Sorry, something went wrong.
There was a problem hiding this comment.
Minimized the new public surface: isCorruptStatisticsColumnType is now private with the column-type check inlined at its call sites, so mayHaveCorruptStatistics(String) is the only new public method. It has to stay public because ParquetMetadataConverter relies on it (otherwise the version-parsing logic would be duplicated).
Sorry, something went wrong.
| * @param createdBy the created-by string from a file footer | ||
| * @return true if the file was written by a version with the corrupt statistics bug | ||
| */ | ||
| public static boolean fileHasCorruptStatistics(String createdBy) { |
There was a problem hiding this comment.
Rename it to mayHaveCorruptStatistics ?
Sorry, something went wrong.
There was a problem hiding this comment.
Renamed to mayHaveCorruptStatistics.
Sorry, something went wrong.
|
@wgtmac addressed the API-minimization and rename: fileHasCorruptStatistics is renamed to mayHaveCorruptStatistics and kept public (ParquetMetadataConverter relies on it, otherwise the version-parsing logic would be duplicated). isCorruptStatisticsColumnType is now private with the column-type check inlined at its call sites, so the only new public method this PR adds is mayHaveCorruptStatistics(String). Behavior is unchanged; tests pass. PTAL. |
Sorry, something went wrong.
|
Could you resolve the conflict? |
Sorry, something went wrong.
…tMetadataConverter
…vel + column-type checks; precompute once per file (schema-gated)
|
@wgtmac rebased onto latest master and resolved the conflict (it was only in TestParquetMetadataConverter - kept master's new NaN/IEEE754 tests plus the two GH-3601 tests). The GH-3601 change is intact; parquet-column + parquet-hadoop tests pass and spotless is clean. |
Sorry, something went wrong.
|
@wgtmac, @yadavay-amzn has addressed your feedback and rebased onto master. Is there anything else |
Sorry, something went wrong.
There was a problem hiding this comment.
Thanks for the updates! I took another look and I still do not think this is the right shape yet.
I have to admit that my earlier suggestion may have pushed this in the wrong direction. I suggested factoring out a file-level corrupt-stats check, but I do not think we should merge a partial solution with side effect. My concerns with the current implementation are:
CorruptStatistics.mayHaveCorruptStatistics adds a new public API for an internal optimization. It is tied to PARQUET-251 semantics, so we would need to maintain that specific meaning as public API.
mayHaveCorruptStatistics is not a pure predicate. It can log warnings and mutate alreadyLogged. Precomputing it can therefore have user-visible side effects.
The fix is still partial. The main footer path avoids the row-group x column parse, but other production paths still parse created_by on each call. Examples include fromParquetStatisticsInternal(String createdBy, ...), buildColumnChunkMetaData(..., String createdBy), page-level stats in ParquetFileReader, and lazy encrypted metadata in ColumnChunkMetaData.
The current fileHasCorruptStats boolean leaks too much PARQUET-251-specific logic into ParquetMetadataConverter. The converter now knows about the schema gate, BINARY / FIXED_LEN_BYTE_ARRAY, and the file-level corrupt-stats flag. This pattern will not scale well if we later need another writer-version-based statistics workaround.
The schema-gate test does not prove the schema gate. testSchemaGateSkipsCorruptStatsCheckForNonBinarySchema only proves INT stats are preserved. That would still pass because of the later per-column gate. It does not prove that parsing or the corrupt-stats check was skipped.
I have not had enough time to fully design the final API, so please treat the following as a possible direction, not a complete design. A cleaner approach may be to cache the parsed created_by result, not a PARQUET-251-specific boolean.
Java already has VersionParser.ParsedVersion in parquet-common. parquet-hadoop already depends on parquet-common. There is also precedent in CorruptDeltaByteArrays.requiresSequentialReads(ParsedVersion, Encoding).
The converter could parse created_by once per file into a ParsedVersion, pass that ParsedVersion through the converter, and keep the PARQUET-251 decision inside CorruptStatistics.
For example:
fromParquetStatisticsInternal(writerVersion, metaData.statistics, type, typeSortOrder)And CorruptStatistics could have:
shouldIgnoreStatistics(ParsedVersion writerVersion, PrimitiveTypeName columnType)The existing String-based API can stay as a compatibility wrapper. This keeps the optimization in ParquetMetadataConverter, keeps the version-specific logic in CorruptStatistics, and avoids exposing mayHaveCorruptStatistics or threading a PARQUET-251 boolean through generic metadata conversion code.
Sorry, something went wrong.
|
@wgtmac Thanks for the review. I agree with the overall direction toward ParsedVersion. Let me share findings from my investigation here,
Agreed, exposing a PARQUET-251-specific predicate as public API is the wrong abstraction. It ties us to maintaining that exact semantic contract. The ParsedVersion-based approach keeps this internal.
I looked at this more closely and I believe the logging behavior is actually unchanged between master and this PR. The schema gate (schemaHasCorruptStatisticsColumnType) ensures mayHaveCorruptStatistics is only called when the schema has BINARY/FIXED_LEN_BYTE_ARRAY columns, the same precondition that would trigger the warning on master. In all 4 cases (corrupt/non-corrupt × has-binary/no-binary), the warning fires or doesn't fire in identical scenarios. The only difference is microsecond-level timing within the same fromParquetMetadata call. That said, I agree the ParsedVersion approach is cleaner here regardless parsing is side-effect-free, and the warning stays inside shouldIgnoreStatistics where it's always been.
I investigated this further and the problem is broader than I initially realized. VersionParser.parse(createdBy) is called from 6 distinct production sites, all parsing the same constant string from FileMetaData.getCreatedBy():
Notably, ColumnReadStoreImpl already parses createdBy into a ParsedVersion and stores it as a field but it does so R times (once per row group) because nobody upstream caches the parsed result. The CorruptDeltaByteArrays.requiresSequentialReads(ParsedVersion, Encoding) overload already exists and is used from ColumnReaderBase this is exactly the pattern you suggested for CorruptStatistics.
Agreed. Threading a boolean specific to one bug doesn't scale. The ParsedVersion approach keeps the converter generic and it just says "here's the writer version" and lets CorruptStatistics decide. Proposed approach: I'd like to propose addressing the root cause separately: cache a ParsedVersion in FileMetaData. Since FileMetaData is constructed once per file and already stores the createdBy string, it's the natural place to parse once and cache: // In FileMetaData:
private final transient ParsedVersion writerVersion;
public FileMetaData(
.... {
this.createdBy = createdBy;
this.writerVersion = parseVersion(createdBy); // ← just add this
...
}
public ParsedVersion getWriterVersion() {
return writerVersion;
}
private static ParsedVersion parseVersion(String createdBy) {
if (Strings.isNullOrEmpty(createdBy)) return null;
try {
return VersionParser.parse(createdBy);
} catch (RuntimeException | VersionParseException e) {
return null;
}
This is purely additive (new field + getter), doesn't break serialization (transient), and enables all 7 call sites to stop re-parsing. ColumnReadStoreImpl could accept ParsedVersion directly instead of re-parsing in its constructor, and CorruptStatistics would gain a shouldIgnoreStatistics(ParsedVersion, PrimitiveTypeName) overload following the established CorruptDeltaByteArrays.requiresSequentialReads(ParsedVersion, Encoding) pattern. Would you be open to a separate issue/PR for adding ParsedVersion caching to FileMetaData? That would be a small, zero-behavior-change change just parsing createdBy in the existing constructor body and exposing a getter. The ParsedVersion-based overloads for CorruptStatistics, ColumnReadStoreImpl, etc. can be added incrementally after that, and this PR could then rebase onto the foundation cleanly. |
Sorry, something went wrong.
| PrimitiveType binaryType = new PrimitiveType(Repetition.OPTIONAL, PrimitiveTypeName.BINARY, "bin_col"); | ||
| Statistics binaryStats = ParquetMetadataConverter.fromParquetStatisticsInternal( | ||
| corruptCreatedBy, formatStats, binaryType, ParquetMetadataConverter.SortOrder.SIGNED); | ||
| assertFalse("BINARY min/max should be ignored for corrupt file", binaryStats.hasNonNullValue()); |
There was a problem hiding this comment.
you might need to rebase the PR because we switched all assertions to AssertJ and removed all JUnit4 usages.
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Summary
Footer conversion previously evaluated CorruptStatistics.shouldIgnoreStatistics(createdBy, columnType) per (row-group x column), re-parsing the created_by version string on every call.
This PR factors CorruptStatistics into two orthogonal checks:
The file-level corrupt-stats flag is precomputed once in fromParquetMetadata, gated on the schema containing at least one affected column type (BINARY or FIXED_LEN_BYTE_ARRAY), so binary-free files skip the created_by parsing entirely and avoid spurious PARQUET-251 warnings. The boolean is threaded through buildColumnChunkMetaData/fromParquetStatisticsInternal instead of re-parsing created_by per column.
Note: an earlier process-wide cache approach was dropped per review feedback.
Tests
Closes #3601