Fix size in bytes calculation for ORC DictionaryBuilder#25615
Closed
mcarmonagonzalez wants to merge 6 commits into
Closed
Fix size in bytes calculation for ORC DictionaryBuilder#25615mcarmonagonzalez wants to merge 6 commits into
mcarmonagonzalez wants to merge 6 commits into
Conversation
Upcoming Iceberg 1.9.0 makes row lineage required. This reverts commit 2645d51.
Helps avoid commit conflicts between OPTIMIZE and adding equality deletes
mcarmonagonzalez
force-pushed
the
orc-dictionary-size-fix
branch
from
8791106 to
61ac830
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
a1fcb0d introduced a change to refactor
VariableWidthBlockBuilderout of the ORC DictionaryBuilder class. Previously the DictionaryBuilder class would report the size in bytes by calling on the underlyingelementBlock.getSizeInBytes()whose size was calculated based off the output slice size and the bytes stored per entry/position.After this refactor, DictionaryBuilder size is now being reported as the sum of the slice output size and the size of the offsets array, which is causing the size to be over-reported as not all positions of the offset array may be used. Over reporting the size in bytes of the DictionaryBuilder causes issues downstream when doing file size optimizations.
This PR corrects the size in bytes calculation to be similar to what it was before the refactor to accurately report how many bytes are actually in use.
Additional context and related issues
Release notes
( x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: