Skip to content

add compression to property files#42

Draft
sfluegel05 wants to merge 2 commits into
devfrom
feature/property-compression
Draft

add compression to property files#42
sfluegel05 wants to merge 2 commits into
devfrom
feature/property-compression

Conversation

@sfluegel05

Copy link
Copy Markdown
Contributor

Property files take up an absurd amount of disk space.

For instance, for a ChEBI50 dataset (v244), the graph_properties\properties folder has nearly 10GB. AtomType_one_hot.pt alone has 6.4GB.

This PR compresses data before saving it to a file (and decompresses when loading). Especially the one-hot encodings don't need to be saved as one-hot tensors. So they are now stored as 1-d and re-converted to one-hot when loading.

File sizes are reduced by a factor of up to 109:

DONE  AtomAromaticity_bool.pt: 106.9 MB -> 59.6 MB (1.8x)
DONE  AtomCharge_one_hot.pt: 743.5 MB -> 59.0 MB (12.6x)
DONE  AtomHybridization_one_hot.pt: 427.6 MB -> 61.3 MB (7.0x)
DONE  AtomNumHs_one_hot.pt: 424.4 MB -> 58.7 MB (7.2x)
DONE  AtomType_one_hot.pt: 6376.0 MB -> 58.5 MB (109.0x)
DONE  BondAromaticity_bool.pt: 109.5 MB -> 59.8 MB (1.8x)
DONE  BondInRing_bool.pt: 107.1 MB -> 58.2 MB (1.8x)
DONE  BondType_one_hot.pt: 331.8 MB -> 58.5 MB (5.7x)
DONE  NumAtomBonds_one_hot.pt: 639.1 MB -> 59.6 MB (10.7x)
DONE  RDKit2DNormalized_asis.pt: 360.7 MB -> 203.5 MB (1.8x)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant