Description
Currently, the bins are first concatenated in preparation of export, then are processed using Huffmann compression (a bitstream compression) before finally being saved. However, this is a horrible idea, because the bins are already encrypted using AES-128 beforehand, and AES increases entropy, making the bit distribution appear roughly equal. This is the worst scenario for Huffmann, as data with high entropy will not compress properly, and the overhead of the symbol table might even make the final size larger. Compression was needed for the assignment but is not required afterwards, so that step can be removed. However, I will still retain the compression code in case I want to compress individual files or entries in the bins to optimise storage.
Tasks
- Remove compression before bin export
- Update export format to remove compression-specific fields
Maybe I can also do a version update for the file format? I would have done that and added a utility to automatically update the format to the latest one if there were users of this application, but as this is more of a technical demo, I will omit these features, opting simply for updating the version of the file, making previous states invalid.
Description
Currently, the bins are first concatenated in preparation of export, then are processed using Huffmann compression (a bitstream compression) before finally being saved. However, this is a horrible idea, because the bins are already encrypted using AES-128 beforehand, and AES increases entropy, making the bit distribution appear roughly equal. This is the worst scenario for Huffmann, as data with high entropy will not compress properly, and the overhead of the symbol table might even make the final size larger. Compression was needed for the assignment but is not required afterwards, so that step can be removed. However, I will still retain the compression code in case I want to compress individual files or entries in the bins to optimise storage.
Tasks
Maybe I can also do a version update for the file format? I would have done that and added a utility to automatically update the format to the latest one if there were users of this application, but as this is more of a technical demo, I will omit these features, opting simply for updating the version of the file, making previous states invalid.