Skip to content

Remove pcap flavours from the report zip archive#2992

Open
RobertBendun wants to merge 2 commits into
kevoreilly:masterfrom
PCSS-PSNC:feature/report-zip-without-pcap-flavours
Open

Remove pcap flavours from the report zip archive#2992
RobertBendun wants to merge 2 commits into
kevoreilly:masterfrom
PCSS-PSNC:feature/report-zip-without-pcap-flavours

Conversation

@RobertBendun
Copy link
Copy Markdown
Contributor

Pcap flavours carry information that can be found already within the archive - original pcap and TLS info are still included.

Removing them from archive can significantly reduce the resulting archive size and reduce the time needed for the API endpoint response

Justification

Analysis with a lot of network traffic produced this artifacts:

/opt/CAPEv2/storage/analyses/537# ls -lahS
total 1.4G
-rw-r--r--   1 cape cape 469M Apr 21 18:15 dump_mixed.pcap
-rw-r--r--   1 cape cape 468M Apr 21 18:18 dump.pcapng
-rw-rw-r--   1 cape cape 467M Apr 21 18:15 dump.pcap
-rw-r--r--   1 cape cape 2.0M Apr 21 18:15 dump_decrypted.pcap
-rw-r--r--   1 cape cape 508K Apr 21 18:15 analysis.log
-rw-rw-r--   1 cape cape 125K Apr 21 18:15 cuckoo.log
-rw-rw-r--   1 cape cape  54K Apr 21 18:18 files.json

When trying to download zip archive with apiv2/tasks/get/report/<id>/all/zip/, archive creation took over 3 minutes (when we increased nginx proxy timeout) blocking the whole CPU core for this computation.

Alternative solution

I understand if you want to keep all of the resulting pcap flavours in the resulting archive - I propose as alternative solution another type of export, something like all_but_derived_pcaps (better name needs to be found).

Pcap flavours carry information that can be found already within the
archive - original pcap and TLS info are still included.

Removing them from archive can significantly reduce the resulting
archive size and reduce the time needed for the API endpoint response
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the tasks_report view to exclude several derived pcap files from the 'all' report format, aiming to reduce the size of generated archives. A suggestion was made to also include dump.pcapng in the exclusion list, as it is a large artifact that aligns with the goal of minimizing archive size.

Comment thread web/apiv2/views.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant