Do we need to cache optimized ePub and HTML images?

[benoit74]

With #375 and #376, we will soon be optimizing "things" again.

Unfortunately due to #288 investigations, we now know we cannot trust the etag / last-modified date from PG since this is supposed to be refreshed every month.

For now #375 and #376 will not focus on caching optimized assets to observe performance without that cache.

Two ways forward:

• performance is sufficient and cache does not bring sufficient added value vs downsides, so we do not cache anything in S3
• performance is too bad, then we cache and we invalidate cache once in a while (e.g. every year) to benefit a bit from it

[benoit74]

https://farm.openzim.org/pipeline/f061f83b-159d-4abe-a04b-9b45c71f1552 (which reoptimize HTML images and ePub content) took a 5 hours 2 mins to create the ZIM

https://farm.openzim.org/pipeline/7f9fbe5b-42c6-4707-843c-eab8b5b48a61 (without any optimization) took 1 hour 40 mins to create the ZIM

This clearly indicates that performance could be better with a caching system.

However, given the complexities of cache expiry and the numerous bug / discrepancies we might have linked to that, I suggest that we live without the cache for the time being. This is the only way to ensure anytime we create a ZIM we get the most up-to-date version of the content, and there is no big downside in my perspective (it is not like on Youtube where we already cache the video which takes way too long to recompress and we know fetching subtitles and chapters from Youtube might be blocked for throttling).

@kelson42 is that ok for you?