Compression Handling of HTTP Response Content

[s1037989]

I'm looking for help with understanding the handling of compression.

Tapping into a typical HTTP response, we can see that the Content-Encoding is not marked gzip, but is being decompressed anyway:

$ perl -Mojo -E 'g("https://mojolicious.org")->tap(sub{warn sprintf "Content-Encoding: %s\n", $_->content->is_compressed ? "gzip" : ref $_->content->{gz} || "not compressed"})'
Content-Encoding: Compress::Raw::Zlib::inflateStream

But even with an explicit gzip, the content is still not detected as compressed:

$ perl -Mojo -E 'g("https://mojolicious.org" => {"Accept-Encoding" => "gzip"})->tap(sub{warn sprintf "Content-Encoding: %s\n", $_->content->is_compressed ? "gzip" : ref $_->content->{gz} || "not compressed"})'
Content-Encoding: Compress::Raw::Zlib::inflateStream

But with Accept-Encoding: identity, it is not being decompressed:

$ perl -Mojo -E 'g("https://mojolicious.org" => {"Accept-Encoding" => "identify"})->tap(sub{warn sprintf "Content-Encoding: %s\n", $_->content->is_compressed ? "gzip" : ref $_->content->{gz} || "not compressed"})'
Content-Encoding: not compressed

Why do none of these situations return true for is_compressed? What situation would create a true is_compressed content? And why does Compress::Raw::Zlib::Inflate get involved if is_compressed is false?

[identity-wael]

You're checking the response after decompression has completed. In Mojo::Content::_decompress, reaching Z_STREAM_END sets the decompressed Content-Length and removes Content-Encoding. Meanwhile, is_compressed simply checks whether the current Content-Encoding header equals gzip.

So the sequence is:

1. Response headers say gzip → is_compressed is true.
2. Mojo inflates the body.
3. At stream end it removes the gzip header → is_compressed is now false.

The private {gz} object can remain allocated afterwards. Its presence records implementation state, whereas is_compressed describes the content's current header; it isn't a “was compressed on the wire” flag.

To inspect the encoded response, disable automatic decompression before receiving it. The UA's transactor supports compressed(0); explicitly request gzip as well if you want to test that case:

use feature 'say';
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
$ua->transactor->compressed(0);
my $res = $ua->get('https://mojolicious.org' => {'Accept-Encoding' => 'gzip'})->result;
say $res->content->is_compressed ? 'gzip' : 'not gzip';

This still depends on the server actually choosing gzip. Also, the last command has identify; the standard token is identity.

[s1037989]

Thank you so much for that great response -- it absolutely improves my understanding of compression handling! It seemed like some magic was happening under the hood, but I knew it wouldn't be magic once uncovered. I specifically didn't realize that Mojo was updating the Content-Length and Content-Encoding headers, I thought they were always as delivered from the server.

With your explanation it can now easily be seen that the default auto-decompression reveals the uncompressed Content-Length whereas the non-default non-decompression reveals the compressed Content-Length as well as the Content-Encoding as reported by the server.

Thanks also for catching my typo of identity -- testing reveals that the outcome is the same, but yes, I should have used identity.

# $t->compressed(1) # default, try to negotiate compression for the response content and decompress it automatically
$ perl -Mojo -E 'app->ua->tap(sub{$_->transactor->compressed(1)})->get("https://mojolicious.org" => {"Accept-Encoding" => "gzip"})->result->tap(sub{warn sprintf "Content-Length: %s\nContent-Encoding: %s\n", $_->headers->content_length, $_->content->is_compressed ? "gzip" : "(none) "})'
Content-Length: 14082
Content-Encoding: (none)

# $t->compressed(0) # disable auto decompression
$ perl -Mojo -E 'app->ua->tap(sub{$_->transactor->compressed(0)})->get("https://mojolicious.org" => {"Accept-Encoding" => "gzip"})->result->tap(sub{warn sprintf "Content-Length: %s\nContent-Encoding: %s\n", $_->headers->content_length, $_->content->is_compressed ? "gzip" : "(none) "})'
Content-Length: 4342
Content-Encoding: gzip