Skip to content

Clarification on Default Compression Behavior in kafka_franz Output #4650

Description

@sdktechsolutions

Hi Redpanda Team,

We are using Benthos (Redpanda Connect) for event delivery orchestration and are currently leveraging the kafka_franz output to publish messages to Kafka clusters.
While reviewing the documentation for kafka_franz, we noticed the following statement regarding the compression setting:

"Set an explicit compression type (optional). The default preference is to use snappy when the broker supports it. Otherwise, use none."

Reference: https://docs.redpanda.com/connect/components/outputs/kafka_franz/

We would like to better understand the default compression behavior when the compression field is not explicitly configured.

Questions:

  • When compression is omitted from the kafka_franz configuration, does Benthos/Redpanda Connect automatically negotiate and select snappy if the Kafka broker advertises support for it?
  • Is snappy the default compression algorithm used internally by the Franz Kafka client, or is this behavior implemented specifically within the kafka_franz output?
  • Is there a way to verify from producer logs or metrics which compression codec has been selected at runtime?

Our goal is to understand exactly which compression codec is applied by default and whether explicit configuration is recommended to ensure consistent behavior across Kafka clusters.

Thanks for your help. We appreciate any additional details on how compression selection is handled internally by kafka_franz.

For reference, below is a simplified version of our current kafka_franz output configuration where the compression field is not explicitly set:

output:
  broker:
    copies: 10
    outputs:
      - fallback:
          - kafka_franz:
              key: ${! meta("kafka_key") }
              max_in_flight: 1000
              max_message_bytes: 1MB
              seed_brokers:
                - xyz-0:12345
                - xyz-1:12345
                - xyz-2:12345
              tls:
                client_certs:
                  - cert_file: /workspace/kafka-destination-cert.pem
                    key_file: /workspace/kafka-destination-key.pem
                enabled: true
                root_cas_file: /workspace/kafka-destination-ca.crt
              topic: sdk.event.dual.kafka.sub-pok
            label: primary_output
          - kafka_franz:
              key: ${! meta("kafka_key") }
              max_in_flight: 1000
              max_message_bytes: 1MB
              seed_brokers:
                - abc-0:67891
                - abc-1:67891
                - abc-2:67891
              tls:
                client_certs:
                  - cert_file: /workspace/kafka-destination-cert.pem
                    key_file: /workspace/kafka-destination-key.pem
                enabled: true
                root_cas_file: /workspace/kafka-destination-ca.crt
              topic: sdk.event.dual.kafka.sub-pok
            label: secondary_output
        label: fallback_output
    pattern: greedy
  label: broker_output

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions