Hi Redpanda Team,
We are using Benthos (Redpanda Connect) for event delivery orchestration and are currently leveraging the kafka_franz output to publish messages to Kafka clusters.
While reviewing the documentation for kafka_franz, we noticed the following statement regarding the compression setting:
"Set an explicit compression type (optional). The default preference is to use snappy when the broker supports it. Otherwise, use none."
Reference: https://docs.redpanda.com/connect/components/outputs/kafka_franz/
We would like to better understand the default compression behavior when the compression field is not explicitly configured.
Questions:
- When compression is omitted from the kafka_franz configuration, does Benthos/Redpanda Connect automatically negotiate and select snappy if the Kafka broker advertises support for it?
- Is snappy the default compression algorithm used internally by the Franz Kafka client, or is this behavior implemented specifically within the kafka_franz output?
- Is there a way to verify from producer logs or metrics which compression codec has been selected at runtime?
Our goal is to understand exactly which compression codec is applied by default and whether explicit configuration is recommended to ensure consistent behavior across Kafka clusters.
Thanks for your help. We appreciate any additional details on how compression selection is handled internally by kafka_franz.
For reference, below is a simplified version of our current kafka_franz output configuration where the compression field is not explicitly set:
output:
broker:
copies: 10
outputs:
- fallback:
- kafka_franz:
key: ${! meta("kafka_key") }
max_in_flight: 1000
max_message_bytes: 1MB
seed_brokers:
- xyz-0:12345
- xyz-1:12345
- xyz-2:12345
tls:
client_certs:
- cert_file: /workspace/kafka-destination-cert.pem
key_file: /workspace/kafka-destination-key.pem
enabled: true
root_cas_file: /workspace/kafka-destination-ca.crt
topic: sdk.event.dual.kafka.sub-pok
label: primary_output
- kafka_franz:
key: ${! meta("kafka_key") }
max_in_flight: 1000
max_message_bytes: 1MB
seed_brokers:
- abc-0:67891
- abc-1:67891
- abc-2:67891
tls:
client_certs:
- cert_file: /workspace/kafka-destination-cert.pem
key_file: /workspace/kafka-destination-key.pem
enabled: true
root_cas_file: /workspace/kafka-destination-ca.crt
topic: sdk.event.dual.kafka.sub-pok
label: secondary_output
label: fallback_output
pattern: greedy
label: broker_output
Hi Redpanda Team,
We are using Benthos (Redpanda Connect) for event delivery orchestration and are currently leveraging the kafka_franz output to publish messages to Kafka clusters.
While reviewing the documentation for kafka_franz, we noticed the following statement regarding the compression setting:
"Set an explicit compression type (optional). The default preference is to use snappy when the broker supports it. Otherwise, use none."Reference: https://docs.redpanda.com/connect/components/outputs/kafka_franz/
We would like to better understand the default compression behavior when the compression field is not explicitly configured.
Questions:
Our goal is to understand exactly which compression codec is applied by default and whether explicit configuration is recommended to ensure consistent behavior across Kafka clusters.
Thanks for your help. We appreciate any additional details on how compression selection is handled internally by kafka_franz.
For reference, below is a simplified version of our current kafka_franz output configuration where the compression field is not explicitly set: