updates for v5.0 #2541
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,161 @@ | ||||||
| # What's new in Fluent Bit v5.0 | ||||||
|
|
||||||
| Fluent Bit `v5.0` adds new inputs and processors, expands authentication and TLS options, and standardizes configuration for HTTP-based plugins. It also delivers an important round of performance and scalability work, especially for pipelines that ingest logs, metrics, and traces through HTTP-based protocols. This page gives a quick user-focused overview of the main changes since Fluent Bit `v4.2`. | ||||||
|
|
||||||
| For migration-impacting changes, see [Upgrade notes](upgrade-notes.md). | ||||||
|
|
||||||
| ## Performance and scalability | ||||||
|
|
||||||
| ### Unified processing and delivery model | ||||||
|
|
||||||
| Fluent Bit `v5.0` continues the move toward a more unified runtime for logs, metrics, and traces. In practice, this means the same core engine improvements benefit more of the pipeline, instead of individual signal paths evolving separately. | ||||||
|
|
||||||
| For end users, the result is a more consistent behavior across telemetry types and a better base for high-throughput pipelines that mix logs, metrics, and traces in the same deployment. | ||||||
|
|
||||||
| ### Refactored HTTP stack | ||||||
|
|
||||||
| One of the most important `v5.0` changes is the refactoring of the HTTP listener stack used by several input plugins. Fluent Bit now uses a shared HTTP server implementation across the major HTTP-based receivers instead of maintaining separate code paths. | ||||||
|
|
||||||
| This work improves: | ||||||
|
|
||||||
| - concurrency through shared listener worker support | ||||||
| - consistency of request handling across HTTP-based inputs | ||||||
| - buffer enforcement and connection handling | ||||||
| - maintainability, which reduces drift between plugin implementations | ||||||
|
|
||||||
| The biggest user-facing beneficiaries are: | ||||||
|
|
||||||
| - [HTTP input](../pipeline/inputs/http.md) | ||||||
| - [Splunk input](../pipeline/inputs/splunk.md) | ||||||
| - [Elasticsearch input](../pipeline/inputs/elasticsearch.md) | ||||||
| - [OpenTelemetry input](../pipeline/inputs/opentelemetry.md) | ||||||
| - [Prometheus remote write input](../pipeline/inputs/prometheus-remote-write.md) | ||||||
|
|
||||||
| If you run large HTTP or OTLP ingestion workloads, `v5.0` is not only a feature release. It is also a meaningful runtime improvement. | ||||||
|
Check warning on line 34 in installation/whats-new-in-fluent-bit-v5.0.md
|
||||||
|
|
||||||
| ## Configuration and operations | ||||||
|
|
||||||
| ### Shared HTTP listener settings | ||||||
|
|
||||||
| HTTP-based inputs now use a shared listener configuration model. The preferred setting names are: | ||||||
|
|
||||||
| - `http_server.http2` | ||||||
| - `http_server.buffer_chunk_size` | ||||||
| - `http_server.buffer_max_size` | ||||||
| - `http_server.max_connections` | ||||||
| - `http_server.workers` | ||||||
|
|
||||||
| Legacy aliases such as `http2`, `buffer_chunk_size`, and `buffer_max_size` still work, but new configurations should use the `http_server.*` names. | ||||||
|
|
||||||
| Affected plugin families include: | ||||||
|
|
||||||
| - [HTTP input](../pipeline/inputs/http.md) | ||||||
| - [Splunk input](../pipeline/inputs/splunk.md) | ||||||
| - [Elasticsearch input](../pipeline/inputs/elasticsearch.md) | ||||||
| - [OpenTelemetry input](../pipeline/inputs/opentelemetry.md) | ||||||
| - [Prometheus remote write input](../pipeline/inputs/prometheus-remote-write.md) | ||||||
|
|
||||||
| ### Mutual TLS for inputs | ||||||
|
|
||||||
| Input plugins that support TLS can now require client certificate verification with `tls.verify_client_cert`. This makes it easier to run mutual TLS (`mTLS`) directly on Fluent Bit listeners. | ||||||
|
|
||||||
| See [TLS](../administration/transport-security.md). | ||||||
|
|
||||||
| ### JSON health endpoint in API v2 | ||||||
|
|
||||||
| The built-in HTTP server exposes `/api/v2/health`, which returns health status as JSON and uses the HTTP status code to indicate healthy (`200`) or unhealthy (`500`) state. | ||||||
|
|
||||||
| See [Monitoring](../administration/monitoring.md). | ||||||
|
|
||||||
| ## Inputs | ||||||
|
|
||||||
| ### New `fluentbit_logs` input | ||||||
|
|
||||||
| The [Fluent Bit logs input](../pipeline/inputs/fluentbit-logs.md) routes Fluent Bit internal logs back into the pipeline as structured records. This lets you forward agent diagnostics to any supported destination. | ||||||
|
|
||||||
| ### HTTP input remote address capture | ||||||
|
|
||||||
| The [HTTP input](../pipeline/inputs/http.md) adds: | ||||||
|
|
||||||
| - `add_remote_addr` | ||||||
| - `remote_addr_key` | ||||||
|
|
||||||
| These settings let you attach the client address from `X-Forwarded-For` to each ingested record. | ||||||
|
|
||||||
| ### `OAuth 2.0` bearer token validation on HTTP-based inputs | ||||||
|
|
||||||
| HTTP-based receivers can validate incoming bearer tokens with: | ||||||
|
|
||||||
| - `oauth2.validate` | ||||||
| - `oauth2.issuer` | ||||||
| - `oauth2.jwks_url` | ||||||
| - `oauth2.allowed_audience` | ||||||
| - `oauth2.allowed_clients` | ||||||
| - `oauth2.jwks_refresh_interval` | ||||||
|
|
||||||
| This is available on the relevant input plugins, including [HTTP](../pipeline/inputs/http.md) and [OpenTelemetry](../pipeline/inputs/opentelemetry.md). | ||||||
|
|
||||||
| ### OpenTelemetry input improvements | ||||||
|
|
||||||
| The [OpenTelemetry input](../pipeline/inputs/opentelemetry.md) in `v5.0` expands user-visible behavior with: | ||||||
|
|
||||||
| - shared HTTP listener worker support | ||||||
| - `OAuth 2.0` bearer token validation | ||||||
| - stable JSON metrics ingestion over `OTLP/HTTP` | ||||||
| - improved JSON trace validation and error reporting | ||||||
|
|
||||||
| ### Kubernetes events state database controls | ||||||
|
|
||||||
| The [Kubernetes events input](../pipeline/inputs/kubernetes-events.md) documents additional SQLite controls: | ||||||
|
|
||||||
| - `db.journal_mode` | ||||||
| - `db.locking` | ||||||
|
|
||||||
| These settings help tune event cursor persistence and database access behavior. | ||||||
|
|
||||||
| ## Processors | ||||||
|
|
||||||
| ### New cumulative-to-delta processor | ||||||
|
|
||||||
| The [cumulative to delta processor](../pipeline/processors/cumulative-to-delta.md) converts cumulative monotonic metrics to delta values, which is useful when scraping Prometheus-style metrics but exporting to backends that expect deltas. | ||||||
|
Check warning on line 120 in installation/whats-new-in-fluent-bit-v5.0.md
|
||||||
|
Contributor
There was a problem hiding this comment. Use neutral phrasing and keep compound adjective hyphenated. Current wording trips style checks. Suggested rewrite: Suggested edit-The [cumulative to delta processor](../pipeline/processors/cumulative-to-delta.md) converts cumulative monotonic metrics to delta values, which is useful when scraping Prometheus-style metrics but exporting to backends that expect deltas.
+The [cumulative-to-delta processor](../pipeline/processors/cumulative-to-delta.md) converts cumulative monotonic metrics to delta values for deployments that scrape Prometheus-style metrics but export to backends that expect deltas.📝 Committable suggestion
Suggested change
🧰 Tools🪛 GitHub Actions: Lint PRs[warning] 120-120: vale (reviewdog) [FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult. 🪛 GitHub Check: runner / vale[warning] 120-120: Raw Output: 🪛 LanguageTool[grammar] ~120-~120: Use a hyphen to join words. (QB_NEW_EN_HYPHEN) 🤖 Prompt for AI Agents |
||||||
|
|
||||||
| ### New topological data analysis processor | ||||||
|
|
||||||
| The [topological data analysis processor](../pipeline/processors/tda.md) adds a metrics processor for topology-based analysis workflows. | ||||||
|
Check warning on line 124 in installation/whats-new-in-fluent-bit-v5.0.md
|
||||||
|
Contributor
There was a problem hiding this comment. Replace flagged terms to satisfy lint rules. Two wording tweaks should remove current warnings: Suggested edit-The [topological data analysis processor](../pipeline/processors/tda.md) adds a metrics processor for topology-based analysis workflows.
+The [topological data analysis processor](../pipeline/processors/tda.md) adds a metrics processor for shape-based analysis workflows.-- [Amazon Kinesis Data Firehose](../pipeline/outputs/firehose.md): `snappy` added alongside existing codecs
-- [Amazon S3](../pipeline/outputs/s3.md): `snappy` added alongside existing codecs
+- [Amazon Kinesis Data Firehose](../pipeline/outputs/firehose.md): `snappy` added alongside existing compression algorithms
+- [Amazon S3](../pipeline/outputs/s3.md): `snappy` added alongside existing compression algorithmsAlso applies to: 147-148 🧰 Tools🪛 GitHub Actions: Lint PRs[warning] 124-124: vale (reviewdog) [FluentBit.Terms] Use 'placement' instead of 'topology'. 🪛 GitHub Check: runner / vale[warning] 124-124: Raw Output: 🤖 Prompt for AI Agents |
||||||
|
|
||||||
| ### Sampling processor updates | ||||||
|
|
||||||
| The [sampling processor](../pipeline/processors/sampling.md) adds `legacy_reconcile` for tail sampling, which helps compare the optimized reconciler with the previous behavior when validating upgrades. | ||||||
|
|
||||||
| ## Outputs | ||||||
|
|
||||||
| ### HTTP output `OAuth 2.0` client credentials | ||||||
|
|
||||||
| The [HTTP output](../pipeline/outputs/http.md) now supports built-in `OAuth 2.0` client credentials with: | ||||||
|
|
||||||
| - `basic` | ||||||
| - `post` | ||||||
| - `private_key_jwt` | ||||||
|
|
||||||
| You can configure token acquisition directly in Fluent Bit with the `oauth2.*` settings. | ||||||
|
|
||||||
| ### More compression options for cloud outputs | ||||||
|
|
||||||
| Several outputs gained additional compression support in the `v4.2` to `v5.0` range: | ||||||
|
|
||||||
| - [Amazon Kinesis Data Streams](../pipeline/outputs/kinesis.md): `gzip`, `zstd`, `snappy` | ||||||
| - [Amazon Kinesis Data Firehose](../pipeline/outputs/firehose.md): `snappy` added alongside existing codecs | ||||||
|
Check warning on line 147 in installation/whats-new-in-fluent-bit-v5.0.md
|
||||||
| - [Amazon S3](../pipeline/outputs/s3.md): `snappy` added alongside existing codecs | ||||||
|
Check warning on line 148 in installation/whats-new-in-fluent-bit-v5.0.md
|
||||||
| - [Azure Blob](../pipeline/outputs/azure_blob.md): `zstd` support for transfer compression | ||||||
|
|
||||||
| ## Monitoring changes | ||||||
|
|
||||||
| ### `fluentbit_hot_reloaded_times` is now a counter | ||||||
|
|
||||||
| The `fluentbit_hot_reloaded_times` metric changed from a gauge to a counter, which makes it safe to use with PromQL functions such as `rate()` and `increase()`. | ||||||
|
|
||||||
| ### New output backpressure visibility | ||||||
|
|
||||||
| `v5.0` adds output backpressure duration metrics so you can observe time spent waiting because of downstream pressure. | ||||||
|
|
||||||
| See [Monitoring](../administration/monitoring.md). | ||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -49,10 +49,13 @@ The plugin supports the following rules: | |
| | `Remove_regex` | `REGEXP:KEY` | _none_ | Remove all key/value pairs with key matching regexp `KEY`. | | ||
| | `Rename` | `STRING:KEY` | `STRING:RENAMED_KEY` | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists and `RENAMED_KEY` doesn't exist. | | ||
| | `Hard_rename` | `STRING:KEY` | `STRING:RENAMED_KEY` | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists. If `RENAMED_KEY` already exists, this field is overwritten. | | ||
| | `Hard_Rename` | `STRING:KEY` | `STRING:RENAMED_KEY` | Equivalent to `Hard_rename`. This spelling is also accepted by the plugin. | | ||
| | `Copy` | `STRING:KEY` | `STRING:COPIED_KEY` | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists and `COPIED_KEY` doesn't exist. | | ||
| | `Hard_copy` | `STRING:KEY` | `STRING:COPIED_KEY` | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists. If `COPIED_KEY` already exists, this field is overwritten. | | ||
| | `Move_to_start` | `WILDCARD:KEY` | _none_ | Move key/value pairs with keys matching `KEY` to the start of the message. | | ||
| | `Move_To_Start` | `WILDCARD:KEY` | _none_ | Equivalent to `Move_to_start`. This spelling is also accepted by the plugin. | | ||
| | `Move_to_end` | `WILDCARD:KEY` | _none_ | Move key/value pairs with keys matching `KEY` to the end of the message. | | ||
| | `Move_To_End` | `WILDCARD:KEY` | _none_ | Equivalent to `Move_to_end`. This spelling is also accepted by the plugin. | | ||
|
|
||
| - Rules are case-insensitive, but parameters aren't. | ||
| - Any number of rules can be set in a filter instance. | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,6 +16,7 @@ | |
| | `parser` | Specify the parser name to interpret the field. Multiple parser entries are allowed (one per line). | _none_ | | ||
| | `preserve_key` | Keep the original `key_name` field in the parsed result. If false, the field will be removed. | `false` | | ||
| | `reserve_data` | Keep all other original fields in the parsed result. If false, all other original fields will be removed. | `false` | | ||
| | `Unescape_key` | Deprecated. This option is retained only for backward compatibility and should not be used in new configurations. | _deprecated_ | | ||
|
Check warning on line 19 in pipeline/filters/parser.md
|
||
|
|
||
| ## Get started | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,6 +23,7 @@ The plugin supports the following configuration parameters: | |
| | `samples` | Limit the number of events generated. For example, if `samples=3`, the plugin generates only three events and stops. `0` means no limit. | `0` | | ||
| | `start_time_nsec` | Set a dummy base timestamp, in nanoseconds. If set to `-1`, the current time is used. | `-1` | | ||
| | `start_time_sec` | Set a dummy base timestamp, in seconds. If set to `-1`, the current time is used. | `-1` | | ||
| | `test_hang_on_exit` | Test-only option that simulates a hang during shutdown for hot reload watchdog testing. Don't use this in production configurations. | `false` | | ||
| | `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | | ||
|
|
||
| ## Get started | ||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Expand
JWTonce in this description to clear lint warning.Suggested update:
Suggested edit
📝 Committable suggestion
🧰 Tools
🪛 GitHub Actions: Lint PRs
[warning] 176-176: vale (reviewdog) [FluentBit.Acronyms] Spell out 'JWT', if it's unfamiliar to the audience.
🪛 GitHub Check: runner / vale
[warning] 176-176:
[vale] reported by reviewdog 🐶
[FluentBit.Acronyms] Spell out 'JWT', if it's unfamiliar to the audience.
Raw Output:
{"message": "[FluentBit.Acronyms] Spell out 'JWT', if it's unfamiliar to the audience.", "location": {"path": "administration/configuring-fluent-bit/yaml/pipeline-section.md", "range": {"start": {"line": 176, "column": 25}}}, "severity": "INFO"}
🤖 Prompt for AI Agents