-
Notifications
You must be signed in to change notification settings - Fork 246
DOCS-1738 - Document 256KB log message size support #6881
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| title: Increased Maximum Log Message Size to 256KB (Search) | ||
| image: https://assets-www.sumologic.com/company-logos/_800x418_crop_center-center_82_none/SumoLogic_Preview_600x600.jpg?mtime=1617040082 | ||
| keywords: | ||
| - log message size | ||
| - large messages | ||
| - ingestion | ||
| hide_table_of_contents: true | ||
| --- | ||
|
|
||
| We're excited to announce that Sumo Logic now supports a maximum log message size of **256KB**, up from 64KB, so large single-line logs such as AWS CloudTrail, Google Cloud, and Microsoft 365 events are ingested without being split as often. Some features still handle messages at 64KB, including LogCompare, LogReduce, Cloud SIEM parsing, and Field Extraction Rules. [Learn more](/docs/search/get-started-with-search/search-basics/search-large-messages/). | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,12 @@ | ||
| --- | ||
| id: search-large-messages | ||
| title: Search Large Messages | ||
| description: When collecting log messages or event logs that are larger than 64KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
| description: When collecting log messages or event logs that are larger than 256KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
| --- | ||
|
|
||
|
|
||
|
|
||
| When collecting log messages or event logs that are larger than 64KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
| When collecting log messages or event logs that are larger than 256KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
|
|
||
| Chunks are ideally created at a line break depending on the Source type, protocol, and size of the message. Each section of the large messages is annotated with metadata to keep the message in order when viewing or searching the log. | ||
|
|
||
|
|
@@ -16,8 +16,17 @@ For additional information on collecting large messages review the relevant [Sou | |
|
|
||
| ## Query to Identify Truncated Logs | ||
|
|
||
| We have a [metadata](built-in-metadata.md) tag called `_size`. The `_size` metadata tag provides the size of the log message in bytes. A log is truncated at the size of 64K or 65536 bytes. | ||
| We have a [metadata](built-in-metadata.md) tag called `_size`. The `_size` metadata tag provides the size of the log message in bytes. A log is truncated at the size of 256K or 262144 bytes. | ||
|
|
||
| To find truncated logs you can reference the `_size` metadata tag, an example would be using the where operator. | ||
|
|
||
| `| where _size = 65536` | ||
| `| where _size = 262144` | ||
|
|
||
| ## Known limitations | ||
|
|
||
| Keep the following limitations in mind when you work with messages larger than 64KB: | ||
|
|
||
| - **LogCompare and LogReduce**. These operators truncate raw 256KB messages to 64KB before matching and grouping the logs into signatures, so content beyond 64KB is not considered. This can also affect response time when you run them against large messages. Learn more in [LogReduce](/docs/search/behavior-insights/logreduce/) and [LogCompare](/docs/search/behavior-insights/logcompare/). | ||
| - **Log Search messages table**. When you view large messages, the messages table displays up to 25 messages per page. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This 25 msg/page UI limitation will be removed as part of the GA based on this comment: https://sumologic.slack.com/archives/C02DUQT0A/p1783604422222719?thread_ts=1777391223.682879&cid=C02DUQT0A The new behavior is that search page will only show upto 64kb in the results table (after expansion), but the complete individual message can be viewed in the message inspector |
||
| - **Cloud SIEM**. Parsing and mapping might not process messages larger than 64KB correctly. | ||
| - **Field Extraction Rules**. The cumulative size of all fields extracted by a rule for a message is limited to 64KB, regardless of the message size. Learn more in [Field Extraction Rule limitations](/docs/manage/field-extractions/create-field-extraction-rule/#limitations). | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AWS CloudTrail, Google Cloud, and Microsoft 365 events - are they known large log sources?