diff --git a/blog-service/2026-07-09-search.md b/blog-service/2026-07-09-search.md new file mode 100644 index 00000000000..2be8adafa6b --- /dev/null +++ b/blog-service/2026-07-09-search.md @@ -0,0 +1,11 @@ +--- +title: Increased Maximum Log Message Size to 256KB (Search) +image: https://assets-www.sumologic.com/company-logos/_800x418_crop_center-center_82_none/SumoLogic_Preview_600x600.jpg?mtime=1617040082 +keywords: + - log message size + - large messages + - ingestion +hide_table_of_contents: true +--- + +We're excited to announce that Sumo Logic now supports a maximum log message size of **256KB**, up from 64KB, so large single-line logs such as AWS CloudTrail, Google Cloud, and Microsoft 365 events are ingested without being split as often. Some features still handle messages at 64KB, including LogCompare, LogReduce, Cloud SIEM parsing, and Field Extraction Rules. [Learn more](/docs/search/get-started-with-search/search-basics/search-large-messages/). diff --git a/docs/reuse/fer-limitations.md b/docs/reuse/fer-limitations.md index 57b7d6b6bb9..6295ad59112 100644 --- a/docs/reuse/fer-limitations.md +++ b/docs/reuse/fer-limitations.md @@ -6,5 +6,5 @@ Ingest Time FERs have the following limitations: ::: * Ingest Time rule expressions are limited to a maximum of 16k (16,384) characters. * Ingest Time rules can extract up to a maximum of 16k (16,384) characters for each field. -* The cumulative size of all fields extracted by a rule for a message/event is limited to 64kb. +* The cumulative size of all fields extracted by a rule for a message/event is limited to 64kb, regardless of the maximum log message size. * Ingest Time rules **only apply to data moving forward**. If you want to [parse](/docs/search/search-query-language/parse-operators) data ingested before the creation of your Ingest Time FER, you can either parse your data in your query, or create [Scheduled Views](/docs/manage/scheduled-views) to extract fields for your historical data. diff --git a/docs/search/behavior-insights/logcompare.md b/docs/search/behavior-insights/logcompare.md index 865e17bc428..06fc591a7c9 100644 --- a/docs/search/behavior-insights/logcompare.md +++ b/docs/search/behavior-insights/logcompare.md @@ -11,6 +11,10 @@ LogCompare allows you to easily compare log data from different time periods to For example, you could determine what was different right before a failure compared to the previous day or previous week. Or, you could easily check if a new release introduced a new issue by reviewing the difference in log streams across time. +:::note +LogCompare truncates raw 256KB messages to 64KB before matching and grouping the logs into signatures, so content beyond 64KB is not compared. This can also affect response time when you run LogCompare against large messages. +::: + ## How LogCompare works LogCompare creates a baseline (historical) query and compares it to a target (current) query. Logs from each query are clustered into patterns, called signatures, then compared and ranked based on the significance of change. diff --git a/docs/search/behavior-insights/logreduce/detect-patterns-with-logreduce.md b/docs/search/behavior-insights/logreduce/detect-patterns-with-logreduce.md index 15319c09ca5..56c09d55a8c 100644 --- a/docs/search/behavior-insights/logreduce/detect-patterns-with-logreduce.md +++ b/docs/search/behavior-insights/logreduce/detect-patterns-with-logreduce.md @@ -12,6 +12,10 @@ The **Signatures** tab displays LogReduce results as signatures. A signature You can refine the results of the LogReduce algorithm to make the outcome more generic or more specific. See [Influencing the LogReduce Outcome](influence-the-logreduce-outcome.md) for more information. +:::note +LogReduce truncates raw 256KB messages to 64KB before matching and grouping the logs into signatures, so content beyond 64KB is not included. This can also affect response time when you run LogReduce against large messages. +::: + ## Will my LogReduce search results match my keyword search results? Generally speaking, no. LogReduce is intended to be a jumping-off point for your analysis. Unlike a keyword search, where you are looking for data related to, say, a specific source or an error message, LogReduce returns signatures that contain messages that *may* be of interest to you using fuzzy logic. If you are not happy with a signature, you can [teach](influence-the-logreduce-outcome.md) LogReduce how you'd like the results to be made more specific. do not think of a signature as an example of what logs are grouped under it; instead think of a signature as a reflection of what LogReduce thinks you will find interesting if that signature catches your eye. Once you begin digging in to LogReduce results, you will then want to structure a keyword query that delivers precise results. diff --git a/docs/search/get-started-with-search/search-basics/search-large-messages.md b/docs/search/get-started-with-search/search-basics/search-large-messages.md index c8cc08f9b53..106d191157c 100644 --- a/docs/search/get-started-with-search/search-basics/search-large-messages.md +++ b/docs/search/get-started-with-search/search-basics/search-large-messages.md @@ -1,12 +1,12 @@ --- id: search-large-messages title: Search Large Messages -description: When collecting log messages or event logs that are larger than 64KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. +description: When collecting log messages or event logs that are larger than 256KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. --- -When collecting log messages or event logs that are larger than 64KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. +When collecting log messages or event logs that are larger than 256KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. Chunks are ideally created at a line break depending on the Source type, protocol, and size of the message. Each section of the large messages is annotated with metadata to keep the message in order when viewing or searching the log. @@ -16,8 +16,17 @@ For additional information on collecting large messages review the relevant [Sou ## Query to Identify Truncated Logs  -We have a [metadata](built-in-metadata.md) tag called `_size`. The `_size` metadata tag provides the size of the log message in bytes. A log is truncated at the size of 64K or 65536 bytes. +We have a [metadata](built-in-metadata.md) tag called `_size`. The `_size` metadata tag provides the size of the log message in bytes. A log is truncated at the size of 256K or 262144 bytes. To find truncated logs you can reference the `_size` metadata tag, an example would be using the where operator. -`| where _size = 65536` +`| where _size = 262144` + +## Known limitations + +Keep the following limitations in mind when you work with messages larger than 64KB: + +- **LogCompare and LogReduce**. These operators truncate raw 256KB messages to 64KB before matching and grouping the logs into signatures, so content beyond 64KB is not considered. This can also affect response time when you run them against large messages. Learn more in [LogReduce](/docs/search/behavior-insights/logreduce/) and [LogCompare](/docs/search/behavior-insights/logcompare/). +- **Log Search messages table**. When you view large messages, the messages table displays up to 25 messages per page. +- **Cloud SIEM**. Parsing and mapping might not process messages larger than 64KB correctly. +- **Field Extraction Rules**. The cumulative size of all fields extracted by a rule for a message is limited to 64KB, regardless of the message size. Learn more in [Field Extraction Rule limitations](/docs/manage/field-extractions/create-field-extraction-rule/#limitations). diff --git a/docs/send-data/hosted-collectors/cloud-syslog-source/index.md b/docs/send-data/hosted-collectors/cloud-syslog-source/index.md index 47d4015725c..67d299170e6 100644 --- a/docs/send-data/hosted-collectors/cloud-syslog-source/index.md +++ b/docs/send-data/hosted-collectors/cloud-syslog-source/index.md @@ -11,7 +11,7 @@ import useBaseUrl from '@docusaurus/useBaseUrl'; You can configure a cloud syslog source to allow a syslog client to send [RFC 5424](https://tools.ietf.org/html/rfc5424)-compliant messages to Sumo Logic. A source-specific token, generated by Sumo Logic, is inserted into each message to identify the source. Transport Layer Security (TLS) 1.3 over TCP is required. -Syslog messages must be compliant with [RFC 5424](https://tools.ietf.org/html/rfc5424) or they are dropped. Messages over 64 KB are truncated. +Syslog messages must be compliant with [RFC 5424](https://tools.ietf.org/html/rfc5424) or they are dropped. Messages over 256 KB are truncated. Sumo Logic manages an elastic scaling set of syslog servers, which scales up and down behind a set of AWS Elastic Load Balancers. The AWS ELB set can also scale up and down. For this reason, instead of IP address-based endpoints, Sumo Logic uses endpoint hostnames in this format: @@ -103,7 +103,7 @@ Follow the configuration instructions for your syslog client: ### Message format -Syslog messages must be in RFC 5424-compliant form or they are dropped. Messages over 64KB in length are truncated. +Syslog messages must be in RFC 5424-compliant form or they are dropped. Messages over 256KB in length are truncated. This diagram shows the RFC 5424 format: diff --git a/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/qualys-vmdr-source.md b/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/qualys-vmdr-source.md index 085293e6cdf..5a1ecf09a37 100644 --- a/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/qualys-vmdr-source.md +++ b/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/qualys-vmdr-source.md @@ -95,7 +95,7 @@ Click [here](/docs/c2c/info) for more information about Cloud-to-Cloud sources. ### Is anything changed with data for computer inventory? -Sometimes the asset information from the computer inventory data can exceed the [Sumo Logic maximum log size of 64KB](/docs/search/get-started-with-search/search-basics/search-large-messages/). Sumo Logic will automatically split log messages exceeding the size limit into smaller chunks. This C2C makes the following changes to the computer inventory asset data collected in order to keep most logs under the size limit and prevent splitting: +Sometimes the asset information from the computer inventory data can exceed the [Sumo Logic maximum log size of 256KB](/docs/search/get-started-with-search/search-basics/search-large-messages/). Sumo Logic will automatically split log messages exceeding the size limit into smaller chunks. This C2C makes the following changes to the computer inventory asset data collected in order to keep most logs under the size limit and prevent splitting: - The `openPortListData` key only contains information about ports open since the last time computer asset was ingested instead of listing all open port history from all time. - The `softwareListData` is reduced down from the full details to simply a list/array of software names using the full name. diff --git a/docs/send-data/reference-information/collect-multiline-logs.md b/docs/send-data/reference-information/collect-multiline-logs.md index fd09696e5a5..069167d0c36 100644 --- a/docs/send-data/reference-information/collect-multiline-logs.md +++ b/docs/send-data/reference-information/collect-multiline-logs.md @@ -15,7 +15,7 @@ The line breaks or carriage returns are control characters used to create new li Multiline messages that are more than 2,000 lines or 512KB in size will get flushed and collected as single log lines due to the default log message size limitations. Depending on the Collector's available memory, you may be able to increase this limit. Contact Support for assistance by navigating to **Help** > **Support** in the Sumo Logic menu. -Messages will be further broken down when they are received and indexed. When collecting log messages that are larger than 64KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. Chunks are ideally created at a line break depending on the Source type, protocol, and size of the message. Each section of the large messages is annotated with metadata to keep the message in order when viewing or searching the log. Review the relevant Source's documentation for additional information. +Messages will be further broken down when they are received and indexed. When collecting log messages that are larger than 256KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. Chunks are ideally created at a line break depending on the Source type, protocol, and size of the message. Each section of the large messages is annotated with metadata to keep the message in order when viewing or searching the log. Review the relevant Source's documentation for additional information. ## Multiline Processing Options