Skip to content

Commit ccdd12a

Browse files
Kawronjplatosz-splunk
Kawron
and
jplatosz-splunk
authored
docs: Add additional information regarding performance fine-tuning (#2833)
* docs: Add additional information regarding performance fine-tuning * docs: Revert removing SC4S_DEST_SPLUNK_HEC_DEFAULT_WORKERS * docs: Change Performance note * docs: Add information about disk buffering * docs: simplify fine-tune section * docs: Suggested changes * docs: Add note formating * docs: Fix spaces in notes * docs: Enhance section headers in fine-tuning.md docs: Updated formatting for sections on UDP socket behavior. * docs: Suggested changes --------- Co-authored-by: JustynaPe <jpatosz@cisco.com>
1 parent 3d90945 commit ccdd12a

8 files changed

Lines changed: 271 additions & 178 deletions

File tree

‎docs/architecture/fine-tuning.md‎

Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
# Finetune SC4S
2+
3+
This section provides guidance on improving SC4S performance by tuning configuration settings.
4+
5+
You can apply these settings to your infrastructure to improve SC4S performance. After making adjustments, run the [performance tests](performance-tests.md#check-your-tcp-performance) and retain the changes that result in performance improvements.
6+
7+
## Disable features that reduce performance
8+
9+
Some SC4S features may negatively impact performance. If you experience performance issues, consider disabling some of the settings like **name cache** and **message grouping**:
10+
11+
Edit `/opt/sc4s/env_file`:
12+
13+
```bash
14+
SC4S_USE_NAME_CACHE=no
15+
SC4S_SOURCE_VMWARE_VSPHERE_GROUPMSG=no
16+
```
17+
18+
Restart SC4S for the changes to take effect.
19+
20+
## Dedicated sc4s instances
21+
22+
If one of the logs sources produces a large percentage of the overall traffic, create an additional dedicated sc4s service on a separate host.
23+
24+
## Tune the receiving buffer
25+
26+
Increasing the receive buffer allows the kernel to queue more incoming data before the application processes it, reducing packet loss during traffic bursts. This requires changes at both the OS level and within SC4S. Start turning the receiving buffer with increasing the kernel buffer OS limits. Perform the following steps to change the buffer size:
27+
28+
1. Edit `/etc/sysctl.conf` and set the receive buffer size to 512 MB:
29+
30+
```bash
31+
net.core.rmem_default = 536870912
32+
net.core.rmem_max = 536870912
33+
```
34+
35+
2. Apply the changes:
36+
37+
```bash
38+
sudo sysctl -p
39+
```
40+
41+
Next configure SC4S to use the larger buffer:
42+
43+
1. Add the following line to `/opt/sc4s/env_file`:
44+
45+
```bash
46+
SC4S_SOURCE_TCP_SO_RCVBUFF=536870912
47+
```
48+
49+
2. Apply the same buffer tuning to each syslog transport you have enabled:
50+
51+
```bash
52+
SC4S_SOURCE_TCP_SO_RCVBUFF=536870912 # Generic syslog over TCP
53+
SC4S_SOURCE_UDP_SO_RCVBUFF=536870912 # Generic syslog over UDP
54+
SC4S_SOURCE_RFC5426_SO_RCVBUFF=536870912 # RFC 5426 (syslog over UDP)
55+
SC4S_SOURCE_RFC6587_SO_RCVBUFF=536870912 # RFC 6587 (syslog over TCP)
56+
SC4S_SOURCE_RFC5425_SO_RCVBUFF=536870912 # RFC 5425 (syslog over TLS)
57+
```
58+
59+
3. Restart SC4S for the changes to take effect.
60+
61+
### Additional considerations
62+
63+
- **Sending buffer:** In rare cases, you may also need to increase the sending buffer size by modifying `net.core.wmem_max` and `net.core.wmem_default` using the same approach.
64+
- **Buffer size limits:** Setting buffers too large can actually decrease performance. Start with the recommended values and adjust based on your testing results.
65+
- **Hardware constraints:** Network driver limitations should be considered when tuning these values. Consult your NIC documentation for maximum supported buffer sizes.
66+
67+
## Tune static input window size
68+
69+
Input window provides flow‑control at the application level. Syslog‑ng uses this feature to temporarily buffer messages when outputs are slow. The mechanism works by pulling messages from the kernel’s receive buffer and placing them into an application buffer.
70+
71+
- The window size defines how many messages this internal buffer can hold.
72+
- Each message fetched from the kernel buffer increases the window counter.
73+
- Each message successfully forwarded to the output decreases the counter.
74+
75+
To change the window size, modify the following options in `/opt/sc4s/env_file`:
76+
77+
**for TCP**:
78+
79+
```bash
80+
SC4S_SOURCE_TCP_IW_SIZE=1000000
81+
```
82+
83+
**for UDP**:
84+
85+
```bash
86+
SC4S_SOURCE_UDP_IW_USE=yes
87+
SC4S_SOURCE_UDP_IW_SIZE=1000000
88+
```
89+
90+
Restart SC4S for the changes to take effect.
91+
92+
In the example above, the input window can store up to **1,000,000 messages**. Note that increasing the window size will increase the application's memory usage.
93+
94+
For **UDP**, if the output becomes slow and this window fills up, syslog‑ng will stop reading from the kernel buffer. As a result, the kernel buffer will begin to fill, and once it becomes full, **incoming UDP packets will be dropped** by the kernel.
95+
96+
A single UDP message can be up to approximately 1 KB. With a window size of 1,000,000 messages, this may require up to **1 GB** of additional memory for buffering.
97+
98+
### Fetch limit
99+
100+
When increasing the input window size, you may also need to increase the **fetch limit**. The fetch limit controls the maximum number of messages retrieved from the source in a single read operation.
101+
102+
- **Too high**: Should not exceed the input window size, as this would fill the entire buffer in one read cycle.
103+
- **Too low**: Results in underutilizing the buffer capacity, requiring more read cycles to process the same volume.
104+
105+
The default value is `1000`.
106+
107+
## Disk buffering
108+
109+
To prevent message loss during HEC connection outages, consider enabling [Disk Buffering](../configuration.md#configure-your-sc4s-disk-buffer). This feature temporarily stores messages on disk when the destination is unavailable.
110+
111+
## Switch to SC4S Lite
112+
113+
Parsing syslog messages can be a CPU-intensive task. During the parsing process, each syslog message goes through multiple parsing rules until a match is found. Some log messages follow longer parsing paths than others, and some parsers use regular expressions, which can be slow.
114+
115+
If you are familiar with your log sources, consider performing an A/B test and switching to SC4S Lite, which includes only the parsers for the vendors you require. Although artificial performance tests may not fully reflect the impact of this change, you may observe an increase in the capacity of your syslog layer when operating with real-world data.
116+
117+
## Finetune for UDP traffic
118+
119+
### Tested configuration:
120+
- **Loggen** - c5.2xlarge
121+
- **SC4S** (3.29.0) + podman - c5.4xlarge
122+
- **Splunk Cloud** 9.2.2403.105 - 30IDX
123+
124+
!!! note "Note"
125+
Performance may vary depending on a version and specifics of your environment.
126+
127+
| Setup for 67,000 EPS (Events per Second) | % Loss |
128+
|------------------------------------------|--------|
129+
| Default | 77.88 |
130+
| OS Kernel Tuning | 24.38 |
131+
| Increasing the Number of UDP Sockets | 22.95 |
132+
| eBPF | 0 |
133+
134+
### Increase the number of UDP sockets
135+
136+
By default, SC4S uses a single UDP socket per port. Increasing the number of sockets allows traffic to be distributed across multiple CPU threads using the kernel's `SO_REUSEPORT` feature.
137+
138+
**Default behavior (without eBPF)**
139+
140+
The kernel assigns packets to sockets based on a hash of the source IP and port. This means:
141+
142+
- **Consistent routing**: All packets from the same sender go to the same socket, preserving message order.
143+
- **Potential imbalance**: If a few senders generate most of the traffic, their packets may all land on the same socket, leaving other sockets underutilized.
144+
145+
**With eBPF enabled**
146+
147+
eBPF provides true per-packet load balancing, where each packet is randomly distributed across all sockets regardless of sender. This results in:
148+
149+
- **Even workload**: Traffic is distributed more uniformly across CPU threads.
150+
- **No ordering guarantee**: Packets from the same sender may be processed out of order.
151+
152+
**Configuration**
153+
154+
Add the following to `/opt/sc4s/env_file`:
155+
156+
```bash
157+
SC4S_SOURCE_LISTEN_UDP_SOCKETS=32
158+
```
159+
160+
Set this value based on the number of CPU cores available. Start with a value equal to your 4 x core count and adjust based on performance testing. Restart SC4S for the changes to take effect.
161+
162+
### Enable eBPF
163+
164+
Find more in the [About eBPF](../../configuration/#about-ebpf) section.
165+
166+
1. Verify that your host supports eBPF.
167+
2. Ensure your container is running in privileged mode.
168+
3. Update the configuration in `/opt/sc4s/env_file`:
169+
```bash
170+
SC4S_SOURCE_LISTEN_UDP_SOCKETS=32
171+
SC4S_ENABLE_EBPF=yes
172+
SC4S_EBPF_NO_SOCKETS=32
173+
```
174+
4. Restart SC4S for the changes to take effect.
175+
176+
## Finetune for TCP traffic
177+
178+
### Tested configuration:
179+
- **Loggen** - c5.2xlarge
180+
- **SC4S** (3.29.0) + podman - c5.4xlarge
181+
- **Splunk Cloud** 9.2.2403.105 - 30IDX
182+
183+
!!! note "Note"
184+
Performance may vary depending on a version and specifics of your environment.
185+
186+
| Setting | EPS (Events per Second) |
187+
|-------------------------------|-------------------------|
188+
| default | 71,327 |
189+
| SC4S_SOURCE_TCP_SO_RCVBUFF | 99,207 |
190+
| SC4S_ENABLE_PARALLELIZE | 101,700 |
191+
| SC4S_SOURCE_TCP_IW_USE | 115,276 |
192+
193+
### Parallelize TCP processing
194+
1. Update `/opt/sc4s/env_file`:
195+
196+
```
197+
SC4S_ENABLE_PARALLELIZE=yes
198+
SC4S_PARALLELIZE_NO_PARTITION=4
199+
```
200+
201+
2. Restart SC4S for the changes to take effect.
202+
203+
Parallelize distributes messages from a single TCP stream across multiple concurrent threads, which is noticeable in production environments with a single high-volume TCP source.
204+
205+
| SC4S parallelize | Loggen TCP connections | % CPUs used | Average rate (msg/sec) |
206+
|---------------------|--------------------------------|------------|------------------------|
207+
| off | 1 | 9.0 | 14,144.10 |
208+
| off | 10 | 59.3 | 73,743.32 |
209+
| on (10 threads) | 1 | 58.4 | 77,842.18 |

‎docs/architecture/lb/f5.md‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,8 @@ If you're using F5 BIG-IP for syslog ingestion, consider the following:
55
- **UDP limitations**: UDP is a protocol prone to data loss, and load balancers can introduce another point of data loss.
66
- **High Availability (HA) mode**: Running the load balancer without HA makes it a single point of failure.
77

8-
**Note**: Splunk only supports SC4S. If you encounter issues related to the load balancer, please contact F5 support.
8+
!!! note "Note"
9+
Splunk only supports SC4S. If you encounter issues related to the load balancer, please contact F5 support.
910

1011
## Set up your F5 BIG-IP
1112
F5 BIG-IP is available in both hardware and virtual editions. This documentation assumes you already have a functioning F5 instance. If you need help with your F5 instance, see [external resource](https://clouddocs.f5.com/cloud/public/v1/aws/AWS_singleNIC.html).
@@ -139,12 +140,16 @@ Follow the [Check TCP Performance](../performance-tests.md#check-your-tcp-perfor
139140
- **SC4S Instances**: `c5.4xlarge`
140141
- **F5 Instance**: `r5.8xlarge, 32vCPU, 256GiB Memory, 10Gbps Network Performance`
141142

143+
!!! note "Note"
144+
Performance may vary depending on a version and specifics of your environment.
145+
142146
| Receiver | Performance |
143147
|----------------------------|--------------------|
144148
| Single SC4S Server | 69,192.11 msg/sec |
145149
| Load Balancer + 2 Servers | 93,832.33 msg/sec |
146150

147-
**Note:** Load balancer support and fine-tuning are beyond the scope of the SC4S team. For assistance in optimizing the TCP throughput of your load balancer instance, contact the F5 support team.
151+
!!! note "Note"
152+
Load balancer support and fine-tuning are beyond the scope of the SC4S team. For assistance in optimizing the TCP throughput of your load balancer instance, contact the F5 support team.
148153

149154
## Stateless Virtual Server for UDP
150155
UDP syslog traffic is unacknowledged and unidirectional by design. To preserve the source IP for UDP ports, configure [stateless virtual servers](https://my.f5.com/manage/s/article/K13675).
@@ -200,11 +205,15 @@ Follow the [Check UDP Performance](../performance-tests.md#check-your-udp-perfor
200205
- **SC4S Instances**: `c5.4xlarge`
201206
- **F5 Instance**: `r5.8xlarge, 32vCPU, 256GiB Memory, 10Gbps Network Performance`
202207

203-
| Receiver / Drops Rate for EPS (msgs/sec) | 4,500 | 9,000 | 27,000 | 50,000 | 150,000 |
208+
!!! note "Note"
209+
Performance may vary depending on a version and specifics of your environment.
210+
211+
| Receiver / Drops rate for EPS (msgs/sec) | 4,500 | 9,000 | 27,000 | 50,000 | 150,000 |
204212
|------------------------------------------|--------|--------|--------|--------|---------|
205213
| Single SC4S Server | 0% | 0.34% | 58.89% | 78.30% | 92.19% |
206214
| Load Balancer + 2 Servers | 0% | 0% | 15.66% | 54.44% | 84.16% |
207215
| Single Finetuned SC4S Server | 0% | 0% | 0% | 0% | 48.62% |
208216
| Load Balancer + 2 Finetuned Servers | 0% | 0% | 0.002% | 0.05% | 0.03% |
209217

210-
**Note:** Load balancer support and fine-tuning are beyond the scope of the SC4S team. For assistance in minimizing UDP drops on the load balancer side, please contact the F5 support team.
218+
!!! note "Note"
219+
Load balancer support and fine-tuning are beyond the scope of the SC4S team. For assistance in minimizing UDP drops on the load balancer side, please contact the F5 support team.

‎docs/architecture/lb/nginx.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,9 @@ echo "11 hello world" | netcat <LB_IP> 601
146146

147147
3. Run performance tests based on the [Check TCP Performance](performance-tests.md#check-your-tcp-performance) section.
148148

149+
!!! note "Note"
150+
Performance may vary depending on a version and specifics of your environment.
151+
149152
| Receiver | Performance |
150153
|----------------------------|--------------------|
151154
| Single SC4S Server | 71,738.98 msg/sec |
@@ -242,7 +245,10 @@ echo "hello world" > /dev/udp/<LB_IP>/514
242245

243246
2. Run performance tests:
244247

245-
| Receiver / Drops Rate for EPS (msgs/sec) | 4,500 | 9,000 | 27,000 | 50,000 | 150,000 | 300,000 |
248+
!!! note "Note"
249+
Performance may vary depending on a version and specifics of your environment.
250+
251+
| Receiver / Drops rate for EPS (msgs/sec) | 4,500 | 9,000 | 27,000 | 50,000 | 150,000 | 300,000 |
246252
|------------------------------------------|--------|--------|--------|--------|---------|---------|
247253
| Single SC4S Server | 0.33% | 1.24% | 52.31% | 74.71% | -- | -- |
248254
| Load Balancer + 2 Servers | 1% | 1.19% | 6.11% | 47.64% | -- | -- |

‎docs/architecture/performance-tests.md‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,14 +54,16 @@ Here is a reference example of performance testing using our lab configuration o
5454
### Tested configuration
5555
* Loggen (syslog-ng 3.25.1) - m5zn.3xlarge
5656
* SC4S(2.30.0) + podman (4.0.2) - m5zn family
57-
* SC4S_DEST_SPLUNK_HEC_DEFAULT_WORKERS=10 (default)
5857
* Splunk Cloud Noah 8.2.2203.2 - 3SH + 3IDX
5958

6059
### Command
6160
```bash
6261
/opt/syslog-ng/bin/loggen -i --rate=100000 --interval=1800 -P -F --sdata="[test name=\"stress17\"]" -s 800 --active-connections=10 <local_hostmane> <sc4s_external_tcp514_port>
6362
```
6463

64+
!!! note "Note"
65+
Performance may vary depending on a version and specifics of your environment.
66+
6567
| SC4S instance | root networking | slirp4netns networking |
6668
|---------------|---------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
6769
| m5zn.large | average rate = 21109.66 msg/sec, count=38023708, time=1801.25, (average) msg size=800, bandwidth=16491.92 kB/sec | average rate = 20738.39 msg/sec, count=37344765, time=1800.75, (average) msg size=800, bandwidth=16201.87 kB/sec |
@@ -74,6 +76,9 @@ Comparing loggen results can be sufficient for A/B testing, but is not adequate
7476

7577
In the following example, loggen was able to send 4.3 mln messages in one minute; however, Splunk indexers required an additional two minutes to process these messages. During that time, SC4S processed the messages and stored them in a queue while waiting for the HEC endpoint to accept new batches.
7678

79+
!!! note "Note"
80+
Performance may vary depending on a version and specifics of your environment.
81+
7782
| Splunk Indexers | Total Processing Time (4.3 mln messages) | Estimated Max EPS |
7883
|-----------------|------------------------------------------|-------------------|
7984
| 3 | 3 min | 22K |
@@ -100,6 +105,9 @@ Example results:
100105
* default configuration
101106
* Splunk Cloud 9.2.2403.105 - 30IDX
102107

108+
!!! note "Note"
109+
Performance may vary depending on a version and specifics of your environment.
110+
103111
| Metric | Default SC4S | Finetuned SC4S |
104112
|--------------|---------------------|---------------------|
105113
| Average Rate | 72,153.75 msg/sec | 115,276.92 msg/sec |
@@ -116,7 +124,10 @@ Over a span of 60 seconds, loggen will attempt to generate 20,000 logs per secon
116124

117125
After running the command, count the number of events that reached Splunk. Since UDP is prone to data loss, messages can be lost anywhere along the path.
118126

119-
| Receiver / Drops Rate for EPS (msgs/sec) | 4,500 | 9,000 | 27,000 | 50,000 | 150,000 |
127+
!!! note "Note"
128+
Performance may vary depending on a version and specifics of your environment.
129+
130+
| Receiver / Drops rate for EPS (msgs/sec) | 4,500 | 9,000 | 27,000 | 50,000 | 150,000 |
120131
|------------------------------------------|--------|--------|--------|--------|---------|
121132
| Default SC4S | 0.33% | 1.24% | 52.31% | 74.71% | -- |
122133
| Finetuned SC4S | 0% | 0% | 0% | 0% | 47.37% |

‎docs/architecture/tcp-optimization.md‎

Lines changed: 0 additions & 71 deletions
This file was deleted.

0 commit comments

Comments
 (0)