Skip to content

Commit a2cb0b2

Browse files
committed
docs: suggested changes 3
1 parent 0b69b78 commit a2cb0b2

6 files changed

Lines changed: 30 additions & 27 deletions

File tree

docs/creating_parsers/index.md

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,12 @@
33
!!! note "Prerequisites"
44
Before reading this section, make sure you are familiar with [Sources](../sources/index.md).
55

6-
This and subsequent sections describe how to create new parsers. SC4S parsers perform operations that would normally be performed during index time, including line-breaking, source and sourcetype setting. You can write your own parser if the parsers available in the SC4S package do not meet your needs or you want to add support for new sourcetype.
6+
This and subsequent sections describe how to create new parsers. SC4S parsers perform operations that would normally be performed during index time, including line-breaking, source and sourcetype setting. You can write your own parser if the parsers available in the SC4S package do not meet your needs or you want to add support for a new sourcetype.
77

88
## Before you start
99

1010
* Make sure you have read our [contribution standards](../CONTRIBUTING.md).
1111
* Obtain a raw log message that you want to parse. If you do not know how to do it, refer to [Obtain raw message events](../troubleshooting/troubleshoot_resources.md#obtain-raw-message-events).
12-
* Create a new branch in the repository where you will apply your changes.
1312
* Prepare your testing environment. With Python>=3.11.0:
1413

1514
```
@@ -35,15 +34,15 @@ The SC4S parser consists of `application` and `block parser` blocks. The `applic
3534
If you find a similar parser in SC4S, you can use it as a reference. In the parser, make sure you assign the proper sourcetype, index, vendor, product, and template. The template shows how your message should be parsed before sending them to Splunk.
3635

3736

38-
The application filter will match all messages that start with the string `Carbon Black App Control event:`, and those events will be parsed by `block parser app-syslog-vmware_cb-protect()`. This parser then will route the message to index: `epintel`, set the sourcetype, source, vendor and product fields, and use speciefed template.
37+
The application filter will match all messages that start with the string `Carbon Black App Control event:`, and those events will be parsed by `block parser app-syslog-vmware_cb-protect()`. This parser then will route the message to index: `epintel`, set the sourcetype, source, vendor and product fields, and use the specified template.
3938

4039
![](../resources/images/parser_dev_basic_output.png)
4140

4241
To learn more about creating filters and parse blocks see pages: [Filter Messages](filter_message.md) and [Parse Messages](parse_message.md).
4342

4443
### Adding parser to SC4S Lite package
4544

46-
For SC4S lite, parsers are grouped into `addons`. Create a folder (if it doesn't already exist) in `package/lite/etc/addons` with the name of vendor. In this folder also create an `addon_metadata.yaml` file with vendor name:
45+
For SC4S lite, parsers are grouped into `addons`. Create a folder (if it does not already exist) in `package/lite/etc/addons` with the name of vendor. In this folder, also create an `addon_metadata.yaml` file with vendor name:
4746

4847
```
4948
---

docs/creating_parsers/parse_message.md

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ rewrite {
3939
};
4040
```
4141

42-
**`r_set_splunk_dest_update_v2`** — overrides specific fields already set by `r_set_splunk_dest_default`. It accepts `index`, `source`, `sourcetype`, and `template` options. You can also use the `condition` option for a conditional expression.
42+
**`r_set_splunk_dest_update_v2`** — overrides specific fields already set by `r_set_splunk_dest_default`. It accepts `index`, `source`, `sourcetype`, `class`, and `template` options. You can also use the `condition` option for a conditional expression.
4343

4444
```
4545
rewrite {
@@ -292,7 +292,13 @@ block parser app-syslog-vendor_product() {
292292
};
293293
294294
if (message(',TRAFFIC,' type(string) flags(substring))) {
295-
parser { csv-parser(columns(...) prefix(".values.") delimiters(',')); };
295+
parser {
296+
csv-parser(
297+
delimiters(chars('') strings('|'))
298+
columns('version', 'device_vendor', 'device_product', 'device_version', 'device_event_class', 'name', 'severity', 'ext')
299+
prefix('.metadata.cef.')
300+
flags(greedy));
301+
};
296302
rewrite {
297303
r_set_splunk_dest_update_v2(
298304
index('netfw')

docs/creating_parsers/unit_tests.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ poetry run pytest -v --tb=long \
99
--splunk_type=external \
1010
--splunk_hec_token=<HEC_TOKEN> \
1111
--splunk_host=<HEC_ENDPOINT> \
12-
--sc4s_type=external
12+
--sc4s_type=external \
1313
--sc4s_host=<SC4S_IP> \
1414
--splunk_user=<SPLUNK_USER> \
1515
--splunk_password=<SPLUNK_PASSWORD> \
@@ -43,7 +43,7 @@ env = Environment(autoescape=select_autoescape(default_for_string=False))
4343
4444
4545
@pytest.mark.addons("<addon-name>")
46-
def test_palo_alto_test_os_cef(
46+
def test_<vendor_name>_<product_name>(
4747
record_property, setup_splunk, setup_sc4s, get_host_key
4848
):
4949
host = get_host_key

docs/sources/index.md

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,16 +20,22 @@ New supported sources are added regularly. Please submit an [issue](https://gith
2020

2121
Many sources can be self supported. While we encourage sharing new sources via the github project to promote consistency and develop best-practices there is no requirement to engage in the community.
2222

23-
Sources that are *compliant* with RFC 5424, RFC 5425, RFC 5426, or RFC 6587 can be onboarded as [simple sources](simple.md).
23+
* Sources that are compliant with RFC 5424, RFC 5425, RFC 5426, or RFC 6587 can be onboarded as [simple sources](simple.md).
24+
* Sources compatible with RFC 3164 can also be onboarded as [simple sources](simple.md). Note that incorrect use of the syslog version, or “creative” formats in the timestamp or other fields may prevent this method of onboarding and will require writing custom parsers.
25+
26+
Other popular log formats that are supported by SC4S are:
27+
28+
* Common Event Format [CEF](base/cef.md), also known as ArcSight format.
29+
* Log Extended Format [LEEF](base/leef.md).
2430

2531
## Common Patterns
2632

2733
This section covers the most basic and common patterns for onboarding data with user-made parsers. If you want to read more, see [Create a Parser](../creating_parsers/index.md).
2834

2935
### Almost Syslog
3036

31-
Sources sending legacy non conformant 3164 like streams can be assisted by the creation of an "Almost Syslog" Parser. In an such a parser the goal is to process the syslog header allowing other parsers
32-
to correctly parse and handle the event. The following example is take from a currently supported format where the source product used epoch in the time stamp field.
37+
Sources sending legacy non conformant 3164 like streams can be assisted by the creation of an "Almost Syslog" Parser. In such a parser the goal is to process the syslog header allowing other parsers
38+
to correctly parse and handle the event. The following example is taken from a currently supported format where the source product used epoch in the timestamp field.
3339

3440
```c
3541
#Example event

docs/sources/vendor/Cisco/cisco_meraki.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
## Meraki (MR, MS, MX)
22

33
## Key facts
4-
* Cisco Meraki messages are not distinctive, which means that it's impossible to parse the sourcetype based on the log message.
5-
* Because of the above you should either configure known Cisco Meraki hosts in SC4S, or open unique ports for Cisco Meraki devices.
4+
* Cisco Meraki messages are not distinctive, which means that it is impossible to parse the sourcetype based on the log message.
5+
* Because of the above, you should either configure known Cisco Meraki hosts in SC4S, or open unique ports for Cisco Meraki devices.
66
* [Splunk Add-on for Cisco Meraki 2.1.0](https://splunkbase.splunk.com/app/5580) does not support syslog. Use [TA-meraki](https://splunkbase.splunk.com/app/3018) instead. `TA-meraki 1.1.5` requires sourcetype `meraki`.
77

88

docs/troubleshooting/troubleshoot_resources.md

Lines changed: 6 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -44,17 +44,9 @@ needed for playback when testing. The community supporting SC4S will always firs
4444

4545
Here are some options for obtaining raw logs for one or more sourcetypes:
4646

47-
* Run `tcpdump` on the collection interface and display the results in ASCII.
48-
49-
``` bash
50-
tcpdump -n -s 0 -S -i any -v port 8088
51-
52-
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
53-
09:54:26.051644 IP (tos 0x0, ttl 64, id 29465, offset 0, flags [DF], proto UDP (17), length 466)
54-
10.202.22.239.41151 > 10.202.33.242.syslog: SYSLOG, length: 438
55-
Facility local0 (16), Severity info (6)
56-
Msg: 2022-04-28T16:16:15.466731-04:00 NTNX-21SM6M510425-B-CVM audispd[32075]: node=ntnx-21sm6m510425-b-cvm type=SYSCALL msg=audit(1651176975.464:2828209): arch=c000003e syscall=2 success=yes exit=6 a0=7f2955ac932e a1=2 a2=3e8 a3=3 items=1 ppid=29680 pid=4684 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=964698 comm=“sshd” exe=“/usr/sbin/sshd” subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=“logins”\0x0a
57-
47+
* Run `tcpdump` on the collection interface and display the results in ASCII. You will see events similar to the following buried in the packet contents:
48+
```
49+
<165>1 2007-02-15T09:17:15.719Z router1 mgd 3046 UI_DBASE_LOGOUT_EVENT [junos@2636.1.1.1.2.18 username="user"] User 'user' exiting configuration mode
5850
```
5951

6052
* Obtain a raw log message using Wireshark.
@@ -67,10 +59,10 @@ You can also read the logs using Wireshark from the .pcap file. From Wireshark g
6759

6860
* For most other sourcetypes, the `RAWMSG` is not displayed, but can be
6961
viewed by changing the output template to one of the JSON variants, including t_JSON_3164 or t_JSON_5424, depending on RFC message type. See
70-
[SC4S metadata configuration](https://splunk-connect-for-syslog.readthedocs.io/en/develop/configuration/#sc4s-metadata-configuration) for
62+
[SC4S metadata configuration](../configuration.md#sc4s-metadata-configuration) for
7163
more details.
7264

73-
* In order to send `RAWMSG` to Splunk regardless the sourcetype you can also temporarily place the following final filter in the local parser directory:
65+
* In order to send `RAWMSG` to Splunk regardless of the sourcetype you can also temporarily place the following final filter in the local parser directory:
7466
```conf
7567
block parser app-finalfilter-fetch-rawmsg() {
7668
channel {
@@ -164,7 +156,7 @@ application app-dest-rewrite-device-d_fmt_hec_default[sc4s-postfilter] {
164156
```
165157
Note that filter match statement should be aligned to your data
166158

167-
The parser accepts time zone in formats: "America/New York" or "EST5EDT", but not short in form such as "EST".
159+
The parser accepts time zone in formats: "America/New York" or "EST5EDT", but not in short form such as "EST".
168160

169161
## Issue: CyberArk log problems
170162
When data is received on the indexers, all events are merged together into one event. Check the following link for CyberArk configuration information:

0 commit comments

Comments
 (0)