feat: skill parser-creator

Author: Kawron

.cursor/skills/parser-creator/SKILL.md

@@ -0,0 +1,223 @@
+---
+name: parser-creator
+description: Creates SC4S syslog-ng parsers. Use when the user wants to create a new parser, add support for a new log source or vendor, or says "create parser", "add parser", "new log source", or "new vendor support".
+---
+
+# Parser Creator
+
+You can use only command to run the unit tests `poetry run pytest test-name -v -s --tb=short -n=0`
+
+## Goal
+
+Create a new SC4S parser and test coverage for a vendor/product pair in both:
+
+- main package (`package/etc/conf.d/conflib`)
+- lite package (`package/lite/etc/addons`)
+
+## Prerequites
+
+Collect this information from the user before you start:
+
+1. `vendor`: vendor name (lowercase, for example `acme`)
+2. `product`: product name (lowercase, for example `firewall`)
+3. `sourcetype`: target Splunk sourcetype using `vendor:product` (for example `acme:firewall`)
+4. `index`: target Splunk index (for example `netfw`)
+5. `sample logs`: one or more raw syslog messages
+
+If any item is missing, ask for it before proceeding. 
+
+## Workflow
+
+### Step 1 - Identify message format
+
+Examine the sample logs and identify the Syslog format.
+
+1. RFC3164: `<PRI>TIMESTAMP HOSTNAME PROGRAM: MESSAGE`
+2. RFC5424: `<PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID SDATA MESSAGE`
+3. CEF: `<PRI>TIMESTAMP HOSTNAME CEF:0|<Device Vendor>|<Device Product>|<Device Version>|<Signature ID>|<Name>|<Severity>|<Extension fields>`
+
+If logs do not match one of these formats, tell the user the format is currently unsupported and stop.
+
+### Step 2 - Create a parser
+
+Start by creating a filter for the log message. Filter has a following structure:
+
+```
+application <filter-name>[<topic>] {
+    filter {
+        <filter_block>
+    };
+    parser { <parser-name>(); };
+};
+```
+
+Filter are grouped into topics. Use one of the following topics:
+
+1. `cef` - for CEF-formatted messages.
+2. `sc4s-syslog-pgm` - matches by program value (`PROGRAM` in RFC3164, `APP-NAME` in RFC5424).
+3. `sc4s-syslog-sdata` - matches by structured data (often a Private Enterprise Number, PEN). If PEN is present, prefer this topic.
+4. `sc4s-syslog` - general filter for RFC3164/RFC5424, usually based on message content.
+5. `sc4s-network-source` - matches by destination port. Use this only when other topics are not viable. Because this requires sending logs to a new port, ask the user for permission first. If the user refuses, stop and explain why parser creation cannot continue.
+
+Next create `block parser`:
+
+```
+block parser <parser-name>() {
+    <parsers and filters blocks>
+    <rewrite block>
+};
+```
+
+If structured data or repeated key/value data exists, include a parser stage (`kv-parser`, `csv-parser`, or `regexp-parser`) before rewrite. Only skip parsing when the message is truly unstructured; if so, explicitly state this in the final response.
+
+There are two rewrite functions. Choose the correct one:
+
+1. `r_set_splunk_dest_default` — sets **all** base Splunk metadata. Every parser MUST call this exactly once as its first rewrite. Always include `index`, `sourcetype`, `vendor`, and `product`. Optionally include `source` and `template`.
+2. `r_set_splunk_dest_update_v2` — **conditionally overrides** specific fields that were already set by `r_set_splunk_dest_default`. Use this ONLY in `if/elif` branches to change a subset of fields (e.g. sourcetype, index) based on message content. Never use it as the first or only rewrite.
+
+`r_set_splunk_dest_default` example (required in every parser):
+
+```
+rewrite {
+    r_set_splunk_dest_default(
+        index('netops')
+        sourcetype('alcatel:switch')
+        vendor("alcatel")
+        product("switch")
+        template('t_hdr_msg')
+    );
+};
+```
+
+`r_set_splunk_dest_update_v2` example (optional, only after default is set):
+
+```
+rewrite {
+    r_set_splunk_dest_update_v2(
+            sourcetype('citrix:netscaler:appfw') condition(message(':(\s+\S+)?\s+APPFW(\s+\S+){3}\s+:'))
+    );
+};
+```
+
+To choose correct template refer to the definitions in file: `t_templates.conf`.
+
+Parser method selection:
+
+Use `kv-parser` when logs contain key/value pairs (`key=value`, quoted values, RFC5424 SDATA blocks).
+   - For RFC5424 SDATA, prefer `template("${SDATA}")`.
+   - Use a scoped prefix like `.values.sdata.`.
+
+Example:
+
+```
+block parser app-syslog-vendor_product() {
+    channel {
+        parser {
+            kv-parser(prefix(".values.") template("$(template t_hdr_msg)"));
+        };
+        # Optional: validate parsing succeeded
+        filter {
+            "${.values.some_required_field}" ne ""
+        };
+        rewrite {
+            r_set_splunk_dest_default(
+                index('netfw')
+                sourcetype('vendor:product')
+                vendor("vendor")
+                product("product")
+                template('t_kv_values')
+            );
+        };
+    };
+};
+```
+
+Use `csv-parser` when logs are consistently delimited and have stable column order.
+
+Example:
+
+```
+parser {
+    csv-parser(
+        columns("col1","col2","col3","col4")
+        prefix(".values.")
+        delimiters(',')
+        quote-pairs('""')
+        flags(escape-double-char)
+    );
+};
+```
+
+Use `regexp-parser` when logs are structured but not key/value or delimited.
+Combine methods when logs have multiple variants.
+
+Example:
+
+```
+parser {
+    regexp-parser(
+        template("${MESSAGE}")
+        patterns("^(?<field1>\\d+) (?<field2>[^ ]+) (?<field3>.*)")
+        prefix(".parsed.")
+    );
+};
+```
+
+You can combine all methods and use conditional branches to parse different message variants:
+
+```
+block parser app-syslog-vendor_product() {
+    channel {
+        rewrite {
+            r_set_splunk_dest_default(
+                index("netops")
+                sourcetype('vendor:log')
+                vendor("vendor")
+                product('product')
+                template('t_msg_only')
+            );
+        };
+
+        if (message(',TRAFFIC,' type(string) flags(substring))) {
+            parser { csv-parser(columns(...) prefix(".values.") delimiters(',')); };
+            rewrite {
+                r_set_splunk_dest_update_v2(
+                    index('netfw')
+                    class('traffic')
+                    sourcetype('vendor:traffic')
+                );
+            };
+        } elif (message(',SYSTEM,' type(string) flags(substring))) {
+            parser { csv-parser(columns(...) prefix(".values.") delimiters(',')); };
+            rewrite {
+                r_set_splunk_dest_update_v2(
+                    index('netops')
+                    class('system')
+                    sourcetype('vendor:system')
+                );
+            };
+        } else { };
+    };
+};
+```
+
+### Step 4 - Create unit test
+
+Create a unit test for the new parser. Testing instructions: [testing-parsers](./references/testing-parsers.md).
+
+### Step 5 - Run parser tests
+
+Run the new test using `poetry run pytest test-name -v -s --tb=short -n=0` command and verify the parser works correctly.
+
+## Completion Checklist
+
+Before finishing, confirm all items:
+
+- Parser/filter created for main package.
+- Parser/filter created for lite package.
+- Parser includes field extraction (`kv-parser`, `csv-parser`, and/or `regexp-parser`) when sample logs are parseable.
+- Lite vendor metadata exists (`addon_metadata.yaml`) when required.
+- `package/lite/etc/config.yaml` updated for new lite vendor addon.
+- Unit tests created and passing for the new parser.
+- User informed about any constraints (for example, unsupported format or required network-source port changes).
+- The parser files have only one `block parser` definition and only one `application` deffinition.

.cursor/skills/parser-creator/references/cef-filter.md

@@ -0,0 +1,28 @@
+Here is an example of a CEF filter (replace values inside `<>` with your own):
+
+```
+application app-cef-<device_vendor>_<device_product>[cef] {
+    filter{
+        match("<device_vendor>" value(".metadata.cef.device_vendor"))
+        and match("<device_product>" value(".metadata.cef.device_product"));
+    };
+    parser { app-cef-<device_vendor>_<device_product>(); };
+};
+```
+
+
+<!-- DEPRECATED:
+
+block parser app-cef-<device_vendor>_<device_product>() {
+    channel {
+        rewrite {
+            r_set_splunk_dest_default(
+                index('<index>'),
+                source('<device_vendor>:<device_product>'),
+                sourcetype('<device_vendor>:<device_product>:cef')
+                vendor('<device_vendor>')
+                product('<device_product>')
+            );
+        };
+    };
+}; -->

.cursor/skills/parser-creator/references/netsource-filter.md

@@ -0,0 +1,19 @@
+Here is an example of a `netsource` filter:
+
+```
+application app-netsource-barracuda_syslog[sc4s-network-source] {
+	filter {
+        not filter(f_is_source_identified)
+        and (
+                (
+                    match("barracuda", value('.netsource.sc4s_vendor'), type(string)) 
+                    and match("syslog", value('.netsource.sc4s_product'), type(string)) 
+                )
+                or (tags("ns_vendor:barracuda") and tags("ns_product:syslog"))
+                or tags(".source.s_BARRACUDA_SYSLOG")
+            )
+        ;
+    };	
+    parser { app-netsource-barracuda_syslog(); };
+};
+```

.cursor/skills/parser-creator/references/pgm-filter.md

@@ -0,0 +1,11 @@
+Here is an example of a `pgm` filter (replace values inside `<>` with your own):
+
+```
+application app-syslog-<vendor_name>_<product_name>[sc4s-syslog-pgm] {
+	filter {
+        program('<program>' type(string) flags(prefix));
+    };	
+    parser { <parser-name>(); };
+};
+```
+

.cursor/skills/parser-creator/references/sdata-filter.md

@@ -0,0 +1,10 @@
+Example of an `sdata` filter:
+
+```
+application app-syslog-<vendor_name>_<product_name>[sc4s-syslog-sdata] {
+    filter {
+        match('@<PEN>' value("SDATA"));
+    };
+    parser { <parser-name>(); };
+};
+```

.cursor/skills/parser-creator/references/syslog-filter.md

@@ -0,0 +1,10 @@
+Example of an `sc4s-syslog` topic filter:
+
+```
+application app-syslog-<vendor_name>_<vendor_product>[sc4s-syslog] {
+	filter {
+        message('Carbon Black App Control event:  '  type(string)  flags(prefix));
+    };	
+    parser { <parser-name>(); };
+};
+```

.cursor/skills/parser-creator/references/testing-parsers.md

@@ -0,0 +1,73 @@
+Below is a template for the unit test file:
+
+```
+# Copyright <current-year> Splunk, Inc.
+#
+# Use of this source code is governed by a BSD-2-clause-style
+# license that can be found in the LICENSE-BSD2 file or at
+# https://opensource.org/licenses/BSD-2-Clause
+import datetime
+import pytest
+
+from jinja2 import Environment, select_autoescape
+
+from .sendmessage import sendsingle
+from .splunkutils import splunk_single
+from .timeutils import time_operations
+
+env = Environment(autoescape=select_autoescape(default_for_string=False))
+
+
+@pytest.mark.addons("<addon-name>")
+def test_palo_alto_test_os_cef(
+    record_property, setup_splunk, setup_sc4s, get_host_key
+):
+    host = get_host_key
+    mt = env.from_string(
+        "{{ mark }}{{ bsd }} {{ host }} <test-message>"
+    )
+
+    dt = datetime.datetime.now(datetime.timezone.utc)
+    _, bsd, _, _, _, _, epoch = time_operations(dt)
+    message = mt.render(mark="<134>", bsd=bsd, host=host)
+
+    # Tune time functions
+    epoch = epoch[:-7]
+    sendsingle(message, setup_sc4s[0], setup_sc4s[1][514])
+    st = env.from_string(
+        f'search _time={epoch} index=netfw host="{host}" sourcetype="<sourcetype>"'
+    )
+    search = st.render(epoch=epoch)
+
+    result_count, _ = splunk_single(setup_splunk, search)
+
+    record_property("resultCount", result_count)
+    record_property("message", message)
+
+    assert result_count == 1
+```
+
+When creating a unit test, pay close attention to time handling. You can use the `.timeutils` module to generate timestamps. The timestamp format you generate should match the original event format. In most cases, start by getting the current UTC time:
+
+`dt = datetime.datetime.now(datetime.timezone.utc)`
+
+Then use `time_operations`, which returns:
+- iso - ISO 8601 / RFC5424-style timestamp e.g. 2026-03-09T15:04:05.123456+01:00
+- bsd - BSD syslog / RFC3164-style timestamp ("%b %d %H:%M:%S") e.g. Mar 09 15:04:05
+- time - time of day with microseconds only ("%H:%M:%S.%f") e.g. 15:04:05.123456
+- date - calendar date ("%Y-%m-%d") e.g. 2026-03-09
+- tzoffset - timezone offset from local tz e.g. +0100
+- tzname - timezone name e.g. UTC, CET, PDT
+- epoch - epoch seconds plus microseconds as a string (`%s.%f`), for example `1741532645.123456`. It is usually trimmed with `[:-7]` for seconds only and `[:-3]` for milliseconds.
+
+When creating a message template, make sure the format matches the original message itself. In some cases, the timestamp is not part of the header. For example, in this CEF message:
+
+```
+mt = env.from_string(
+    "{{ mark }} CEF:0|A10|vThunder|4.1.4-GR1-P12|WAF|session-id|2|rt={{ bsd }} src=1.1.1.1 spt=34860 dst=1.1.1.1 dpt=80 dhost=test.host.local cs1=uiext_sec_waf cs2=1 act=learn cs3=learn app=HTTP requestMethod=GET cn1=0 request=/sales/ msg=New session created: Id\=1\n"
+)
+```
+
+the timestamp is part of the `rt` field.
+
+Always use the full event in the test; do not truncate it. If the user provides multiple events (fewer than 10), use all of them in the tests (parameterize the test).

.gitignore

@@ -388,3 +388,7 @@ replay
 .addon/
 Pipfile
 event.txt
+
+# vscode
+
+.vscode/launch.json