Repository navigation
refactor: convert the analyst1 app to the splunk soar sdk - #3
athreya-splunk merged 32 commits into
Conversation
Port the SDK app from the old repo's feature/sdk-conversion branch (sdk_app/, commit 1045422) into the fork using the standard SDK connector layout (src/, templates/, pyproject.toml, uv.lock). - remove classic scaffold (analyst1_connector.py, analyst1.json, analyst1_consts.py, root __init__.py) - rename logos to logo_analyst1.svg / logo_analyst1_dark.svg - set manifest identity: name Analyst1, publisher Analyst1, appid preserved (8ee822f1-a285-44f4-b8e7-303245811a32), app_version 2.0.0, min_phantom_version 7.0.0, package_name phantom_analyst1, license Copyright (c) Analyst1 - bump splunk-soar-sdk 3.2.3 -> >=3.25.3 with mechanical API fixes: asset.auth_state replaces actions_manager.auth_state and vault.get_attachment replaces vault.get_vault_file_info
drop call-publish-template.yml, call-push-template.yml and setup-from-template.yml so the workflow set matches current sdk-format connectors (virustotalv3, splunk), which carry only call-publish.yml
set target-version to py313 to match requires-python (mirrors the splunk sdk connector); delete src/views.py, which nothing imports — app.py registers its own view handler
output of splunk's dev-cicd-tools and lint hooks: copyright headers and year bump, generated readme docs and notice file, pyupgrade rewrites for py313, djlint template formatting, and an unreleased release note for the sdk conversion
- model the classic nested action_result.data.* datapath contract for the nine lookup actions as typed ActionOutput sub-models, replacing the raw_data json-blob flattening - preserve classic runtime decorations: base_url, actor links, and friendly enrichment result names; keep enrichmentResults result as a json string (documented delta, the sdk cannot type an arbitrary object) - restore classic summary keys: lookup summary.id, upload summary.uuid, evidence-status summary.message/evidence_id, and get-evidence paging keys - restore the classic not-found lookup shape: no data records, empty summary - surface get_evidence's evidence type filter as the classic type parameter and restore classic page semantics (absent means multi-page, must be >= 1) - make upload_evidence_file's source_id required again and skip falsy form fields like classic - emit get_evidence records as untyped passthrough rows like classic - render the custom view from the typed models instead of re-parsed raw_data
…rity - default a missing enrichment result type to an empty friendly name like classic instead of raising KeyError - default the sort keys the display template uses (name on originatingIps, subjects and fileNames elements; value on ports elements) so Jinja's sort filter cannot raise on elements missing the key - fail the lookup ip action on an invalid IP address like classic instead of silently proceeding as type ip
check a batch of indicator values against GET /batchCheck (type
auto-detected), parsing the {"results": [...]} envelope. enforce the
6000-char url-length cap on the normalized csv input. extract the 1_1
links object->list normalization into a module-level _links_to_list and
add the shared EnumDto/AkaDto output models.
fetch indicators, actors, and malware families by their analyst1
numeric id. indicator-by-id strips hash-id type suffixes (14131-md5)
and reuses the lookup output contract, custom view, and runtime
decorations via an extracted _decorate_indicator helper. actor and
malware map the spec resources with {id, name, classification} triples
for country/sponsor/primaryMotivation and category/stage.
add get sensors (paginated like get evidence, with the live _links -> links row rename), get sensor taskings and get sensor diff (extended 200s timeout for parity with the xsoar integration), and get sensor config (raw text config delivered to the vault). a sensor with no finalized config version answers the diff with http 500 "no finalized versions found", now recognized via a structured status attribute on Analyst1APIError and treated as an empty result rather than a failure.
live verification against the partner instance shows both the 1_1 (oauth)
and 1_0 (basic) apis serialize links as a rel-keyed object on
/indicator/match, /indicator/{id}, and /actor/{id}; the list-of-{rel,href}
form exists only in the classic manifest contract and xsoar's fixtures.
the coercion to list form is unchanged and still required to make the
declared links.* datapaths resolve.
…tors entry the contributors injection rides on the sdk's app_meta_info setattr propagation (no first-class mechanism in sdk 3.25.3); add a round-trip test so an sdk upgrade that drops the path fails loudly.
svg logos serve correctly but the soar 8.5 app header falls back to the default splunk logo instead of rendering them; the classic app and other working sdk apps on the instance ship png. restore the classic analyst1.png / analyst1_dark.png assets and point the app metadata and the lookup view at them.
the analyst1 auth tier answers bad basic-auth credentials with a text/html tomcat 401 page; surface its readable text in the action error message instead of the raw html blob, restoring classic 1.2.1 _process_html_response behavior (minus its brace-escaping, which only guarded classic phantom's format pipeline). applies to both _make_request and get_sensor_config_text; non-html bodies still pass through verbatim. declares beautifulsoup4 as a direct dependency (already bundled as a transitive of splunk-soar-sdk).
…metadata replace splunk inc. with analyst1 in the apache 2.0 headers of all source, test, and template files, the license appendix boilerplate, and the pyproject license field. per splunk's LEGAL.md, contributors retain copyright on contributions while licensing them under apache 2.0. normalize the license string to years-then-name so the copyright hook maintains it. regenerate NOTICE so its header picks up the new license string. leave the generated readme footer and the splunk-soar-sdk attribution block in NOTICE, which correctly credit splunk.
add custom contains (analyst1 indicator id, analyst1 actor id, analyst1 malware id, analyst1 sensor id) to the new actions' id parameters and output datapaths so the soar ui offers contextual pivots between them, following the evidence upload key precedent. batch check values gets broad indicator contains (ip, domain, url, hash, email) and lookup ipv6 gains ipv6; id-bearing sub-models subclass the shared ClassifiedIdName so campaigns, targets, cves, akas and other plain references stay untagged. classic lookup parameters are unchanged (verified manifest-identical). manifest guard tests pin the contains contract.
marketing-supplied 3:1 svg lockups (light and dark theme variants) replace the interim classic pngs, whose 7.4:1 strip rendered compressed in the playbook picker and whose png format failed splunk's svg-only logo check. the svgs pass the light_and_dark_theme_logos static test.
every dark-theme logo across splunk-soar-connectors is monochrome white (surveyed 16 incl. slack, fortinet, sentinelone, check point). recolor the gold fills white and drop the black shield underlay so the falcon renders as a transparent knockout.
the contributor wire names (contributorOrg, iso country/region codes, and the misspelled contibutorConsent) were defined in both the classic and sdk field maps but never exposed as action parameters, so no version of the app ever sent them; the api spec documents no such fields on the evidence upload endpoint.
per analyst1 product direction, 2.x is reserved for the upcoming analyst1 v2 api; the sdk conversion ships as 1.3.0. release notes now state the splunk soar 7.0 minimum explicitly since the version number alone no longer signals the platform floor.
replaces the letterboxed dark variant with marketing's full-bleed 1560x204 export so the artwork renders larger in the soar ui.
the openapi spec describes http 204 from /evidence/uploadStatus/{uuid} as
"an error occurred processing the evidence. no evidence record available."
previously the empty 204 body collapsed to the same null message/id as a
200 with a null id (ingest still processing), so a permanently failed
ingest looked like it was processing forever. the client now surfaces the
status code for this endpoint and the action reports "evidence processing
failed (no evidence record available)" on 204. the 204 semantics are
spec-only (not yet observed live); confirmation requested from analyst1
engineering. 200 and 404 behavior is unchanged.
get_evidence_upload_status bypasses _make_request to expose the http status code, but dropped _make_request's refresh-and-retry-once on a server-rejected oauth token. check evidence status is a polled endpoint, so playbook loops routinely cross token-expiry windows; without the retry, a stale token failed the action until the local expiry passed. the method now clears the cached token, forces a refresh, and retries exactly once, matching _make_request.
- fix widgets-pane blank render: template license headers become jinja
comments so the widget div is the first parsed node of the response
(the pane injects via $(data)[0] and silently drops anything else)
- fix stored xss: remove |safe on api-sourced description and enrichment
text; preserve line breaks with white-space: pre-line instead
- fix widget logo: stop overriding the platform-supplied themed
title_logo paths (404 + missing dark logo) and drop title text that
painted over the centered logo
- redesign the indicator view: verdict badges, sectioned layout, shared
macros and styles (a1_macros.html), theme-safe colors, noopener links
- add contextual views for the ten actions that rendered blank default
tables: batch check, actor/malware by id, evidence upload/status/list,
sensors, sensor taskings/config/diff (tabular views for lists, detail
views for single entities)
- deep-link entities to the asset's analyst1 portal: indicators, actors,
malware, sensors (/sensors/{id}), evidence (/files/{id}); link bases
derive from api self-links or the asset base_url injected at action
time, never hardcoded, with an http(s) scheme guard
- unwrap live classified {name/date} objects in evidence rows and show
the reported date column
- keep monospace blocks theme-colored (the platform styles bare pre
near-black on both themes)
- add a view rendering test suite covering the injection contract,
escaping regressions, empty states, and portal links
|
Thank you for your submission! We will take a look as soon as we can. |
|
Following up now that phantomcyber/dev-cicd-tools#233 has merged: pre-commit is passing. Two notes on the remaining check failures, which both appear to be on the CI side rather than in the app:
Happy to retrigger or adjust anything on our side if needed. |
|
Hi @mvforgione - I had a quick look at the Pr and I would like to highlight following issues that we might want to resolve before we can merge the PR. Here's the generated summary - |
address first review round on the upstream pr: - url-encode user-supplied indicator id and evidence uuid before inserting them into api paths - strip lookup values and fail locally on blank input before any api call, matching the classic app's required-value semantics - give get_sensor_config_text the same oauth 401 clear-refresh-retry behavior as evidence upload status polling
|
Thanks for the review @athreya-splunk. All three items are addressed in 46b5d93:
Each fix has test coverage (encoded-path assertions, blank rejection with no outbound request, and single-retry behavior on 401). Happy to adjust anything further. One note on the CI run for this push: every job now fails at the checkout step with "Refusing to check out fork pull request code from a 'pull_request_target' workflow", which looks like a recent actions/checkout default change affecting fork PRs org-wide rather than anything in this branch. The previous run on July 20 checked out and ran normally. |
existing assets must trust the api server certificate chain or explicitly disable verification, matching the platform security baseline for connector asset configuration
|
@mvforgione thanks for fixing. Could you also please take a look at this PR and add them to this sdkfied code? |
|
Thanks @athreya-splunk. Both security items from #4 are covered on this branch:
The LICENSE, NOTICE, and dependency metadata changes in #4 are specific to the classic format; the SDK build generates those artifacts. One sequencing note: #4 and this PR modify the same files, so whichever merges second will need a rebase. If it is simplest, this PR can carry the security changes and #4 can be closed, but happy to rebase either way. |
202b40c
into
splunk-soar-connectors:main
|
🎉 This PR is included in version 1.0.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
|
I will create an internal request to release this app in splunkbase. I'll update here once it's released. |
|
@athreya-splunk one flag before the Splunkbase release request goes out: this release was tagged 1.0.0, but Splunkbase already has Analyst1 at 1.2.1 under this app id, released before the app moved to GitHub. A 1.0.0 release would register as a downgrade and break the upgrade path for existing installs. The PR set the version to 1.3.0 for that reason; semantic-release recomputed it to 1.0.0 because the repo has no baseline tag from the pre-GitHub releases. Could the release go out as 1.3.0? One option is to baseline the repo with a 1.2.1 tag on the pre-merge history and re-run the release so the commits compute to 1.3.0. Happy to help however is useful. |
@mvforgione - I thought this was the first release of the app. Missed the description. I didn't see any past releases of this app. Could you please share splunkbase link so that I can take a look? I only see a splunk app https://splunkbase.splunk.com/app/6816 and not SOAR integration app. |
|
You are right, and I should correct my earlier comment. This connector was never published on Splunkbase, so there is no link to share. App 6816 is a different product, our add-on for Splunk Enterprise (TA-Analyst1). Analyst1 distributed the SOAR connector directly to customers, and 1.2.1 is the version they are running today. The 1.3.0 request is about those existing installs rather than a Splunkbase listing. Customer SOAR instances have the classic app installed under this same app id, which we kept so the SDK app upgrades those installs in place. Publishing as 1.0.0 would appear as a downgrade to them. The generated release notes for this release also state that version 1.3.0 replaces the classic Analyst1 app (v1.2.1), so the 1.0.0 tag does not match its own notes. Would it be possible to release as 1.3.0? One approach is to tag the pre-merge commit as 1.2.1 so semantic-release computes 1.3.0 from the merged commits. Happy to help however is useful. |
|
I’ll need to check internally whether we can publish the app as v1.3.0. |
|
Thanks, the rename works for us. "Analyst1 for Splunk SOAR" would follow the convention other connectors use when the vendor already has a Splunk Enterprise app, for example GreyNoise for SOAR, TrackMe for Splunk SOAR, Dataminr Pulse for Splunk SOAR, and Censys for Splunk SOAR. Happy to use that, or a different name if you prefer one. On the app UUID, could you confirm the upgrade behavior for us? Our understanding is that if the UUID stays the same, a customer running the classic app gets an in-place upgrade with their assets and playbooks intact, and that if the UUID changes they get a parallel install instead, leaving the old app on the instance and requiring assets to be recreated with their credentials and playbooks repointed. If that is right, keeping the UUID would spare existing customers that migration, and a rename alone should be enough to resolve the listing name. For what it is worth, Recorded Future appears to have taken that route: the connector was renamed from "Recorded Future" to "Recorded Future For Splunk SOAR" in 4.1.0 and kept appid 6efe0e1b-76ac-4ffd-8fa0-ac58fd6efd77 from its earlier versions. The version request follows from the same answer. If the UUID stays, the published version would need to be above 1.2.1 so existing installs see an upgrade. If the UUID has to change, then 1.0.0 is correct and I will drop the request. How would you like the rename delivered? I can open a follow-up PR against main is that is easiest. |
|
I think there are two separate questions here. Keeping the same UUID while renaming the app is one decision, and Recorded Future is a useful precedent for that. It was renamed to a SOAR-specific app name while retaining the same app UUID. The version request is different. Our connector release flow is automated through semantic-release: the workflow computes the next version from the repository’s release/tag history and then updates the app metadata during publish. Since the earlier Analyst1 builds were distributed outside this GitHub/Splunkbase release flow, I’d prefer not to manually add release artifacts, tags, or metadata only to make the first public release come out as Given that this is a conversion to the SOAR SDK,if we keep the same UUID and treat this as the next version of the existing customer-distributed connector, then this should be a major version release, e.g. My preference is to treat this as a clean new Splunkbase SOAR app: update the app UUID and app name in |
|
Thanks for laying out both options. Would it be possible to keep the same UUID and release as 2.0.0, the version you indicated for that path? Our concern with the new UUID is the existing installs. Every customer running the current connector would have to install the new app alongside the old one, recreate each asset and re-enter its credentials, and repoint each Analyst1 action block in their visual playbooks. Keeping the UUID would make it an ordinary upgrade, with assets and playbooks left in place. On the release-history point, this would not require backfilling anything from the private distribution. No tags, artifacts, or metadata behind the existing 1.0.0 tag. The follow-up PR would just need to carry a breaking change commit so the flow computes 2.0.0 from that tag, which seems reasonable given the SDK conversion and the move to a 7.0 minimum. Happy to open the follow-up PR against main with the rename whenever you decide. I think the appid_to_name entry in dev-cicd-tools would need the new name as well, since the app name check compares against it, and I can open that PR too if it helps. |
|
Hey @mvforgione - could you please create a PR renaming the app to I've created a PR to bump version to 2.0.0 #5. I'll merge once we rename the app. |
|
Both registry PRs are open, and the rename is #6. phantomcyber/dev-cicd-tools#256 renames the entry, and splunk-soar-connectors/.github#620 adds it. That file was missing this app id, so it is an addition there rather than an edit. #6 will show a red One question on sequencing. Should the rename commit carry the breaking change marker so it publishes 2.0.0 on merge, or stay non-releasing so #5 does the bump? I ask because the build and enqueue-publish jobs do not look gated on semantic-release producing a new version, and I was not sure whether a non-releasing merge would re-enqueue 1.0.0. It is a #6 also touches |
|
@mvforgione - https://splunkbase.splunk.com/app/9495 app released to splunkbase with the available versions starting from 2.0.0 |
Features
Manual Documentation
manual_readme_content.md documents both authentication methods (basic auth for the 1_0 API and OAuth2 client credentials for the 1_1 API).
Other information