Skip to main content
OpenSRE queries Splunk using the REST API to surface relevant log evidence during alert investigations — searching indexes with SPL, correlating error patterns with incidents, and identifying root causes.

​
Prerequisites

  • Splunk Enterprise or Splunk Cloud instance (version 8.x or later)
  • REST API access on port 8089
  • A bearer token with search capability (see Generating a Bearer Token)

​
Setup

​
Option 1: Interactive CLI

opensre integrations setup
Select Splunk when prompted and provide your REST API base URL and bearer token.

​
Option 2: Environment variables

Add to your .env: 
SPLUNK_URL=https://splunk.corp.com:8089    # REST API base URL (port 8089 default)
SPLUNK_TOKEN=your-bearer-token             # API bearer token (NOT an HEC token)
SPLUNK_INDEX=main                          # Default index to search (optional)
SPLUNK_VERIFY_SSL=true                     # Set false to skip SSL verification (optional)
SPLUNK_CA_BUNDLE=/etc/ssl/certs/corp-ca.pem  # Path to custom CA bundle (optional)
VariableDefaultDescription
SPLUNK_URL—Required. REST API base URL including port
SPLUNK_TOKEN—Required. Bearer token with search capability
SPLUNK_INDEXmainDefault index searched when no index is specified in the alert
SPLUNK_VERIFY_SSLtrueSet to false to disable SSL verification (dev/local only)
SPLUNK_CA_BUNDLE—Path to a PEM CA bundle for enterprise self-signed certificates

​
Option 3: Persistent store

{
  "version": 1,
  "integrations": [
    {
      "id": "splunk-prod",
      "service": "splunk",
      "status": "active",
      "credentials": {
        "base_url": "https://splunk.corp.com:8089",
        "token": "your-bearer-token",
        "index": "main",
        "verify_ssl": true,
        "ca_bundle": "/etc/ssl/certs/corp-ca.pem"
      }
    }
  ]
}

​
Multi-instance setup

To connect multiple Splunk instances (e.g. separate prod and staging clusters): 
SPLUNK_INSTANCES='[
  {"name":"prod","tags":{"env":"prod"},"credentials":{"base_url":"https://splunk-prod:8089","token":"prod-token","index":"prod"}},
  {"name":"staging","tags":{"env":"staging"},"credentials":{"base_url":"https://splunk-staging:8089","token":"staging-token","index":"staging"}}
]'
When SPLUNK_INSTANCES is set it overrides the single-instance SPLUNK_URL / SPLUNK_TOKEN variables.

​
Generating a bearer token

OpenSRE uses bearer tokens — not basic auth and not HEC tokens. To generate one: Via the Splunk UI:
  1. Go to Settings → Tokens
  2. Click New Token
  3. Set a name (e.g. opensre) and an expiry date
  4. Copy the generated token
Via the REST API (replace <PASSWORD> with your admin password): 
curl -sk -u admin:<PASSWORD> \
  https://splunk.corp.com:8089/services/authorization/tokens \
  -X POST \
  --data-urlencode "name=opensre" \
  --data-urlencode "expires_on=+90d" \
  --data-urlencode "output_mode=json" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['entry'][0]['content']['token'])"
The token needs the search capability. The admin role includes this by default. For a dedicated service account, ensure the role includes:
  • search
  • read_splunkd_private_settings (needed for the verify call against /services/server/info)

​
Verify

opensre integrations verify splunk
Expected output: 
Service: splunk
Status: passed
Detail: Connected to Splunk 9.x.x

​
How queries are generated

OpenSRE builds SPL queries deterministically from the alert payload — the LLM selects which tool to call but never writes the query itself. This keeps investigations reproducible and auditable. Query construction priority:
PrioritySourceExample
1annotations.splunk_query — verbatim SPL from your alertindex=prod "PaymentTimeout" | head 50
2annotations.query or annotations.log_queryAny pre-populated query field
3error_message field — keyword search built automaticallysearch index=main "NullPointerException" | head 50
4alert_name — last-resort keyword searchsearch index=main "payments-error-spike" | head 50
5Fallback — index scansearch index=main | head 50
To pass a specific SPL query through an alert, set commonAnnotations.splunk_query: 
{
  "alert_name": "Payment service errors",
  "commonAnnotations": {
    "splunk_query": "index=prod sourcetype=app_logs \"NullPointerException\" | head 50"
  }
}

​
Test with the built-in alert template

opensre investigate --template splunk
This runs a synthetic investigation using a pre-built alert fixture — no live alert infrastructure needed.

​
Troubleshooting

SymptomFix
SSL: CERTIFICATE_VERIFY_FAILEDSet SPLUNK_CA_BUNDLE=/path/to/corp-ca.pem (preferred) or SPLUNK_VERIFY_SSL=false (dev only)
HTTP 401 UnauthorizedToken expired or was generated with the wrong account — regenerate
HTTP 403 ForbiddenToken lacks search capability — check the role assigned to the token
Empty search resultsData may not have been ingested yet, or the index name is wrong
Connection refused on port 8089Splunk management port may be firewalled; confirm network access
opensre integrations verify failsCheck SPLUNK_URL includes the protocol and port (https://host:8089)

​
Security best practices

  • Use a read-only bearer token — never use an admin token in production.
  • Store SPLUNK_TOKEN in .env or the credential store, not in source code or CI logs.
  • Prefer a dedicated opensre service account with only the search capability.
  • For enterprise self-signed certificates, set SPLUNK_CA_BUNDLE to the CA bundle path rather than disabling verification entirely.
  • Set SPLUNK_VERIFY_SSL=false only in local or dev environments when you cannot supply a CA bundle.
  • Rotate tokens on a schedule and revoke them when no longer needed.