Skip to main content

Splunk

Splunk is a leading platform for searching, monitoring, and analyzing machine-generated data from applications, systems, and infrastructure. It provides real-time insights, powerful search capabilities, and comprehensive monitoring for IT operations, security, and business analytics.

Authentication Types​

Splunk supports 1 authentication method:

  • API Token - Use Splunk authentication tokens for secure, long-lived API access.
    • Pros: Secure, no password exposure, can be easily revoked, supports different permission levels
    • Cons: Requires initial setup to generate token

General Settings​

Before using the connector, you need to configure:

  • Splunk Instance URL - Your Splunk instance URL including the management port (default 8089)
    • Splunk Enterprise example: https://splunk.yourcompany.com:8089
    • Splunk Cloud example: https://yourcompany.splunkcloud.com:8089
tip

The URL must include the management port (typically 8089) and use HTTPS. Don't include /services in the URL - the connector will add that automatically.

Setting up API Token Authentication​

Step 1: Log in to Splunk​

  1. Open your Splunk instance in a web browser
  2. Log in with your username and password

Step 2: Generate an Authentication Token​

For Splunk Enterprise or Splunk Cloud​

  1. Navigate to Settings > Users and authentication > Tokens

  2. Click New Token or Enable Token Authentication if not already enabled

  3. Fill in the token details:

    • User: Select your username (or the user the token should be associated with)
    • Token name: Give it a descriptive name (e.g., "Webrix MCP Connector")
    • Audience: Leave as default or specify your Splunk instance
    • Expiration: Choose an expiration time or select "Never" for long-lived tokens

  4. Click Create

  5. Important: Copy the generated token immediately - you won't be able to see it again!

Alternative: Generate Token via REST API​

If you prefer to generate a token programmatically:

curl -k -u admin:password https://your-splunk-instance:8089/services/authorization/tokens \
  -d name=webrix-token \
  -d audience=your-splunk-instance

Step 3: Configure in Webrix​

  1. In Webrix, navigate to the Splunk connector settings

  2. Enter your Splunk Instance URL in General Settings:

    • Format: https://your-instance:8089
    • Examples:
      • Enterprise: https://splunk.company.com:8089
      • Cloud: https://company.splunkcloud.com:8089

  3. In the authentication section, paste your API Token

  4. Click Save and then Test Connection to verify

Splunk Enterprise vs Splunk Cloud​

This connector supports both deployment types:

Splunk Enterprise (On-Premises)​

  • Use your custom domain and port (typically 8089)
  • Ensure the management port is accessible from where Webrix is running
  • May require firewall rules or VPN access
  • Example URL: https://splunk.yourcompany.com:8089

Splunk Cloud Platform​

  • Use your Splunk Cloud URL with port 8089
  • Management port may need to be opened via Splunk Support
  • Example URL: https://yourcompany.splunkcloud.com:8089
note

For Splunk Cloud free trial accounts, REST API access may be restricted. Contact Splunk Support to enable port 8089 access if needed.

Common Use Cases​

Running Searches​

Use Create Search Job to start a search, then Get Search Job Status to monitor progress, and Get Search Results to retrieve the data:

1. Create Search Job with query: "index=main error | stats count by source"
2. Get Search Job Status with the returned search ID
3. When complete, Get Search Results to retrieve findings

Managing Alerts​

  • Use List Fired Alerts to see recent alert triggers
  • Use Get Fired Alert Details to investigate specific alerts
  • Use Acknowledge Alert to mark alerts as reviewed

Working with Saved Searches​

  • List Saved Searches - Browse all saved/scheduled searches
  • Get Saved Search - View configuration of a specific search
  • Create Saved Search - Set up new scheduled searches or alerts
  • Update Saved Search - Modify existing searches
  • Delete Saved Search - Remove unneeded searches

Monitoring Your Environment​

  • Get Server Info - Check Splunk version and configuration
  • List Indexes - See available data repositories
  • List Data Inputs - View configured data sources
  • List Users and List Roles - Audit access control

Troubleshooting​

Connection Failed​

Cause: Cannot reach the Splunk management port

Solution:

  1. Verify the URL includes https:// and the port (:8089)
  2. Ensure port 8089 is accessible (not blocked by firewall)
  3. For Splunk Cloud, verify port 8089 is enabled via Splunk Support
  4. Test connectivity: curl -k https://your-instance:8089/services/server/info

Authentication Failed​

Cause: Invalid or expired token

Solution:

  1. Verify you copied the complete token (they're typically long strings)
  2. Check if the token has expired in Splunk settings
  3. Ensure the user associated with the token has appropriate permissions
  4. Try generating a new token
  5. Verify token with: curl -k -H "Authorization: Bearer YOUR_TOKEN" https://your-instance:8089/services/authentication/current-context

Search Job Not Found​

Cause: Search job has expired or been deleted

Solution:

Splunk automatically removes old search jobs. By default, jobs are removed after their TTL (time-to-live) expires. Either:

  • Retrieve results promptly after job completion
  • Increase the job's TTL when creating it
  • Use Export Search Results for immediate streaming results

Permission Denied​

Cause: User doesn't have required capabilities

Solution:

  1. Check the user's role has necessary capabilities:
    • For searches: search capability
    • For saved searches: schedule_search capability
    • For alerts: edit_tcp or admin_all_objects
    • For users/roles: edit_user or edit_roles_grantable
  2. In Splunk, go to Settings > Access controls > Roles
  3. Edit the user's role and add required capabilities
  4. Re-test the connection with the updated permissions

SSL Certificate Errors​

Cause: Self-signed or invalid SSL certificate

Solution:

This connector uses HTTPS by default. If you have a self-signed certificate:

  1. For production, install a valid SSL certificate on your Splunk instance
  2. For testing/development, the connector should handle self-signed certificates
  3. Verify your Splunk instance is accessible via HTTPS in a browser

Rate Limiting​

Cause: Too many API requests in short time

Solution:

Splunk has rate limits on API endpoints. If you encounter rate limiting:

  1. Reduce the frequency of requests
  2. Use pagination with smaller page sizes
  3. Cache results when possible
  4. For large data exports, use Export Search Results instead of multiple paginated calls

Best Practices​

Search Performance​

  • Use specific time ranges rather than searching all data
  • Limit searches to specific indexes when possible
  • Use field filters early in your search queries
  • Consider using Export Search Results for large result sets

Security​

  • Use tokens instead of username/password for API access
  • Create tokens with appropriate expiration times
  • Use service accounts with minimal required permissions
  • Regularly audit and rotate tokens
  • Never commit tokens to version control

Resource Management​

  • Cancel long-running searches that are no longer needed with Cancel Search Job
  • Clean up old saved searches that are no longer used
  • Monitor index sizes with List Indexes and Get Index Info
  • Use List Search Jobs to track active searches

Alert Management​

  • Review fired alerts regularly with List Fired Alerts
  • Acknowledge alerts after investigation with Acknowledge Alert
  • Test new alerts thoroughly before enabling in production
  • Document alert response procedures