Skip to main content

Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine designed for horizontal scalability, reliability, and real-time search capabilities. It's built on Apache Lucene and excels at full-text search, log analytics, metrics analysis, and complex queries across large datasets. Elasticsearch is widely used for application search, log and event data analysis, infrastructure monitoring, and business analytics.

Authentication Types​

Elasticsearch supports API Key authentication:

  • API Key - Use Elasticsearch API keys for secure, token-based authentication
    • Pros: Secure, can be scoped to specific privileges, easy to rotate
    • Cons: Requires API key creation in Elasticsearch
    • Best for: Production use with fine-grained access control

General Settings​

Before using the connector, you need to configure:

  • Elasticsearch URL - The base URL of your Elasticsearch instance
    • Examples:
      • Local: http://localhost:9200
      • Hosted: https://your-cluster.es.us-east-1.aws.elastic.cloud:9243
      • Self-managed: https://elasticsearch.yourcompany.com
    • Important: Include the protocol (http/https) and port
tip

For Elastic Cloud deployments, you can find your Elasticsearch endpoint in the Cloud console under "Deployments" → Your deployment → "Elasticsearch" → "Endpoint".

Setting up API Key Authentication​

API Key authentication is the recommended method for production use.

Creating an API Key in Elasticsearch​

  1. Via Kibana UI:

    • Go to Stack Management → Security → API Keys
    1. Click Create API key
    1. Configure the API key:
      • Name: Give it a descriptive name (e.g., "Webrix Integration")
      • Expiration: Set an expiration date or leave it to never expire
      • Role descriptors (optional): Define specific privileges
    1. Click Create API key
    2. Important: Copy the API key immediately - it won't be shown again
    1. The key format will be: VnVhQ2ZHY0JDZGJrUW0tZTVhT3g6dWkybHAyYXhUTm1zeWFrdzl0dk5udw==

Configuring in Webrix​

  1. In Webrix connector settings, select API Key authentication
  2. Paste the encoded API key in the Token field
  3. Click Save Changes

For full functionality, the API key or user should have:

Cluster privileges:

  • monitor - View cluster health and stats
  • manage_index_templates - Create and manage index templates
  • manage_ilm - Manage Index Lifecycle Management (optional)

Index privileges (for relevant indices):

  • read - Search and retrieve documents
  • write - Index and update documents
  • create_index - Create new indices
  • delete_index - Delete indices
  • view_index_metadata - View index settings and mappings
  • manage - Modify index settings and mappings

You can scope these privileges to specific index patterns (e.g., logs-*, metrics-*) for tighter security.

Common Use Cases​

1. Log and Event Analysis​

Use Elasticsearch tools to search logs, count error events, analyze trends with aggregations, and correlate events across time.

Implement full-text search for your application data, with relevance tuning, filtering, faceting, and highlighting.

3. Metrics and Monitoring​

Store and analyze time-series metrics, create dashboards with aggregations, and monitor system health using cluster APIs.

4. Data Management​

Manage index lifecycle with templates, aliases, and automated operations. Implement blue-green deployments with aliases.

5. Bulk Operations​

Efficiently index large datasets using bulk operations, and perform mass updates or deletions with query-based operations.

Troubleshooting​

Connection Issues​

Symptom: "Connection refused" or timeout errors

Cause: Incorrect Elasticsearch URL, network issues, or Elasticsearch not running

Solution:

  1. Verify Elasticsearch is running: curl http://localhost:9200
  2. Check the URL in connector settings matches your Elasticsearch endpoint
  3. For Elastic Cloud, ensure you're using the correct endpoint from the Cloud console
  4. Check firewall rules and network connectivity

Authentication Errors​

Symptom: "401 Unauthorized" or "403 Forbidden" errors

Cause: Invalid API key, expired credentials, or insufficient permissions

Solution:

  1. For API Key authentication:

    • Verify the API key is correctly copied (encoded format)
    • Check if the API key has expired in Elasticsearch
    • Create a new API key if needed

  2. For permission errors (403):

    • Review the API key privileges
    • Ensure required permissions are granted (see "Recommended Permissions")
    • Check Elasticsearch logs for detailed permission errors

SSL/TLS Certificate Issues​

Symptom: "SSL certificate problem" or "certificate verify failed"

Cause: Self-signed certificates or certificate chain issues

Solution:

  1. For self-signed certificates in development:

    • Add certificate to system trust store, or
    • Configure Elasticsearch to use a valid certificate

  2. For Elastic Cloud:

    • Should work automatically as Elastic Cloud uses valid certificates
    • If issues persist, verify network proxy settings

Query Timeout​

Symptom: "Request timeout" or slow responses

Cause: Complex queries, large datasets, or under-resourced cluster

Solution:

  1. Optimize queries:

    • Use filters instead of queries when scoring isn't needed
    • Reduce result size or use pagination
    • Use specific index patterns instead of searching all indices

  2. Check cluster health:

    • Run "Get Cluster Health" tool
    • Review shard allocation and node resources
    • Consider scaling the cluster if under-resourced

  3. Increase timeout (if appropriate):

    • Add timeout parameter to search queries
    • Example: {"query": {...}, "timeout": "30s"}

Index Not Found​

Symptom: "index_not_found_exception"

Cause: Trying to access a non-existent index

Solution:

  1. Use "List Indices" tool to see available indices
  2. Check for typos in index names
  3. Verify index patterns with wildcards are correct
  4. Create the index if it should exist using "Create Index" tool

Mapping Conflicts​

Symptom: "mapper_parsing_exception" or "illegal_argument_exception"

Cause: Trying to index a field with a different type than the mapping

Solution:

  1. Use "Get Mapping" tool to view current field types
  2. Fix the document to match existing mappings, or
  3. If changing field type is necessary:
    • Create a new index with the correct mapping
    • Use reindex API to copy data
    • Update aliases to point to new index
    • Delete old index

Best Practices​

Security​

  • Use API Key authentication for production
  • Scope API keys to minimum required privileges
  • Rotate API keys regularly
  • Enable SSL/TLS for all connections
  • Never expose Elasticsearch directly to the internet

Performance​

  • Use bulk operations for indexing multiple documents
  • Implement index templates for consistent settings
  • Use aliases for zero-downtime index management
  • Set appropriate refresh intervals (default 1s can be increased for write-heavy workloads)
  • Use filters for boolean logic (cached) vs queries (scored)

Data Management​

  • Use Index Lifecycle Management (ILM) for time-series data
  • Implement appropriate shard sizing (target 10-50GB per shard)
  • Set replica count based on availability requirements
  • Use index templates to standardize settings across indices
  • Regular monitoring of cluster health and resources

Search Optimization​

  • Use specific index patterns instead of wildcard searches
  • Implement pagination with from and size or search_after
  • Use _source filtering to return only needed fields
  • Leverage aggregations for analytics instead of post-processing results
  • Use the explain API to understand and optimize relevance scoring

Additional Resources​