VictoriaMetrics

VictoriaMetrics is a fast, cost-effective, and scalable monitoring solution and time-series database. It's fully compatible with Prometheus, offering long-term storage, high performance, and extended query capabilities through MetricsQL. VictoriaMetrics can be used as both a Prometheus replacement and as long-term storage for Prometheus metrics, with significant improvements in resource efficiency and query performance.

Authentication Types

VictoriaMetrics supports 1 authentication method through this connector:

• API Key (Basic Auth) - Username and password authentication using HTTP Basic Authentication
  • Pros: Simple to set up, compatible with self-hosted instances, standard HTTP authentication
  • Cons: Credentials are base64-encoded (not encrypted), requires HTTPS for security
  • Best for: Self-hosted VictoriaMetrics deployments, internal monitoring infrastructure

Note for VictoriaMetrics Cloud users: This connector is configured for self-hosted instances. VictoriaMetrics Cloud users should configure their deployment URL and use their access token as the password with any username (or set up an HTTP proxy to translate the X-VM-Cloud-Access header).

General Settings

Before using the connector, you need to configure:

• VictoriaMetrics Instance URL - The base URL of your VictoriaMetrics server
  • Self-hosted single-node: http://localhost:8428
  • Self-hosted cluster (vmselect): http://vmselect:8481/select/0/prometheus
  • VictoriaMetrics Cloud: https://your-deployment.victoriametrics.cloud

Setting up Basic Auth for Self-Hosted

To use VictoriaMetrics with Webrix, you need to configure authentication if your instance requires it.

1. Configure Basic Auth on VictoriaMetrics

VictoriaMetrics supports several authentication methods. The simplest is using vmauth as a proxy.

Option A: Using vmauth (Recommended)

1. Install vmauth (comes with VictoriaMetrics):

# Download vmauth
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.97.0/vmutils-linux-amd64-v1.97.0.tar.gz
tar xvf vmutils-linux-amd64-v1.97.0.tar.gz

2. Create an auth.yml configuration file:

users:
  - username: "admin"
    password: "your-password"
    url_prefix: "http://localhost:8428"

3. Start vmauth:

./vmauth-prod -auth.config=auth.yml -httpListenAddr=:8427

4. Use http://localhost:8427 as your VictoriaMetrics Instance URL

Option B: Using nginx as reverse proxy

1. Configure nginx with Basic Auth:

server {
    listen 8427;
    location / {
        auth_basic "VictoriaMetrics";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://localhost:8428;
    }
}

2. Create htpasswd file:

htpasswd -c /etc/nginx/.htpasswd admin

2. Configure in Webrix

1. In Webrix, go to Integrations → New → Built-in

2. Select VictoriaMetrics and click Use

3. Under General Settings, enter your VictoriaMetrics Instance URL

   • Self-hosted: http://localhost:8427 (if using vmauth)
   • Cluster: http://vmselect:8481/select/0/prometheus
   • Cloud: https://your-deployment.victoriametrics.cloud

4. Under Authentication Type, select API Key

5. Enter your Username (e.g., admin)

6. Enter your Password

7. Click Save Changes

3. Test the Connection

1. After saving, click Connect to test the authentication

2. Try running a simple query using "Execute Instant Query" with query: up

3. You should see metrics data returned successfully

Setting up VictoriaMetrics Cloud

If you're using VictoriaMetrics Cloud, follow these steps:

1. Get Your Access Token

1. Log in to VictoriaMetrics Cloud Console

2. Navigate to your deployment

3. Go to the Access tokens tab

4. Click Generate Token

5. Enter a name (e.g., "Webrix Integration")

6. Select access level:

   • Read - For querying only
   • Write - For data ingestion only
   • Read/Write - For full access (queries + data import)

7. Copy the token value (you won't be able to see it again)

2. Configure in Webrix

1. In Webrix, select VictoriaMetrics connector

2. Under General Settings, enter your deployment URL:

   • Example: https://your-deployment.victoriametrics.cloud

3. Under Authentication Type, select API Key

4. Enter token as the Username (or leave empty)

5. Paste your Access Token as the Password

6. Click Save Changes

tip

VictoriaMetrics Cloud uses the X-VM-Cloud-Access header for authentication, which is different from standard Basic Auth. The connector will attempt to work with the token as a password, but you may need to use a reverse proxy to translate the authentication header if this doesn't work directly.

MetricsQL vs PromQL

VictoriaMetrics uses MetricsQL, an extended version of PromQL that includes:

• All PromQL functions - Fully compatible with Prometheus queries
• Additional aggregation functions - quantiles(), mad(), outliers_mad()
• Extended rollup functions - rollup(), rollup_rate(), rollup_deriv()
• Time series filtering - limit_offset(), topk_min(), bottomk_max()
• Better performance - Optimized query execution and memory usage

Example MetricsQL extensions:

# Get rate with rollup
rollup_rate(http_requests_total[5m])

# Find outliers using MAD algorithm
outliers_mad(5, cpu_usage) by (instance)

# Multiple quantiles at once
quantiles("phi", 0.5, 0.9, 0.99, response_time)

Common Use Cases

Querying Metrics

Use the query tools to retrieve metrics data:

• Execute Instant Query - Get current metric values

  • Example: up - Check which targets are up
  • Example: rate(http_requests_total[5m]) - Request rate over last 5 minutes
  • Example: increase(requests[1h]) - Total requests in the last hour

• Execute Range Query - Get metrics over a time period

  • Example: Query cpu_usage from 1 hour ago to now with 1-minute resolution
  • Perfect for creating graphs and dashboards
  • VictoriaMetrics optimizes range queries for better performance

• Execute Multiple Queries - Run multiple queries in parallel

  • More efficient than separate requests
  • Ideal for dashboards with multiple panels

Discovering Metrics

Explore what metrics and labels are available:

• List All Label Names - See all available labels
• Get Label Values - Find all values for a specific label
• Find Series by Label Matchers - Discover time series matching criteria
• List Metric Names - Get all available metric names
• Get TSDB Statistics - Analyze cardinality and resource usage

Data Import/Export

VictoriaMetrics provides powerful import and export capabilities:

Importing Data:

• Import JSON Data - Bulk import from VictoriaMetrics JSON format
• Import Prometheus Data - Import from Prometheus exposition format
• Import CSV Data - Import from CSV with custom column mapping
• Import Native Format - Fast import from VictoriaMetrics native binary format

Exporting Data:

• Export Series as JSON - Backup or migrate data in JSON format
• Export Series as CSV - Export for analysis in spreadsheets
• Export Series in Native Format - Most efficient format for backups
• Export Specific Time Range - Extract data for specific periods

Management Operations

Manage your VictoriaMetrics instance:

• Delete Time Series - Remove unwanted metrics permanently
• Reset Rollup Result Cache - Clear query cache for fresh calculations
• Get Build Info - Check version and build information
• Health Check - Verify server availability
• Get Active Queries - Monitor currently executing queries

Troubleshooting

Authentication Failed (401 Unauthorized)

You receive 401 errors when trying to query VictoriaMetrics.

Cause: Incorrect credentials, or authentication not configured.

Solution:

1. Verify your username and password are correct
2. Check that vmauth or your reverse proxy is running
3. Test authentication manually:

   curl -u username:password http://victoriametrics-url/api/v1/query?query=up

4. For VictoriaMetrics Cloud, ensure you're using a valid access token

Connection Refused or Timeout

Cannot connect to VictoriaMetrics instance.

Cause: Incorrect instance URL, VictoriaMetrics not running, or network issues.

Solution:

1. Verify the Instance URL is correct and includes the protocol (http:// or https://)
2. Check that VictoriaMetrics is running:

   curl http://localhost:8428/health

3. For cluster deployments, ensure you're connecting to vmselect, not vmstorage
4. Ensure no firewall rules are blocking access
5. For HTTPS instances, ensure the certificate is valid

Query Timeout or Performance Issues

Queries are slow or timing out.

Cause: Query is too expensive, high cardinality, or insufficient resources.

Solution:

1. Use "Get Active Queries" to see what's running
2. Use "Execute Query with Trace" to understand query execution
3. Check TSDB statistics for high cardinality labels
4. Add more specific label matchers to reduce data scanned
5. Consider using recording rules for expensive queries
6. Increase resources (memory, CPU) if consistently slow

Import/Export Failures

Data import or export operations fail.

Cause: Incorrect format, large data size, or network issues.

Solution:

1. Verify data format matches the import endpoint
   • JSON import expects VictoriaMetrics JSON line format
   • Prometheus import expects exposition format
   • CSV import requires format specification
2. For large imports, consider batching data
3. Use native format for fastest import/export
4. Check disk space on both client and server
5. Verify timestamps are in the correct format (Unix milliseconds for JSON)

High Memory Usage

VictoriaMetrics consuming too much memory.

Cause: High cardinality, large queries, or insufficient cache configuration.

Solution:

1. Use "Get TSDB Statistics" to identify high cardinality labels
2. Avoid labels with unbounded values (user IDs, UUIDs, timestamps)
3. Use -memory.allowedPercent flag to limit memory usage
4. Configure appropriate retention period with -retentionPeriod
5. Use -search.maxMemoryPerQuery to limit per-query memory
6. Consider deduplication interval with -dedup.minScrapeInterval

VictoriaMetrics Cloud: X-VM-Cloud-Access Header

Authentication with cloud access token not working.

Cause: Connector uses Basic Auth, but VictoriaMetrics Cloud expects X-VM-Cloud-Access header.

Solution:

Option 1: Use HTTP proxy to translate headers

Set up nginx or similar to translate authentication:

location / {
    proxy_pass https://your-deployment.victoriametrics.cloud;
    proxy_set_header X-VM-Cloud-Access $http_authorization;
    # Remove Basic Auth header
    proxy_set_header Authorization "";
}

Option 2: Use access token as password

Some VictoriaMetrics Cloud deployments support both authentication methods. Try:

• Username: (leave empty or use "bearer")
• Password: your access token

MetricsQL Function Not Found

Query fails with unknown function error.

Cause: Using VictoriaMetrics-specific MetricsQL functions not available in Prometheus.

Solution:

1. Check VictoriaMetrics version - some functions are newer
2. Refer to MetricsQL documentation
3. Ensure you're querying VictoriaMetrics, not Prometheus
4. Fall back to standard PromQL if needed for compatibility

Data Not Appearing After Import

Imported data successfully but can't query it.

Cause: Incorrect timestamps, wrong tenant (for cluster), or out of retention period.

Solution:

1. Verify timestamps are within retention period
2. For cluster deployments, ensure correct tenant ID in URL
3. Check that metric names and labels don't conflict with existing series
4. Use "Find Series by Label Matchers" to verify data exists
5. Check VictoriaMetrics logs for import errors
6. Ensure timestamps are in correct format (Unix seconds or milliseconds)

Performance Best Practices

Query Optimization

• Use specific label matchers to reduce data scanned
• Use rollup_rate() instead of rate() for better performance
• Leverage topk() and bottomk() to limit results
• Add step parameter appropriate for your time range
• Use limit_offset() for pagination in large result sets

Storage Optimization

• Configure appropriate retention period: -retentionPeriod=12w
• Enable deduplication for reduced storage: -dedup.minScrapeInterval=30s
• Use downsampling for old data if supported
• Regularly monitor cardinality with TSDB statistics
• Delete unused metrics to reclaim space

Import/Export Best Practices

• Use native format for fastest backup/restore
• Batch large imports to avoid overwhelming the system
• Use JSON format for cross-system compatibility
• CSV format is best for external analysis tools
• Export in chunks for very large datasets

High Availability

• Use VictoriaMetrics cluster for production workloads
• Set up remote write from Prometheus for redundancy
• Regular backups using export functionality
• Monitor query performance with active queries endpoint
• Configure appropriate resource limits

Important Notes

Security Considerations

• Always use HTTPS in production to protect credentials
• Use VictoriaMetrics Cloud for managed security and updates
• Limit access to delete and admin endpoints
• Regular backups using export functionality
• Consider network-level access controls

VictoriaMetrics vs Prometheus

VictoriaMetrics offers several advantages over Prometheus:

• Better resource usage - Up to 7x less memory, less CPU
• Horizontal scalability - Cluster version scales to millions of series
• Long-term storage - Native support without separate TSDB
• Faster queries - Optimized query execution engine
• Enhanced MetricsQL - Additional functions and operators
• Deduplication - Built-in deduplication of identical samples
• Downsampling - Reduce storage for historical data

When to Use VictoriaMetrics

• Long-term metrics retention (months to years)
• High-cardinality metrics (millions of time series)
• Resource-constrained environments
• Large-scale monitoring (multiple Prometheus instances)
• Need for faster query performance
• Multi-tenancy requirements (cluster version)

Additional Resources

• VictoriaMetrics Official Documentation
• MetricsQL Reference
• HTTP API Reference
• Quick Start Guide
• VictoriaMetrics Cloud
• Best Practices
• Migration from Prometheus