Token Optimization
Token optimization is a feature that helps reduce token usage and costs by converting tool outputs into more compact formats. This is especially useful when dealing with large datasets or frequent API calls.
Overview
When MCP tools return data, they typically return it in JSON format with pretty printing. While this is readable, it can consume a significant number of tokens. Token optimization allows you to automatically convert all tool outputs to more efficient formats, potentially saving 20-70% of tokens depending on your data and chosen format.
How It Works
Token optimization is applied automatically to all tool outputs in your organization. When a tool returns data, the system converts it to your configured format before sending it to the AI model.
Configuration
Token optimization is configured at the organization level in the admin settings:
- Navigate to Admin Settings → Token Optimization
- Select your preferred Tool Output Format from the dropdown
- Click Save Changes
Available Formats
Default
No conversion is applied. Tool outputs are returned as-is from the MCP tools. In most cases, this is equivalent to JSON (Compact), which typically offers the best balance of compatibility and token efficiency. This is usually the recommended starting point.
Token Impact: Baseline (0% savings) - equivalent to JSON (Compact) in most cases
JSON (Pretty)
Standard JSON format with indentation and newlines.
{
"name": "John",
"email": "[email protected]",
"age": 30
}
Token Impact: +20-30% tokens compared to compact formats (uses MORE tokens)
JSON (Compact)
JSON format without whitespace. Removes all unnecessary spaces and newlines. This is the baseline for comparison.
{"name":"John","email":"[email protected]","age":30}
Token Impact: Baseline (0% savings/increase) - most compatible format
CSV
Comma-separated values format. Best for arrays of objects with consistent structure.
name,email,age
John,[email protected],30
Jane,[email protected],25
Token Impact: ~20-40% savings for tabular data
YAML
YAML format, which is more human-readable than JSON while maintaining similar token efficiency.
name: John
email: [email protected]
age: 30
Token Impact: Usually +20-30% more tokens
TOON (Table-Oriented Object Notation)
An optimized format specifically designed for tabular data. Uses a schema-aware approach with headers and delimited values. Learn more at the TOON Format repository.
repositories[2]{name,email,age}:
John,[email protected],30
Jane,[email protected],25
Token Impact: ~30-50% savings for structured data compared to JSON (Compact)
Testing with the Playground
The Token Optimization Playground allows you to test different formats with your own data before applying them to your organization.
How to Use the Playground
- Click the Playground button in the Token Optimization settings
- Choose one of two testing methods:
Test with Custom Data
- Enter or paste your test data (can be text, JSON, or any format)
- Press Cmd/Ctrl + Enter or click Test All Formats
- Review the results
Test with Real Data
- Click Test with Real Data button
- The system will automatically:
- Fetch the last 10 successful tool responses from your audit logs
- Test all formats against your actual data
- Calculate average token counts across all samples
- Review the averaged results
This is the recommended approach as it shows real-world impact based on your organization's actual tool usage patterns.
Understanding Results
The playground shows you:
- Tokens: The number of tokens the converted data will use
- Savings: Percentage reduction (green arrow down) or increase (red arrow up) compared to your original input
- Characters: Character count of the converted output
- Current: Indicator showing which format is currently active
- Sample Count: When using real data, shows how many responses were tested (displayed at the top of results)
Best Practices
When to Use Each Format
- Default: Recommended starting point - equivalent to JSON (Compact) in most cases, offering the best compatibility
- JSON (Pretty): Uses MORE tokens than other formats
- JSON (Compact): Explicit baseline format. Use when you want guaranteed JSON structure
- CSV: Best for arrays of objects with consistent fields (e.g., user lists, database query results). Saves ~20-30% tokens
- YAML: Use when you need human readability. Usualy +20-30% token usage.
- TOON: Maximum savings for tabular data. Best for large datasets with consistent structure. Saves ~30-50% tokens
Recommendations
- Start with Default: This is the recommended starting point as it uses the tool's native output format (usually JSON Compact)
- Avoid JSON (Pretty): This format uses 20-30% MORE tokens than other formats
- Test with Real Data: Use the playground with actual data from your tools to see real-world impact
- Consider Your Use Case:
- Heavy database queries or lists? → CSV or TOON (20-50% savings)
- Mixed data types? → Default or JSON (Compact)
- Need human readability in logs? → YAML
- Monitor Token Usage: Track your token consumption before and after enabling optimization
Limitations
- Data Structure: CSV and TOON work best with arrays of objects. Complex nested structures may not convert well
- Readability Trade-off: Compact formats save tokens but may be harder for humans to read in logs
- AI Model Compatibility: Most AI models can handle all formats, but test to ensure compatibility with your specific use case
FAQ
Will this affect AI model performance?
No. Modern AI models can understand data in various formats equally well. In some cases, compact formats may even improve performance by reducing the context window size.
Can I change the format at any time?
Yes. Format changes take effect immediately for all subsequent tool calls. Existing tool outputs are not affected.
What happens if conversion fails?
If format conversion fails for any reason, the original data is returned unchanged. An error is logged, but the tool execution continues normally.
Does this work with custom tools?
Yes. Token optimization applies to all tool outputs, including custom MCP tools.
How do I know which format is best for my organization?
Use the "Test with Real Data" button in the playground. This will automatically test all formats against your last 10 successful tool responses and show you the average token savings for each format. This gives you an accurate, data-driven answer based on your organization's actual tool usage patterns.
Alternatively, you can manually test with representative samples by entering them in the playground text area.
What if I don't have enough audit log data yet?
The "Test with Real Data" feature will use whatever successful tool responses are available (up to 10). If you have fewer than 10, it will average the results across whatever samples exist. If you have no tool responses yet, you can still use the manual testing method by entering sample data into the playground text area.