Provides daily updated authoritative data and APIs tracking state-of-the-art AI models across categories from LMArena, Artificial Analysis, and HuggingFace.
# SOTA Tracker
**The definitive open-source database of State-of-the-Art AI models.**
Auto-updated daily from [LMArena](https://lmarena.ai), [Artificial Analysis](https://artificialanalysis.ai), and [HuggingFace](https://huggingface.co).
## Why This Exists
AI models are released weekly. Keeping track is impossible. This project:
1. **Curates authoritative data** - LMArena Elo rankings, manual curation for video/image/audio models
2. **Updates daily** via GitHub Actions
3. **Exports to JSON/CSV/SQLite** - Use in your own projects
4. **Provides multiple interfaces** - Static files, REST API, or MCP server
## Quick Start: Use the Data
### Option 1: Download JSON/CSV
```bash
# Latest data (updated daily)
curl -O https://raw.githubusercontent.com/romancircus/sota-tracker-mcp/main/data/sota_export.json
curl -O https://raw.githubusercontent.com/romancircus/sota-tracker-mcp/main/data/sota_export.csv
```
### Option 2: Clone and Query Locally
```bash
git clone https://github.com/romancircus/sota-tracker-mcp.git
cd sota-tracker-mcp
# Query with sqlite3
sqlite3 data/sota.db "SELECT name, sota_rank FROM models WHERE category='llm_api' ORDER BY sota_rank LIMIT 10"
# List forbidden/outdated models
sqlite3 data/sota.db "SELECT name, reason, replacement FROM forbidden"
```
### Option 3: Use with Claude Code (Recommended)
The recommended approach for Claude Code users is **static file embedding** (lower token cost than MCP):
```bash
# Set up daily auto-update of CLAUDE.md
cp scripts/update_sota_claude_md.py ~/scripts/
# Enable systemd timer (runs at 6 AM daily)
systemctl --user enable --now sota-update.timer
# Or run manually
python ~/scripts/update_sota_claude_md.py --update
```
This embeds a compact SOTA summary directly in your `~/.claude/CLAUDE.md` file.
### Option 4: REST API
```bash
# Start the API server
uvicorn rest_api:app --host 0.0.0.0 --port 8000
# Query endpoints
curl "http://localhost:8000/api/v1/models?category=llm_api"
curl "http://localhost:8000/api/v1/forbidden"
curl "http://localhost:8000/api/v1/models/FLUX.1-dev/freshness"
```
### Option 5: MCP Server (Optional)
MCP support is available but disabled by default (higher token cost). To enable:
```bash
# Edit .mcp.json to add the server config
cat > .mcp.json << 'EOF'
{
"mcpServers": {
"sota-tracker": {
"command": "python",
"args": ["server.py"]
}
}
}
EOF
```
## Data Sources
| Source | Data | Update Frequency |
|--------|------|------------------|
| [LMArena](https://lmarena.ai/leaderboard) | LLM Elo rankings (6M+ human votes) | Daily |
| [Artificial Analysis](https://artificialanalysis.ai) | LLM benchmarks, pricing, speed | Daily |
| [HuggingFace](https://huggingface.co) | Model downloads, trending | Daily |
| Manual curation | Video, Image, Audio, Video2Audio models | As needed |
## Categories
| Category | Description | Top Models (Feb 2026) |
|----------|-------------|----------------------|
| `llm_api` | Cloud LLM APIs | Gemini 3 Pro, Grok 4.1, Claude Opus 4.5 |
| `llm_local` | Local LLMs (GGUF) | Qwen3, Llama 3.3, DeepSeek-V3 |
| `llm_coding` | Code-focused LLMs | Qwen3-Coder, DeepSeek-V3 |
| `image_gen` | Image generation | Z-Image-Turbo, FLUX.2-dev, Qwen-Image |
| `video` | Video generation | LTX-2, Wan 2.2, HunyuanVideo 1.5 |
| `video2audio` | Video-to-audio (foley) | MMAudio V2 Large |
| `tts` | Text-to-speech | ChatterboxTTS, F5-TTS |
| `stt` | Speech-to-text | Whisper Large v3 |
| `embeddings` | Vector embeddings | BGE-M3 |
## REST API Endpoints
| Endpoint | Description |
|----------|-------------|
| `GET /api/v1/models?category=X` | Get SOTA for a category |
| `GET /api/v1/models/:name/freshness` | Check if model is current or outdated |
| `GET /api/v1/forbidden` | List outdated models to avoid |
| `GET /api/v1/compare?model_a=X&model_b=Y` | Compare two models |
| `GET /api/v1/recent?days=30` | Models released in past N days |
| `GET /api/v1/recommend?task=chat` | Get recommendation for a task |
| `GET /health` | Health check |
## Run Your Own Scraper
```bash
# Install dependencies
pip install -r requirements.txt
pip install playwright
playwright install chromium
# Run all scrapers
python scrapers/run_all.py --export
# Output:
# data/sota_export.json
# data/sota_export.csv
# data/lmarena_latest.json
```
## GitHub Actions (Auto-Update)
This repo uses GitHub Actions to:
- **Daily**: Scrape all sources, update database, commit changes
- **Weekly**: Create a tagged release with JSON/CSV exports
To enable on your fork:
1. Fork this repo
2. Go to Settings → Actions → Enable workflows
3. Data will auto-update daily at 6 AM UTC
## File Structure
```
sota-tracker-mcp/
├── server.py # MCP server (optional)
├── rest_api.py # REST API server
├── init_db.py # Database initialization + seeding
├── requirements.txt # Dependencies
├── data/
│ ├── sota.db # SQLite database
│ ├── sota_export.json # Full JSON export
│ ├── sota_export.csv # CSV export
│ └── forbidden.json # Outdated models list
├── scrapers/
│ ├── lmarena.py # LMArena scraper (Playwright)
│ ├── artificial_analysis.py # AA scraper (Playwright)
│ └── run_all.py # Unified runner
├── fetchers/
│ ├── huggingface.py # HuggingFace API
│ └── cache_manager.py # Smart caching
└── .github/workflows/
└── daily-scrape.yml # GitHub Actions workflow
```
## Contributing
Found a model that's missing or incorrectly ranked?
1. **For manual additions**: Edit `init_db.py` and submit a PR
2. **For scraper improvements**: Edit files in `scrapers/`
3. **For new data sources**: Add a new scraper and update `run_all.py`
See [CONTRIBUTING.md](CONTRIBUTING.md) for full developer setup and PR process.
## OpenCode / Agents.md Integration
The repo now supports updating `agents.md` files for OpenCode agents:
```bash
# Update your agents.md with latest SOTA data
python update_agents_md.py
# Minimal version (top 1 model per category, lightweight)
python update_agents_md.py --minimal
# Custom categories and limit
python update_agents_md.py --categories llm_local image_gen --limit 3
# Force refresh from sources first
python update_agents_md.py --refresh
```
### Automation
Add to your cron or systemd timer for daily updates:
```cron
# ~: crontab -e
@daily python ~/Apps/sota-tracker-mcp/update_agents_md.py
```
Or systemd:
```bash
# ~/.config/systemd/user/sota-update.service
[Unit]
Description=Update SOTA models for agents
After=network.target
[Service]
ExecStart=%h/Apps/sota-tracker-mcp/update_agents_md.py
[Install]
WantedBy=default.target
# ~/.config/systemd/user/sota-update.timer
[Unit]
Description=Daily SOTA data update
OnCalendar=daily
AccuracySec=1h
[Install]
WantedBy=timers.target
# Enable
systemctl --user enable --now sota-update.timer
```
See [CONTRIBUTING.md](CONTRIBUTING.md) for full setup guide
## Data Attribution & Legal
This project aggregates **publicly available benchmark data** from third-party sources. We do not claim ownership of rankings, Elo scores, or benchmark results.
### Data Sources (Used With Permission)
| Source | Data | Permission |
|--------|------|------------|
| [LMArena](https://lmarena.ai) | Chatbot Arena Elo rankings | `robots.txt: Allow: /` |
| [Artificial Analysis](https://artificialanalysis.ai) | LLM quality benchmarks | `robots.txt: Allow: /` (explicitly allows AI crawlers) |
| [HuggingFace](https://huggingface.co) | Model metadata, downloads | Public API |
| [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) | Open-source LLM benchmarks | CC-BY license |
### Disclaimer
- All benchmark scores and rankings are the intellectual work of their respective sources
- This project provides aggregation and tooling, not original benchmark data
- Data is scraped once daily to minimize server load
- If you are a data source and wish to be excluded, please open an issue
### Fair Use
This project:
- Aggregates factual data (not copyrightable)
- Adds value through tooling (API server, unified format, forbidden list)
- Attributes all sources with links
- Does not compete commercially with sources
- Respects robots.txt permissions
## License
MIT - See [LICENSE](LICENSE) for details.
The **code** in this repository is MIT licensed. The **data** belongs to its respective sources (see attribution above).
don't have the plugin yet? install it then click "run inline in claude" again.