SOTA AI Model Tracker

SKILL.md

# SOTA Tracker

**The definitive open-source database of State-of-the-Art AI models.**

Auto-updated daily from [LMArena](https://lmarena.ai), [Artificial Analysis](https://artificialanalysis.ai), and [HuggingFace](https://huggingface.co).

## Why This Exists

AI models are released weekly. Keeping track is impossible. This project:

1. **Curates authoritative data** - LMArena Elo rankings, manual curation for video/image/audio models
2. **Updates daily** via GitHub Actions
3. **Exports to JSON/CSV/SQLite** - Use in your own projects
4. **Provides multiple interfaces** - Static files, REST API, or MCP server

## Quick Start: Use the Data

### Option 1: Download JSON/CSV

```bash
# Latest data (updated daily)
curl -O https://raw.githubusercontent.com/romancircus/sota-tracker-mcp/main/data/sota_export.json
curl -O https://raw.githubusercontent.com/romancircus/sota-tracker-mcp/main/data/sota_export.csv
```

### Option 2: Clone and Query Locally

```bash
git clone https://github.com/romancircus/sota-tracker-mcp.git
cd sota-tracker-mcp

# Query with sqlite3
sqlite3 data/sota.db "SELECT name, sota_rank FROM models WHERE category='llm_api' ORDER BY sota_rank LIMIT 10"

# List forbidden/outdated models
sqlite3 data/sota.db "SELECT name, reason, replacement FROM forbidden"
```

### Option 3: Use with Claude Code (Recommended)

The recommended approach for Claude Code users is **static file embedding** (lower token cost than MCP):

```bash
# Set up daily auto-update of CLAUDE.md
cp scripts/update_sota_claude_md.py ~/scripts/

# Enable systemd timer (runs at 6 AM daily)
systemctl --user enable --now sota-update.timer

# Or run manually
python ~/scripts/update_sota_claude_md.py --update
```

This embeds a compact SOTA summary directly in your `~/.claude/CLAUDE.md` file.

### Option 4: REST API

```bash
# Start the API server
uvicorn rest_api:app --host 0.0.0.0 --port 8000

# Query endpoints
curl "http://localhost:8000/api/v1/models?category=llm_api"
curl "http://localhost:8000/api/v1/forbidden"
curl "http://localhost:8000/api/v1/models/FLUX.1-dev/freshness"
```

### Option 5: MCP Server (Optional)

MCP support is available but disabled by default (higher token cost). To enable:

```bash
# Edit .mcp.json to add the server config
cat > .mcp.json << 'EOF'
{
  "mcpServers": {
    "sota-tracker": {
      "command": "python",
      "args": ["server.py"]
    }
  }
}
EOF
```

## Data Sources

| Source | Data | Update Frequency |
|--------|------|------------------|
| [LMArena](https://lmarena.ai/leaderboard) | LLM Elo rankings (6M+ human votes) | Daily |
| [Artificial Analysis](https://artificialanalysis.ai) | LLM benchmarks, pricing, speed | Daily |
| [HuggingFace](https://huggingface.co) | Model downloads, trending | Daily |
| Manual curation | Video, Image, Audio, Video2Audio models | As needed |

## Categories

| Category | Description | Top Models (Feb 2026) |
|----------|-------------|----------------------|
| `llm_api` | Cloud LLM APIs | Gemini 3 Pro, Grok 4.1, Claude Opus 4.5 |
| `llm_local` | Local LLMs (GGUF) | Qwen3, Llama 3.3, DeepSeek-V3 |
| `llm_coding` | Code-focused LLMs | Qwen3-Coder, DeepSeek-V3 |
| `image_gen` | Image generation | Z-Image-Turbo, FLUX.2-dev, Qwen-Image |
| `video` | Video generation | LTX-2, Wan 2.2, HunyuanVideo 1.5 |
| `video2audio` | Video-to-audio (foley) | MMAudio V2 Large |
| `tts` | Text-to-speech | ChatterboxTTS, F5-TTS |
| `stt` | Speech-to-text | Whisper Large v3 |
| `embeddings` | Vector embeddings | BGE-M3 |

## REST API Endpoints

| Endpoint | Description |
|----------|-------------|
| `GET /api/v1/models?category=X` | Get SOTA for a category |
| `GET /api/v1/models/:name/freshness` | Check if model is current or outdated |
| `GET /api/v1/forbidden` | List outdated models to avoid |
| `GET /api/v1/compare?model_a=X&model_b=Y` | Compare two models |
| `GET /api/v1/recent?days=30` | Models released in past N days |
| `GET /api/v1/recommend?task=chat` | Get recommendation for a task |
| `GET /health` | Health check |

## Run Your Own Scraper

```bash
# Install dependencies
pip install -r requirements.txt
pip install playwright
playwright install chromium

# Run all scrapers
python scrapers/run_all.py --export

# Output:
# data/sota_export.json
# data/sota_export.csv
# data/lmarena_latest.json
```

## GitHub Actions (Auto-Update)

This repo uses GitHub Actions to:
- **Daily**: Scrape all sources, update database, commit changes
- **Weekly**: Create a tagged release with JSON/CSV exports

To enable on your fork:
1. Fork this repo
2. Go to Settings → Actions → Enable workflows
3. Data will auto-update daily at 6 AM UTC

## File Structure

```
sota-tracker-mcp/
├── server.py                    # MCP server (optional)
├── rest_api.py                  # REST API server
├── init_db.py                   # Database initialization + seeding
├── requirements.txt             # Dependencies
├── data/
│   ├── sota.db                  # SQLite database
│   ├── sota_export.json         # Full JSON export
│   ├── sota_export.csv          # CSV export
│   └── forbidden.json           # Outdated models list
├── scrapers/
│   ├── lmarena.py               # LMArena scraper (Playwright)
│   ├── artificial_analysis.py   # AA scraper (Playwright)
│   └── run_all.py               # Unified runner
├── fetchers/
│   ├── huggingface.py           # HuggingFace API
│   └── cache_manager.py         # Smart caching
└── .github/workflows/
    └── daily-scrape.yml         # GitHub Actions workflow
```

## Contributing

Found a model that's missing or incorrectly ranked?

1. **For manual additions**: Edit `init_db.py` and submit a PR
2. **For scraper improvements**: Edit files in `scrapers/`
3. **For new data sources**: Add a new scraper and update `run_all.py`

See [CONTRIBUTING.md](CONTRIBUTING.md) for full developer setup and PR process.

## OpenCode / Agents.md Integration

The repo now supports updating `agents.md` files for OpenCode agents:

```bash
# Update your agents.md with latest SOTA data
python update_agents_md.py

# Minimal version (top 1 model per category, lightweight)
python update_agents_md.py --minimal

# Custom categories and limit
python update_agents_md.py --categories llm_local image_gen --limit 3

# Force refresh from sources first
python update_agents_md.py --refresh
```

### Automation

Add to your cron or systemd timer for daily updates:

```cron
# ~: crontab -e
@daily python ~/Apps/sota-tracker-mcp/update_agents_md.py
```

Or systemd:

```bash
# ~/.config/systemd/user/sota-update.service
[Unit]
Description=Update SOTA models for agents
After=network.target

[Service]
ExecStart=%h/Apps/sota-tracker-mcp/update_agents_md.py

[Install]
WantedBy=default.target

# ~/.config/systemd/user/sota-update.timer
[Unit]
Description=Daily SOTA data update
OnCalendar=daily
AccuracySec=1h

[Install]
WantedBy=timers.target

# Enable
systemctl --user enable --now sota-update.timer
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for full setup guide

## Data Attribution & Legal

This project aggregates **publicly available benchmark data** from third-party sources. We do not claim ownership of rankings, Elo scores, or benchmark results.

### Data Sources (Used With Permission)

| Source | Data | Permission |
|--------|------|------------|
| [LMArena](https://lmarena.ai) | Chatbot Arena Elo rankings | `robots.txt: Allow: /` |
| [Artificial Analysis](https://artificialanalysis.ai) | LLM quality benchmarks | `robots.txt: Allow: /` (explicitly allows AI crawlers) |
| [HuggingFace](https://huggingface.co) | Model metadata, downloads | Public API |
| [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) | Open-source LLM benchmarks | CC-BY license |

### Disclaimer

- All benchmark scores and rankings are the intellectual work of their respective sources
- This project provides aggregation and tooling, not original benchmark data
- Data is scraped once daily to minimize server load
- If you are a data source and wish to be excluded, please open an issue

### Fair Use

This project:
- Aggregates factual data (not copyrightable)
- Adds value through tooling (API server, unified format, forbidden list)
- Attributes all sources with links
- Does not compete commercially with sources
- Respects robots.txt permissions

## License

MIT - See [LICENSE](LICENSE) for details.

The **code** in this repository is MIT licensed. The **data** belongs to its respective sources (see attribution above).