ModelAnalysis.si launches LLM cost, benchmark, and VRAM comparison tool
Developer Zack Ku announced ModelAnalysis.si, a free dashboard for comparing LLM benchmarks, API token pricing, VRAM requirements, and self-hosting economics. The tool supports side-by-side and radar-chart comparisons, API spend simulation, cloud-GPU break-even calculations, and deployment commands for Ollama and vLLM. According to Ku, the site draws on benchmark and pricing data covering more than 250 models, while a later post says it covers over 280 models. These coverage and accuracy claims have not been independently verified.
ModelAnalysis.si gives developers a single place to compare model quality, inference costs, and hardware requirements, though its benchmark, pricing, and model-coverage claims remain self-reported.
Hey everyone,
Every time a new model drops (like DeepSeek-R1 or the recent Kimi k3.1 Pro), finding reliable data usually means jumping between three or four places: checking leaderboard scores, digging up actual API pricing (inputs vs. outputs vs. caching), and guessing whether an open model will actually fit in VRAM with decent context.
I put together a clean dashboard to keep all of this in one spot: https://modelanalysis.si
Here is what it does:
- Benchmarks comparison: Instead of just standard MMLU, it pulls real-world benchmark data including SWE-bench Verified (actual GitHub issues resolved), LMSYS Arena-Hard, GPQA, and competition math. You can pick any models and overlay them on radar charts or compare them side-by-side.
- Price vs. Performance chart: A scatter plot showing which models deliver the best benchmark scores relative to their per-token API cost.
- VRAM calculator: Select model size, quantization (4-bit up to FP16), and context length to calculate the exact VRAM needed for model weights plus KV cache, along with ready-to-use Ollama and vLLM commands.
- Spend simulator: Drag sliders for monthly tokens and prompt cache percentage to project monthly API bills and see where self-hosting becomes cheaper.
It covers 280+ models, runs fast, requires no login, and has no ads or affiliate links.
Link: https://modelanalysis.si Benchmarks lab: https://modelanalysis.si/benchmarks/ Hardware/VRAM tool: https://modelanalysis.si/hardware/
Feedback or suggestions on other benchmarks you'd like to see are welcome.
submitted by /u/zackku [link] [comments]
#ModelAnalysis.si#OpenAI#Anthropic#DeepSeek#Qwen#Mistral#Meta#RunPod#Ollama#vLLM#SWE-bench Verified