llama.cpp
by ggml (Hugging Face)
C/C++ engine for running LLMs locally, with an OpenAI-compatible server
llama.cpp is an open-source inference engine for running large language models on local hardware, using the GGUF model format. It runs on CPUs and on Apple Silicon, CUDA, HIP, Vulkan, and SYCL GPUs, and includes a CLI plus a server with an OpenAI-compatible API and a built-in web UI. It underlies many other local LLM tools.
Key features
- GGUF models and quantization
- Metal, CUDA, HIP, Vulkan, and SYCL backends
- OpenAI-compatible API server
- Built-in web chat UI
- Download models directly from Hugging Face
Similar tools
Ollama
Run open large language models locally with a simple CLI and API
Claude Code
Agentic coding tool from Anthropic that works in your terminal
OpenAI Codex
OpenAI's coding agent for the terminal, IDE, desktop, and cloud
OpenCode
Open-source coding agent for the terminal, desktop, and IDE
Cline
Open-source coding agent for VS Code and the terminal
Cursor
AI-native code editor built on VS Code
