Small language models, often called SLMs, are AI models with fewer parameters than frontier models like GPT-4o, Claude Opus, or Gemini Ultra. There is no strict cutoff, but models under roughly 10 billion parameters are usually described as small. The most capable small models today can run on a laptop, a phone, or an edge device without a cloud API.
For most of 2023, the gap between small and large models was large enough that the choice was obvious: use the big one. In 2025 and 2026, that gap has narrowed significantly on structured tasks. Small models are now worth evaluating for any workflow where cost, latency, privacy, or local deployment matters.
What makes small models attractive
- Lower cost per token, often by an order of magnitude versus frontier models.
- Much lower latency, which matters for interactive or real-time applications.
- Can run locally on a laptop or device, keeping data entirely on-premise.
- Lower energy and compute requirements for high-volume batch tasks.
- Easier to fine-tune on specific domains with less data and compute.
The notable small models
Gemma (Google DeepMind)
Gemma is Google DeepMind's family of open models. The family spans a range of sizes from 2B to 27B parameters, with versions optimized for instruction following. Google positions Gemma as a capable, efficient open model for developers who want local inference or more deployment flexibility than hosted proprietary models allow.
Phi (Microsoft)
Microsoft's Phi family is built around a hypothesis that high-quality training data matters more than raw parameter count. Phi-4, announced in late 2024, demonstrated strong benchmark performance for its size class, particularly on reasoning and coding tasks. Microsoft targets Phi at mobile and edge use cases as part of its Azure AI strategy.
Llama small variants (Meta)
Meta's Llama 3.2 release introduced 1B and 3B parameter models specifically designed for edge and mobile devices. These are genuinely small models designed to run efficiently on hardware that cannot handle even a 7B model. They lose capability compared to larger Llama variants but are designed for the constraints of on-device deployment.
Qwen small variants (Alibaba)
The Qwen model family from Alibaba includes small variants that perform well on multilingual tasks. If your workflow involves non-English languages, particularly Asian languages, the Qwen family is worth evaluating because its multilingual training data coverage is strong at the smaller model sizes.
SmolLM (Hugging Face)
SmolLM2 from Hugging Face targets the extreme small end: 135M to 1.7B parameters. These models are designed to run in-browser or on very constrained devices. At this scale, capability is limited, but they remain useful for specific narrow tasks like classification, keyword extraction, or simple intent detection on a device with no connectivity.
Where small models work well
- Classification: labeling support tickets, triaging emails, categorizing content.
- Extraction: pulling structured data from documents or forms.
- Short text generation: drafts, summaries, response templates.
- Translation: for common language pairs, quality is often good enough.
- Intent detection: understanding what a user wants in a chat or voice interface.
- Routing: deciding which tool or category to send input to.
Where frontier models still earn their cost
- Complex multi-step reasoning where the chain of logic is long.
- Long-context tasks that require synthesizing thousands of tokens at once.
- Ambiguous instructions where reading implicit meaning accurately matters.
- High-stakes outputs where a small reasoning error has significant consequences.
- Novel domains with unusual vocabulary or specialized knowledge.
On-device use: where the real opportunity is
The most distinctive use of small models is running them without any network connection. This matters for medical devices, industrial equipment, offline-capable mobile apps, privacy-first local tools, and anything where cloud latency is unacceptable. Tools like Ollama and LM Studio make it practical to run small models locally without needing to manage GPU infrastructure.
The question is not whether a small model is as smart as GPT-4o. The question is whether it is smart enough for this specific task at a fraction of the cost.
Dera's recommendation
Evaluate small models for every task before defaulting to a frontier model. For classification, extraction, routing, and structured generation, there is a good chance a small model is good enough and materially cheaper. For complex reasoning, long context, and judgment-heavy outputs, frontier models continue to provide value that small models cannot match consistently.