Don't stop here
Hand-picked guides our readers explore right after this one.
Stunning image generation with Midjourney prompt mastery
Read the guideExpert guide to Claude prompts with XML tags, artifacts, and complex reasoning
Read the guideCreate stunning AI images with Flux by Black Forest Labs using structured prompt techniques
Read the guideThe stage where a trained AI model is actually used to produce output, as opposed to training, when it learns.
Inference is the moment you actually use an AI model, when it takes your prompt and produces an answer. It is the opposite of training, which is the earlier, one-time process of teaching the model from data. Training a model is like a student studying for years; inference is that student sitting an exam and answering a question. Every time you send a message to ChatGPT, you are running inference.
Training is writing and memorizing a cookbook after years of practice; inference is cooking a specific dish when a customer orders it. The cookbook (the model's weights) does not change while you cook. Each order (prompt) is handled by using what was already learned, quickly and repeatedly.
Inference is the forward pass of a trained model: given an input, the model computes outputs using its fixed learned parameters, without updating them. For LLMs, inference generates tokens autoregressively, each new token conditioned on all previous ones, which is why longer outputs cost more and take longer. Inference performance is measured in latency (time to first token, tokens per second) and throughput. It is optimized with techniques like quantization, batching, KV-caching and specialized hardware (GPUs, TPUs, inference accelerators). At scale, inference, not training, often dominates a deployed model's total compute cost.
Each reply you get is one inference run on the trained model.
Autocomplete, translation and voice assistants all run inference on the fly.
API pricing per token is essentially the cost of inference, charged per use.
Running smaller models directly on phones or laptops, no server round-trip.
Training is the one-time, expensive process of teaching a model from data, adjusting its parameters. Inference is using the finished model to answer a query, with parameters fixed. Training happens once (or occasionally); inference happens every time anyone uses the model, which is why it dominates ongoing costs.
Each request runs the full model, and large models require significant compute per token generated. One request is cheap, but millions of users sending many requests add up fast. That is why providers optimize heavily with batching, caching and efficient hardware, and why smaller or distilled models are attractive for high-volume use.
Increasingly, yes, for smaller models. Phones and laptops can run compact models locally for privacy and offline use. The largest frontier models still need powerful servers, but the trend is toward capable models small enough to run on-device, a field often called edge or local inference.
The basic unit AI models use to process text, roughly corresponding to word parts, common words, or character sequences.
📚A neural network trained on massive text data to understand and generate human-like language.
🎯The process of further training a pre-trained AI model on specific data to specialize its behavior for a particular task or domain.
📈A branch of AI where systems learn patterns from data and improve at a task, instead of being explicitly programmed with rules.
Our free AI course teaches you to use these ideas in real projects.
Start Free AI Course →