Inference

How it works

Inference is the act of running a trained model to produce an output: you send a prompt in, the model computes, and a response comes out. It is the runtime counterpart to training. Training is a one-time (or periodic) expense that produces the model weights; inference is the recurring cost you pay on every request in production. For LLMs, inference cost and latency scale with the number of tokens processed and generated, and with the model size, which is why techniques like quantization and smaller distilled models matter for economics.

When it matters

Inference matters because it is where your ongoing AI bill and user-facing latency live. Design decisions (model size, context window usage, batching, caching, streaming) are all really inference-optimization decisions. It is also the surface attackers hit at runtime: every production inference call is a chance for prompt injection or abuse, which is why runtime monitoring is part of any serious AI security program.

Common misconceptions

• "The cost is in training." For most deployed products, inference is the dominant lifetime cost because it recurs per request.
• "Bigger models are always worth it." A smaller or quantized model that meets the bar is cheaper and faster at inference.
• "Inference is just a lookup." It is a full forward pass through the model for every request.