Running LLM Inference on CPUs: When Cores Are Enough

The default assumption in AI infrastructure is simple: large language models need GPUs. For training and for high-throughput serving, that is usually true. For a surprising number of practical applications, however, CPU-only inference is good enough, and it can be easier to obtain, cheaper to run and simpler to operate.

Projects such as llama.cpp have made CPU inference practical with optimized code and aggressive quantization. This guide explains what determines CPU inference speed, which workloads fit and how to test before renting hardware.

Why CPU Inference Is Back on the Table

Several developments make CPUs viable for many inference tasks. Smaller open models have become much more capable, so a model with a few billion parameters can handle classification, extraction, summarization and internal assistants well. Quantization shrinks models dramatically. And inference engines now use modern CPU instructions efficiently: llama.cpp, for instance, supports AVX, AVX2, AVX-512 and AMX on x86 processors and integer quantization from 1.5 to 8 bits.

The workloads that fit best share a pattern: moderate request volumes, tolerance for a few seconds of response time, or batch processing where latency hardly matters. Internal tools, document processing pipelines, nightly enrichment jobs and development environments are typical examples. Data privacy is another frequent reason: running a model on your own server keeps prompts and documents inside infrastructure you control. Availability is a practical factor, too: CPU servers are widely available in many locations, which makes it easier to place inference close to users or data.

Memory Bandwidth Sets the Speed Limit

To understand CPU inference, separate its two phases. Prompt processing, where the model reads the input, is compute-bound and benefits from many cores. Token generation, where the model produces the answer one token at a time, must read most of the model's weights from memory for every token. That makes generation speed largely a function of memory bandwidth.

This is why server processors with many memory channels perform far better than desktop processors with the same core count. A platform with eight or twelve memory channels can feed the cores much more data per second than a dual-channel desktop system. When you compare servers for CPU inference, memory channels and memory speed deserve as much attention as cores.

On dual-socket systems, keep memory locality in mind. Running a model across two sockets can be slower than expected if threads constantly reach for memory attached to the other processor. Many teams start with a single-socket server to keep things simple. Thread settings matter as well. Using every core is not always fastest during generation, because extra threads compete for the same memory bandwidth, so test a few thread counts rather than assuming that more is better.

Quantization and Model Size

Quantization stores model weights with fewer bits, which reduces memory use and increases speed, because less data has to move for each token. The trade-off is some loss of quality, which grows as the bit count falls. Moderate quantization levels usually keep quality close to the original for many tasks, while very aggressive levels can noticeably hurt accuracy. llama.cpp uses its own GGUF format, and many open models are published in it already, often in several quantization levels, which makes it easy to compare options side by side.

Scenario	CPU-only viable?	What to check
Small model for classification or extraction	Usually yes	Accuracy on your own test set
Mid-sized model for an internal assistant with a few users	Often yes	Tokens per second at typical prompt length
Batch summarization overnight	Yes	Total throughput per hour
Large model with many concurrent users	Rarely	GPU likely needed
Real-time voice or chat at scale	Rarely	Latency targets, GPU likely needed

Always evaluate quality on your own data. Benchmarks published for a model are a guide, but your prompts and documents are what matter.

Concurrency and Latency Targets

A single user chatting with a model is a very different workload from twenty users at once. On CPUs, concurrent requests compete for the same memory bandwidth, so total throughput rises more slowly than the number of users. Define your targets before testing: acceptable time to the first token, acceptable generation speed and the number of simultaneous requests during peak hours.

If requests can wait, a queue that processes them in order often gives better results than trying to serve everyone at the same time. llama.cpp includes an OpenAI-compatible API server, which makes it straightforward to put a CPU-backed model behind existing tools and measure how it behaves under realistic load. Long prompts deserve separate attention: retrieval pipelines that paste several documents into every request spend much of their time in prompt processing, where more cores help directly. Monitor memory headroom closely, because each concurrent request needs memory for its context, and long prompts combined with many users can exhaust RAM even when the model itself fits comfortably.

Testing Before You Commit

The most reliable way to decide is to test your actual models and prompts on the hardware you are considering:

1. Pick two or three candidate models and quantization levels.
2. Build a test set of real prompts with expected outputs.
3. Measure prompt processing speed, generation speed and quality for each combination.
4. Repeat the tests with your expected number of concurrent users.
5. Compare the results with the same workload on a GPU, if one is available.

Include a test of the full pipeline, not just the model, because tokenization, retrieval and post-processing can add noticeable time on CPU-only systems.

A short AMD EPYC server rental is a practical way to run these tests on server-class hardware with many cores and memory channels, without buying anything. Providers such as Unihost offer EPYC servers across several generations, so you can check how much a newer platform with faster memory changes the results for your workload.

Record the exact model files, quantization levels, thread settings and software versions with every result. Inference engines improve quickly, and a test repeated a few months later on the same hardware can produce noticeably different numbers.

CPU inference will not replace GPUs for large-scale serving, but it covers more use cases than many teams assume. Understand that memory bandwidth drives generation speed, choose model size and quantization by measured quality, define concurrency targets and test on real hardware before deciding.