Why Your On-Prem LLM Is Slow: The Serving-Stack Levers in Order

A company buys a GPU server for private AI, and the verdict arrives in week one. A few colleagues try it and the answers crawl onto the screen. Someone decides on-prem "isn't ready." That conclusion is usually wrong, and it is an expensive one to act on.

The hardware is often fine. The bottleneck is the software that serves the model.

This matters because a slow internal AI tool never fails loudly. People just stop opening it. That quiet drop-off is how many AI pilots end. The system works on paper, usage fades after a month, and the server you paid for sits mostly idle. On-premise AI is chosen for privacy and control. It only pays back if people use it.

Below are the levers in the order we check them. The first two cost little and fix most complaints. The rest matter as load grows.

Measure two numbers before changing anything

Pull two measurements from your own system first. Without them, every fix is a guess.

• ▸Time to first token: how long a user stares at a blank screen before the answer starts. This is what people mean when they say "slow."
• ▸Throughput under load: the total text the server produces per second while many people use it at once. This sets how many users one server can carry.

These two fail for different reasons. If the first token is slow even with one user, the prompt is usually too long. If one user is fine and twenty users break it, look at the serving engine. Teams that mix the two up end up buying a second GPU they didn't need.

Lever 1: the serving engine

This is the biggest single variable, and teams often pick it by accident. Many deployments start on whatever installed most easily on a laptop and never move off it.

The core issue is concurrency. Basic serving setups handle requests largely one after another. With fifteen people asking questions, the fifteenth waits behind fourteen others. Production engines like vLLM, SGLang and NVIDIA's TensorRT-LLM change this with two techniques:

• ▸Continuous batching. The engine slots each new request into the GPU's work as soon as there is room. It doesn't wait for a whole batch to finish. So throughput rises with load instead of falling. Hugging Face documents continuous batching in its Text Generation Inference server as a way to raise total throughput.
• ▸Paged working memory. Each conversation needs scratch memory on the GPU, called the KV cache. Managing it in small pages, the way an operating system manages RAM, stops waste and fits more users at once. The vLLM team's PagedAttention paper (Kwon et al., 2023) reported 2 to 4 times the throughput of the systems it was compared against, at similar latency. The vLLM team's launch post claimed up to 24x the throughput of Hugging Face Transformers, with no model changes.

How the three production engines differ, in practical terms:

• ▸vLLM is the right default. It supports the most models, has the largest community, and needs no compile step.
• ▸SGLang is strong for RAG and chat. It reuses the shared opening of prompts, such as a long system instruction, so the model doesn't re-read it on every request.
• ▸TensorRT-LLM aims for maximum speed on NVIDIA hardware, for a model you rarely change. In return you compile each model version and take on more operational work.

Ollama and llama.cpp still have a place. For development, laptops and a handful of users, they are excellent. For a shared service with more than a few people using it at once, a production engine is the right choice.

You will find published speed rankings for these engines, but they shift with hardware, model and settings. Benchmark on your own prompts before trusting any ranking.

Lever 2: how much text you send the model

This one surprises people. For RAG systems, the ones that answer from your documents, it often beats every other lever.

The model has to read the entire prompt before it writes a single word. That reading step is called prefill, and its cost rises with prompt length. For very long prompts it rises faster than the length does. If your system pastes fifteen document excerpts into every question, every user waits while all fifteen are read.

Check this with your own numbers:

Prompt tokens per question = system instructions + (excerpts retrieved × tokens per excerpt) + chat history

If retrieved excerpts make up most of that total, you have probably found your latency. The fix is selective retrieval. Fetch a wide set of candidates first. Then a reranker, a small second model that scores relevance, keeps the best three or four, and only those go to the main model. Answer quality often improves as well, because the model has less irrelevant text to get confused by.

The other half of this lever is which model reads the prompt at all. Many internal questions are short lookups that a small model answers as well as a large one. If you route easy questions to a smaller model and keep the large one for the hard ones, the large model's queue shortens for everybody.

With cloud APIs, the same change cuts your bill, since you pay per input token. On your own hardware, it frees GPU capacity for more users.

Lever 3: quantization

Quantization stores the model's internal numbers at lower precision. For the business, that means the same model fits in less GPU memory. That leaves room for more simultaneous users, and the model usually runs faster too.

You can do the memory sums yourself:

Weight memory ≈ number of parameters × bytes per parameter

At 16-bit precision each parameter takes 2 bytes. So a 14-billion-parameter model needs about 28 GB for its weights alone. 8-bit formats halve that. 4-bit formats such as AWQ, GPTQ or GGUF Q4 cut it to roughly a quarter, plus some overhead. Every gigabyte freed goes to working memory, which is what lets more people use the system at once.

Quality cost varies by model and task. 8-bit usually performs close to the original. 4-bit loses more, and the loss tends to show first in long reasoning chains and exact formatting. RAG answering is usually more forgiving, because the model is mostly restating text it was given.

"Usually" is not a guarantee. Score the quantized model on a fixed set of your real questions before it goes live. Our RAG evaluation framework sets out how to build that test set.

Lever 4: engine settings

Choosing vLLM or SGLang is only half the job. Their defaults suit general use, not your workload. Three settings matter most:

• ▸Prefix caching. If every request opens with the same long instruction, cache it once. SGLang does this automatically. vLLM supports it too, so confirm it is switched on in your version.
• ▸Maximum context length. If you set this to the model's full advertised window, the engine reserves memory you may never use. Cap it at what your real prompts need.
• ▸GPU memory share. The engine claims a fixed fraction of GPU memory for working memory. Set it too low and capacity goes to waste. Too high, and the server can crash when another process needs the card.

None of these cost money. They cost an afternoon from someone who knows where to look.

Lever 5: speculative decoding

Speculative decoding pairs your main model with a small, fast draft model. The draft guesses the next few words, and the main model checks all of them in one pass. Correct guesses are kept almost for free, and wrong ones are thrown away. The final output matches what the main model would have written alone.

The original paper by Leviathan et al. (2023) reported generation 2 to 3 times faster on its test models. Your gain depends on how often the draft guesses right. That rate is high for predictable text like code or templated replies. It drops for open-ended writing. NVIDIA reported throughput gains of up to 3.61x from TensorRT-LLM speculative decoding with a 405B Llama model as the target.

Use it when you serve one fixed model at steady volume and can measure the hit rate on real traffic. Skip it if you swap models often. It means one more model to run, monitor and upgrade.

Lever 6: using more than one GPU

With several GPUs, you have two options.

• ▸Replication: put a full copy of the model on each GPU and split users between the copies. If the model fits on one card, this is usually the better choice for throughput, because the copies never wait on each other.
• ▸Sharding: split a single model across several GPUs. You need this when the model is too big for one card. The GPUs then exchange data constantly, so the connection between them matters. A direct GPU link like NVLink helps, and ordinary PCIe slots are slower.

A common mistake is sharding a model that would have fit on one card. You pay the communication overhead and get nothing for it.

Diagnosing your slowdown

Find your symptom, then go to its lever. Work down the table in order. Earlier fixes often make later ones unnecessary.

What this does to the business case

On-prem AI has a different cost shape from cloud APIs. The server is a fixed cost. Each extra question costs little beyond power. So your real unit cost is:

Cost per answer = (monthly hardware cost + power + support) ÷ answers served per month

A tuned serving stack raises the bottom of that fraction. So does a fast system, because people actually use it. Both push your cost per answer down on hardware you already own. Our piece on on-premise AI ROI in the UAE and Saudi Arabia runs that calculation in more depth.

Frequently asked questions

What is the smallest GPU that works for a shared on-prem deployment? Add up three things: the weight memory from the formula above, working memory for your expected users, and some headroom. A 24 GB card comfortably holds a 7B to 14B model at 4-bit, with room left for several users. At 4-bit, a 70B model's weights alone take roughly 35 to 40 GB. That means a 48 GB card or two GPUs working together. For a single team on one consumer card, the 6 to 8 GB build shows what fits and what does not.

Is Ollama ever the right choice in production? Yes, for a single user, a small team, or a device at the edge where simple setup matters more than throughput. It stops being the right choice once several people use the system at the same time. That is the point to move to vLLM or SGLang.

Is TensorRT-LLM worth the extra complexity? It pays off for a model you will run unchanged on NVIDIA hardware at high volume. In that case the compile step is a one-time cost per model version. If you are still testing models or switching them every few months, vLLM or SGLang will get you to production faster with less upkeep.

Your next step this week: pull one day of logs. Note the average prompt length and the engine serving the model. Those two facts tell you which row of the table you are in.