Private RAG on One 6-8GB GPU: A Small Qwen Model, nomic-embed and Qdrant

If your documents can't leave the building, a cloud API is ruled out before anyone compares models. That covers privileged legal files, patient records and a lot of regulated financial material. The usual next idea is a GPU server costing five figures, and that decision often stalls for a quarter.

There is a smaller first step. A single consumer GPU with 6 to 8GB of memory can run a private document Q&A system for one team. It answers from your files, cites the page, and sends nothing to a third party.

The stack has three parts. A Qwen-family model of roughly 3 billion parameters writes the answers. nomic-embed-text turns text into searchable vectors. Qdrant stores and searches them.

When this stack is the right call

Many pilots fail at the same point. They run well on ten sample files, then nobody checks whether they work on the real corpus. Building small on real data avoids that trap.

What fits in 6GB, and what needs 8GB

Estimate memory before you buy anything. Model weights take roughly parameters × bits per weight ÷ 8. A 3-billion-parameter model at Q4_K_M quantization, which averages a little under 5 bits per weight, needs about 1.7 to 2.0GB. Quantization stores the weights at lower precision so the model fits on smaller cards. Hugging Face lists the base 3B Qwen model at 3.09B parameters, with a 32,768-token context.

The rest goes to the context cache, which holds the question and the retrieved passages. It grows with the context window and with every parallel request. The embedding model adds a few hundred MB. Qdrant runs in system RAM, not on the GPU. Qdrant documents that its quantization keeps compressed vectors in RAM and moves the full-precision originals to disk.

On a 6GB card, plan for Q4_K_M, an 8K context and one or two parallel requests. An 8GB or 12GB card buys more parallel requests or a higher-precision quantization. Don't trust our estimate on your hardware. Run nvidia-smi under load and read the real number.

Setting up Ollama

Ollama runs quantized models locally and exposes an OpenAI-compatible API. Model tags in its library change between releases. Pick a Qwen-family instruct model of about 3B parameters from the library, check the exact tag, and set it once.

# Install Ollama (Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Set the exact tag you picked from the Ollama library
export LLM_TAG="<qwen-3b-tag-from-library>"

ollama pull "$LLM_TAG"
ollama pull nomic-embed-text

# Confirm the model is on the GPU, not the CPU
ollama run "$LLM_TAG" "Hello" > /dev/null
ollama ps    # PROCESSOR column should read 100% GPU

If ollama ps shows a CPU/GPU split, the model didn't fit. Shrink the context or the quantization until it reads 100% GPU.

A model profile for grounded answers

cat > Modelfile << EOF
FROM ${LLM_TAG}

SYSTEM """You are a precise knowledge assistant. Answer ONLY from the provided context.
If the answer is not in the context, say exactly: "This information is not in the available documents."
Cite sources using [Source N]. Be concise. Do not speculate."""

# Low temperature for factual answers
PARAMETER temperature 0.1
PARAMETER top_p 0.9
# Context window: the main VRAM lever after quantization
PARAMETER num_ctx 8192
# Cap answer length
PARAMETER num_predict 512
PARAMETER repeat_penalty 1.1
EOF

ollama create verel-rag -f Modelfile

Some newer Qwen models have a reasoning mode. It adds latency and buys little when the answer sits in the retrieved passage. Check the model card and turn it off for this workload.

Qdrant, self-hosted

docker run -d --name qdrant \
  -p 6333:6333 -p 6334:6334 \
  -v $(pwd)/qdrant_storage:/qdrant/storage:z \
  qdrant/qdrant:latest

curl http://localhost:6333/healthz

Pin a specific image version in production rather than latest. An unplanned upgrade on restart is not something you want to debug at 9am.

Ingestion, with hybrid search built in from day one

Semantic search finds passages with similar meaning. It is weak on exact strings: product codes, clause numbers, drug names. Keyword search (BM25) catches those. Running both and merging the results is called hybrid search. It's cheaper to set up at ingestion than to retrofit later, so we create both vector types now.

Two details trip people up. First, nomic-embed-text expects task prefixes: search_document: on stored text and search_query: on questions. Leave them off and retrieval quality quietly drops. Second, LangChain's splitter counts characters, not tokens. The nomic-embed-text model card on Hugging Face lists an 8192-token context length.

import uuid
import ollama
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import DirectoryLoader, PyPDFLoader
from fastembed import SparseTextEmbedding
from qdrant_client import QdrantClient, models

COLLECTION  = "enterprise_docs"
EMBED_MODEL = "nomic-embed-text"
EMBED_DIM   = 768

qdrant = QdrantClient(host="localhost", port=6333)
bm25   = SparseTextEmbedding(model_name="Qdrant/bm25")

if not qdrant.collection_exists(COLLECTION):
    qdrant.create_collection(
        collection_name=COLLECTION,
        vectors_config={
            "dense": models.VectorParams(size=EMBED_DIM, distance=models.Distance.COSINE),
        },
        sparse_vectors_config={
            "bm25": models.SparseVectorParams(modifier=models.Modifier.IDF),
        },
    )
    # Index the tenant field so filtered searches stay fast
    qdrant.create_payload_index(COLLECTION, "namespace", models.PayloadSchemaType.KEYWORD)

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1500,      # characters, roughly 350-400 English tokens
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""],
)

def embed_docs(texts: list[str]) -> list[list[float]]:
    prefixed = [f"search_document: {t}" for t in texts]
    return ollama.embed(model=EMBED_MODEL, input=prefixed)["embeddings"]

def embed_query(text: str) -> list[float]:
    return ollama.embed(model=EMBED_MODEL, input=[f"search_query: {text}"])["embeddings"][0]

def ingest_directory(path: str, namespace: str = "default", batch_size: int = 32):
    docs   = DirectoryLoader(path, glob="**/*.pdf", loader_cls=PyPDFLoader).load()
    chunks = splitter.split_documents(docs)

    for i in range(0, len(chunks), batch_size):
        batch  = chunks[i : i + batch_size]
        texts  = [c.page_content for c in batch]
        dense  = embed_docs(texts)
        sparse = list(bm25.embed(texts))

        points = [
            models.PointStruct(
                id=str(uuid.uuid4()),
                vector={
                    "dense": d,
                    "bm25": models.SparseVector(
                        indices=s.indices.tolist(), values=s.values.tolist()
                    ),
                },
                payload={
                    "text": c.page_content,
                    "source": c.metadata.get("source", "unknown"),
                    "page": c.metadata.get("page", 0),
                    "namespace": namespace,
                },
            )
            for c, d, s in zip(batch, dense, sparse)
        ]
        # Upsert per batch so a failure mid-run doesn't lose everything
        qdrant.upsert(collection_name=COLLECTION, points=points)

Time a full ingestion run on your own corpus. Scanned PDFs need OCR first, and that step is usually slower than everything above combined.

Notice where this stack stops. The only filter is the tenant field. There are no dates, no draft-or-approved status and no per-group permissions on a chunk. A single team's pilot can live with that. Anything shared across departments needs access groups, dates and status on every chunk before it goes wider.

Retrieval and generation

Qdrant's Query API runs both searches as "prefetches" and merges them with Reciprocal Rank Fusion. That method combines the two ranked lists, so no score calibration is needed. Put the tenant filter on each prefetch, so no other tenant's chunk can reach the merge.

from openai import OpenAI

llm = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

def retrieve(query: str, namespace: str = "default", top_k: int = 6) -> list[dict]:
    tenant = models.Filter(must=[
        models.FieldCondition(key="namespace", match=models.MatchValue(value=namespace))
    ])
    s = list(bm25.query_embed(query))[0]

    result = qdrant.query_points(
        collection_name=COLLECTION,
        prefetch=[
            models.Prefetch(query=embed_query(query), using="dense", filter=tenant, limit=20),
            models.Prefetch(
                query=models.SparseVector(indices=s.indices.tolist(), values=s.values.tolist()),
                using="bm25", filter=tenant, limit=20,
            ),
        ],
        query=models.FusionQuery(fusion=models.Fusion.RRF),
        limit=top_k,
        with_payload=True,
    )
    return [
        {"text": p.payload["text"], "source": p.payload["source"], "page": p.payload["page"]}
        for p in result.points
    ]

def answer(query: str, namespace: str = "default") -> dict:
    chunks = retrieve(query, namespace)
    if not chunks:
        return {"answer": "This information is not in the available documents.", "sources": []}

    context = "\n\n---\n\n".join(
        f"[Source {i}: {c['source']}, p.{c['page']}]\n{c['text']}"
        for i, c in enumerate(chunks, 1)
    )
    prompt = f"Context:\n{context}\n\nQuestion: {query}\n\nAnswer with [Source N] citations:"

    reply = llm.chat.completions.create(
        model="verel-rag",
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content

    cited = [
        {"index": i, "file": c["source"], "page": c["page"]}
        for i, c in enumerate(chunks, 1) if f"[Source {i}]" in reply
    ]
    return {"answer": reply, "sources": cited}

An answer with no citation at all is a warning sign. Log those cases and review them weekly. They show you where the model is filling gaps from memory.

Estimating response time before users do it for you

We don't publish benchmark numbers for this stack. They vary too much with the card, the driver, the corpus and the answer length. Measure your own with this formula:

Response time ≈ query embedding + search + (prompt tokens ÷ prefill speed) + (answer tokens ÷ generation speed) + queue wait

Here's a worked example using your own variables. Six chunks of about 375 tokens give a context payload of 2,250 tokens. With instructions, the prompt is slightly larger. A 200-token answer at your measured generation speed G takes 200 ÷ G seconds. Ollama prints prefill and generation rates when you run ollama run --verbose.

Queue wait is the term that hurts. Ollama's OLLAMA_NUM_PARALLEL setting controls how many requests run at once. Each parallel slot reserves its own context memory, so a 6GB card supports few slots. When users exceed the slots, they wait in line. Load-test with your real peak user count, not an average.

Pre-launch checklist

• ▸ ollama ps shows 100% GPU under real load, at your chosen context size
• ▸ nvidia-smi -pm 1 enables persistence mode, so the driver stays initialized between requests
• ▸ OLLAMA_KEEP_ALIVE set so the model isn't unloaded during quiet periods
• ▸ Ollama and Qdrant run as services that restart on crash
• ▸ Qdrant image version pinned, data on a persistent volume
• ▸ Daily Qdrant snapshots (POST /collections/{name}/snapshots), copied off the machine
• ▸ Tenant filter applied on every search path, tested with a cross-tenant query that must return nothing
• ▸ Rate limit per user at the API layer, so one person can't block the queue
• ▸ A test set of real questions with known answers, run before every model or chunking change
• ▸ Monitoring on GPU memory, queue depth and the share of answers with no citation

Frequently asked questions

Can this run on a CPU-only server? Yes, but generation becomes several times slower, and answers take many seconds. That suits overnight batch jobs, like summarizing a document backlog. It does not suit someone waiting at a desk for an answer.

What if our documents are in Arabic? Switch the embedding model before anything else. multilingual-e5-large is a proven choice for Arabic and English retrieval, and the normalization rules to apply before indexing Arabic matter as much as the model. It uses its own prefixes (query: and passage: ) and outputs 1,024 dimensions, so update the collection size. Qwen-family models handle Modern Standard Arabic reasonably well. Test dialect and mixed-script documents on your own question set. Our post on Arabic AI in production covers tokenization, dialects and search that finds the right document.

Is 4-bit quantization accurate enough for regulated work? In RAG, most wrong answers trace back to retrieval, not model precision. If the right passage never reaches the model, no precision setting saves the answer. Don't argue it in theory. Run your test set at Q4_K_M and at a higher quantization, then compare. Keep 4-bit only if the scores match on your documents.

Start with the test set, not the hardware order. Write fifty real questions with the right answer and page for each, drawn from what your team asks in a normal week. Run them through this stack on one card. The result tells you whether to roll it out, upgrade the GPU, or fix retrieval first.