Retrieval-augmented generation for business knowledge bases

By Marco Lopes · February 8, 2023 · AI & Machine Learning

Most businesses already have a substantial knowledge base. It just does not behave like one. Answers are scattered across shared drives, internal wikis, support tickets, product specifications, and conversations. Finding the right document often requires knowing who wrote it, where they put it, and which version still applies.

At Kordspace, we see an opportunity to make that information easier to use without pretending a language model knows everything about a business. Retrieval-augmented generation, or RAG, combines document search with text generation: we retrieve relevant information first, then ask a model to answer using that evidence.

As of February 2023, the ingredients are available, but turning them into a dependable business tool still takes careful engineering. We would start with a narrow workflow, explicit permissions, and a testable definition of a good answer, not a promise to replace every internal search box.

What retrieval-augmented generation actually does

A language model generates text from patterns learned during training and information supplied in its prompt. On its own, it will not reliably know our latest onboarding process, a private product decision, or yesterday’s revised expense policy. Fluent wording is not evidence that an answer is correct.

RAG adds a retrieval step. When someone asks a question, the application searches an approved collection of documents, selects relevant passages, and includes them with the question in a model request. The model then produces an answer grounded in that supplied context.

For example, an employee might ask, “How do we request access to the staging environment?” Rather than generating a plausible procedure, the system retrieves the current access guide and summarizes its steps, linking back to the source.

This does not eliminate hallucinations. Retrieval can miss the right document, and generation can misread the evidence. It does give us a clearer engineering target: find authoritative material, preserve its meaning, and make the answer verifiable.

Start with one useful business workflow

We would resist beginning with “chat with everything.” A broad assistant inherits every inconsistency, outdated page, and access-control problem in the organization. A focused assistant gives us a manageable source collection and people who can judge whether its answers help.

Good starting points include internal IT guidance, product documentation, employee onboarding, or support-agent assistance. We would choose a workflow with recurring questions, identifiable document owners, and a reasonable consequence if the assistant gets something wrong.

Before building, we would ask:

• Who will use the assistant, and what decisions will its answers support?
• Which sources are authoritative, and who maintains them?
• What should happen when an answer is missing or ambiguous?
• Which information must never cross team or customer boundaries?

For a first release, we would keep the assistant advisory. It can explain a procedure or draft a response for review. Granting access, issuing refunds, or changing infrastructure introduces a separate action and authorization problem that a retrieval pipeline does not solve.

Prepare documents before choosing a model

The quality of a knowledge assistant depends heavily on the material it searches. We would inventory the source collection, remove obvious duplicates, identify superseded documents, and preserve useful structure during extraction.

Headings, lists, tables, and code blocks carry meaning. A table of account limits becomes misleading if extraction separates values from their column labels. Scanned PDFs may need optical character recognition, followed by checks on names, numbers, and formatting.

Next, we divide documents into passages that fit comfortably within the generation model’s context budget. We prefer boundaries that follow sections or ideas rather than arbitrary character counts. Some overlap can preserve continuity, but excessive overlap fills retrieval results with near-duplicates.

Every passage should retain metadata: document ID, title, source link, section, revision date, owner, and access restrictions. We would also keep enough provenance to reconstruct exactly which text was presented to the model. Without that, debugging a convincing but incorrect answer becomes guesswork.

Combine semantic retrieval with ordinary search

Embeddings represent text as numerical vectors. Passages with related meanings can appear near one another in that vector space, allowing a search for “getting a new laptop” to retrieve a page titled “Equipment procurement.”

We generate embeddings for document passages and store them in a searchable index. At query time, we embed the question using the same compatible embedding model and retrieve nearby passages.

Semantic search is useful, but we would not assume it replaces keyword search. Product codes, policy identifiers, acronyms, and exact error messages often benefit from lexical matching. A hybrid approach can combine both, then rank the resulting candidates.

Retrieval settings need testing. Returning more passages may improve coverage while also adding distracting evidence and increasing cost. We would compare candidate counts, chunk sizes, and ranking strategies against real questions. For questions about a specific version or department, metadata filters may matter more than semantic similarity.

A practical Google Cloud architecture

Google Cloud is our preferred starting point, although the same design can run on AWS or Azure. We would separate ingestion from question answering so document processing does not slow down interactive requests.

A straightforward Google Cloud implementation could use:

• Cloud Storage for approved source exports and extracted artifacts.
• Cloud Run for connector services, ingestion workers, and the application API.
• Pub/Sub to queue document changes and decouple processing stages.
• Vertex AI custom prediction endpoints to serve a suitable embedding model, with Vertex AI Matching Engine for vector retrieval when the workload justifies it.
• Firestore or Cloud SQL for document metadata, permissions, and processing state.

For generation, we can call an available language-model API or host a suitable model ourselves. The choice depends on quality, latency, licensing, data-handling terms, and operational capacity. We would not assume that every managed model service meets a particular organization’s confidentiality requirements.

On AWS or Azure, we would map these responsibilities to equivalent storage, compute, messaging, and search components rather than redesigning the whole application. Keeping provider-specific code behind clear interfaces makes later changes less disruptive.

Make answers accountable to their sources

The generation prompt should state the assistant’s job narrowly: answer from the supplied evidence, distinguish facts from uncertainty, and say when the material does not support an answer. Retrieved passages should have stable reference labels that the application can map to source links.

We would ask for citations next to the claims they support, not merely a list of documents beneath an otherwise unsupported response. We should validate that cited reference IDs exist, while recognizing that a valid citation does not prove the passage supports the claim.

For conflicting policies, we would prefer an explicit explanation over an invented compromise. The interface might say that two sources disagree and show their revision dates. Where authority rules are known, such as an approved policy taking precedence over a discussion thread, those rules should be encoded deliberately.

Abstention also belongs in the product design. “We could not find an approved answer” is useful when paired with relevant documents or a route to the responsible team. A similarity score alone is not a calibrated measure of answer confidence.

Treat permissions and retrieved text as security boundaries

A document being searchable does not mean every user may see it. We would carry access metadata through ingestion and enforce authorization during retrieval, before any passage enters the generation request. Hiding a source link after generation is too late if its contents have already influenced the answer.

Where an index cannot enforce the required restrictions directly, the retrieval layer must perform reliable authorization checks before assembling context. For stronger isolation requirements, separate indexes or storage boundaries may be appropriate.

We also need to treat retrieved content as untrusted data. A document could contain instructions such as “ignore the previous rules” or request disclosure of unrelated information. Prompt wording can help distinguish evidence from instructions, but it is not a complete defense. We would limit tool access and keep authorization outside the model.

Questions, logs, caches, and embeddings deserve the same review as source documents. We would examine retention, encryption, regional processing, provider terms, and who can inspect debugging traces before sending confidential information through the system.

Evaluate retrieval and generation separately

We would build a small evaluation set before polishing the interface. Subject-matter experts can provide representative questions, acceptable answers, and the passages that should support them. The set should include ambiguous questions, outdated terminology, missing answers, and requests for restricted information.

First, we test retrieval: does the correct evidence appear among the passages the model receives? If not, changing the generation prompt is unlikely to fix the underlying problem.

Then we test answers: are they correct, supported, complete enough, and appropriately cautious? We would inspect whether citations actually substantiate the associated claims and whether the assistant refuses to answer when evidence is absent.

Finally, we measure the user experience: response latency, cost per request, and whether people can verify answers efficiently. We would compare against the existing search workflow rather than judging generated prose in isolation. Every change to extraction, embeddings, ranking, or prompts should run against the same evaluation set.

Keep the knowledge current and costs understandable

An assistant grounded in stale documents still gives stale answers. We would track source revisions and reprocess changed passages, with scheduled reconciliation to catch missed events. Deleted documents and revoked permissions must propagate to indexes, metadata stores, and relevant caches.

Changing the embedding model usually means regenerating document embeddings and rebuilding or migrating the index. We would version the model, chunking rules, prompts, and index configuration together so results remain reproducible.

For cost control, we would inspect the full request path: query embedding, search, optional reranking, generation, and hosting. Supplying fewer, better passages often helps both cost and answer quality. Caching can reduce repeated work, but cache keys and invalidation rules must account for permissions and document changes. Shared answers must never become a shortcut around access checks.

A short launch checklist

Before a limited rollout, we would confirm that:

• The use case, approved sources, and document owners are defined.
• Extraction and chunking preserve important document meaning.
• Authorization is enforced before evidence reaches the model.
• Answers include usable citations and a clear no-answer path.
• Tests cover incorrect, missing, conflicting, and restricted information.
• Updates, deletions, monitoring, and operating costs have owners.

We would then release to a small group, review failures with domain experts, and expand only when the evidence supports it.

Build a knowledge assistant, not an oracle

RAG is valuable because it connects language generation to information a business can inspect and maintain. Its reliability comes from the whole system: document quality, retrieval, permissions, grounded answers, and ongoing evaluation. The model is an important component, but it is not the knowledge-management strategy.

At Kordspace, we work remotely across collaborative software, AI, Web3, hardware, and design, bringing product thinking and implementation together. If we can help turn your business knowledge into a practical, source-backed assistant, start a project with us, or explore our related services to see how we approach the work.