ChatGPT changed client expectations: building LLM features responsibly

By Marco Lopes · January 15, 2023 · AI & Machine Learning

ChatGPT has changed the starting point for conversations about artificial intelligence. Instead of asking whether software can understand a paragraph, people are asking why their software cannot already explain documents, draft responses, or answer questions conversationally.

We understand the excitement. A natural-language interface makes capabilities that once felt abstract immediately tangible. But a convincing conversation is not the same thing as a dependable product feature.

At Kordspace, we work as a remote collaborative software, AI, Web3, hardware, and design studio. Our job is to connect technical possibility with useful experiences and maintainable systems. Right now, that means helping teams separate what a large language model can demonstrate from what a business can responsibly promise.

Here is how we approach that work in January 2023.

Reset expectations without losing momentum

We start by distinguishing ChatGPT, the hosted conversational product, from the models and developer interfaces available for building applications. As of this writing, we would not plan a production integration around an assumed public ChatGPT API. Existing language-model APIs offer useful capabilities, but their behavior and integration requirements need their own assessment.

We also separate fluency from accuracy. A model can produce a polished explanation that contains unsupported claims. It can follow an instruction in one example and miss an important constraint in another. A successful demonstration proves that something is possible, not that it is reliable across the intended workload.

Our response is not to dismiss the technology. We turn broad expectations into bounded questions: what task are we improving, what information is available, and what happens when the answer is wrong?

Choose a narrow job before choosing a model

“Add AI” is not a product specification. We prefer a concrete workflow: draft a support reply from an approved article, summarize an internal note for review, or suggest categories for an incoming request.

For each candidate, we define the user, the input, the expected output, and the acceptance criteria. We also identify the existing process. Without that baseline, we cannot tell whether the feature saves effort or merely moves effort into checking its output.

A useful first feature usually has reversible consequences and an obvious reviewer. Drafting text is easier to contain than sending it automatically. Suggesting a category is safer than changing an account’s permissions.

We keep open-ended advice in sensitive domains out of an initial release unless the necessary expertise, oversight, and safeguards are already in place. A disclaimer alone does not make a risky workflow responsible.

Put the model behind an application boundary

We treat the model as one component, not as the application’s decision-making authority. The surrounding software remains responsible for authentication, authorization, data access, validation, and consequential actions.

A practical request path looks like this:

1. We authenticate the user and check the requested operation.
2. We retrieve only information that user may access.
3. We assemble a bounded prompt with clear instructions and context.
4. We call the model through a server-side integration.
5. We validate the response before displaying it or passing it onward.
6. We record operational signals without unnecessarily retaining sensitive content.

Google Cloud is our preferred starting point. A service on Cloud Run, secrets in Secret Manager, and appropriately configured monitoring can provide a straightforward foundation. AWS or Azure can serve the same architectural needs when they fit an existing environment better.

Hosting our application on Google Cloud does not mean an external model provider processes data there. We map that separate boundary explicitly, including network paths, credentials, contractual terms, and data handling.

Ground answers in material we control

When a feature needs to answer questions about an organization, we do not expect a general-purpose model to know its current policies or private documentation.

Instead, we can retrieve relevant passages and include them with the request. Depending on the material, ordinary search may be enough. Embeddings can help find passages whose meaning matches a question even when the wording differs.

The important work is not just choosing a search technique. We need usable document boundaries, source identifiers, update handling, and permission checks before retrieved content reaches the model.

We ask for answers supported by the supplied material and make the sources visible. We also design a legitimate “we could not find enough information” outcome.

Retrieval reduces some uncertainty; it does not eliminate fabrication. A generated citation can itself be wrong, so we check that references correspond to actual retrieved sources rather than trusting their appearance.

Treat prompts and documents as untrusted input

Natural-language instructions do not create a security boundary. A user can ask a model to ignore its instructions, and a retrieved document can contain text that attempts the same thing.

We therefore assume that anything placed in the context might influence the output. We separate instructions from reference material clearly, but we do not rely on formatting alone to prevent prompt injection.

Secrets never belong in prompts. Access controls belong in application code. If a feature can trigger an operation, we expose only narrowly defined operations and validate every argument independently. High-impact actions require explicit confirmation and, where appropriate, human approval.

The same principle applies beyond chat. We would not let generated text directly authorize a Web3 transaction or issue a hardware command. An entertaining answer and an irreversible action have very different failure costs.

We also escape generated content before rendering it. Model output is not automatically safe HTML, a trusted database query, or executable code.

Decide what data may leave the system

Before implementation, we create a simple data inventory: what enters the feature, which services receive it, what gets stored, who can inspect it, and when it is deleted.

We review provider terms for the specific service and account arrangement under consideration. We do not assume that consumer-product settings, API terms, retention policies, and enterprise agreements are interchangeable.

Where possible, we remove identifiers and unnecessary detail before sending a request. We avoid copying entire records when a few approved fields will do. Redaction helps, but it is not a guarantee that remaining context cannot identify someone.

Logs deserve the same attention as prompts. A debugging trace can quietly become a second store of customer information. We prefer operational metadata where possible and restrict access to any retained content.

If the data cannot be shared under the applicable obligations, we change the design or decline that integration. Convenience does not override confidentiality.

Evaluate behavior, not just impressive examples

We build a small, representative evaluation set before polishing the interface. It includes ordinary requests, ambiguous questions, missing information, misleading context, and attempts to cross permission boundaries.

For a grounded question-answering feature, we review several dimensions separately:

• Does the answer address the question?
• Are its claims supported by the supplied sources?
• Does it preserve important qualifications?
• Does it decline when evidence is insufficient?
• Does it expose information the user should not receive?

We record the prompt version, model configuration, retrieval inputs, and review criteria so that comparisons remain meaningful. Where outputs vary, we repeat tests rather than judging a single favorable response.

We also compare against a simpler alternative. Search results with useful excerpts may outperform a generated answer for some tasks. We should be willing to ship that simpler experience.

Evaluation informs a release decision; it does not certify that unexpected failures are impossible.

Design an interface that invites appropriate trust

A conversational interface can make software feel more capable and more certain than it is. We use design to clarify the feature’s role, not amplify that illusion.

We label drafts as drafts. We show supporting material near factual answers. We give people a straightforward way to edit, reject, or report an output, and we preserve the original input when that helps verification.

We avoid presenting a model’s self-reported confidence as a calibrated probability. A confident sentence is a style of output, not evidence of correctness.

For assisted workflows, the review step must be meaningful. If accepting a suggestion is effortless but checking it requires opening several other screens, we have designed for automatic acceptance. We bring the evidence and the decision together wherever possible.

Budget for latency, limits, and failure

A prototype often hides the operational costs of a language-model feature. Longer prompts and outputs consume more tokens, while retrieval, network calls, and retries add time. Model context limits also constrain how much material we can include in a single request.

We set limits on input size, output length, concurrency, and retries. We apply request timeouts and rate limits, and we track usage by feature so an experiment cannot quietly become an uncontrolled expense.

Caching can help for stable, repeatable tasks, but cache keys and access rules must prevent one user’s private result from being served to another.

We define a fallback before launch: show source documents, preserve a draft, queue work for later, or explain that the assistant is unavailable. We do not disguise a provider error as a successful answer.

Where portability matters, we isolate provider-specific code without pretending that different models behave identically.

Roll out gradually and assign ownership

We prefer an internal trial before a broad release. A limited pilot gives us room to inspect failures, improve documentation, and understand whether the workflow actually helps.

We assign owners for product behavior, security, operational support, and content quality. In a remote team, those responsibilities need to be written down rather than left to whoever happens to be online.

Our release notes include known limitations, and our runbook explains how to disable the feature or return to the previous prompt configuration. We review user reports alongside technical monitoring, because an API call can succeed while the answer fails the user.

Changes to prompts, models, or retrieval logic go through evaluation again. Small edits can alter behavior in surprisingly broad ways.

Our short pre-launch checklist

Before we ship, we check that we have:

• Defined one bounded task and its failure consequences.
• Enforced permissions outside the model.
• Reviewed provider terms and minimized shared data.
• Tested unsupported answers and adversarial inputs.
• Made sources, limitations, and review controls visible.
• Set usage limits, timeouts, and fallback behavior.
• Assigned an owner and documented a rollback path.

ChatGPT has raised expectations, and we see that as an opportunity to build better software, not a reason to skip the difficult parts. Our aim is to make language models useful within clear technical and human boundaries. If we can help turn an idea into a responsible, testable feature, start a project with us, or explore our related services pages to see how we bring AI, software engineering, cloud architecture, and design together.