GPT-3 in production: lessons from early API integrations

By Marco Lopes · November 3, 2021 · AI & Machine Learning

GPT-3 makes a compelling first impression. We provide a few examples, write an instruction, and receive text that looks surprisingly close to something a person might produce. Connecting that experience to a product, however, exposes a different set of problems: unpredictable responses, sensitive inputs, variable latency, and an invoice attached to every experiment.

At Kordspace, we approach these integrations as a remote collaborative studio working across software, AI, Web3, hardware, and design. Our concern is not simply whether a model can complete a task. We want to understand whether the surrounding system can make that capability useful, understandable, and maintainable.

As of November 2021, GPT-3 access and deployment remain subject to OpenAI’s API access process and applicable requirements. Within that context, these are the engineering principles we would bring to an early production integration.

Start with a bounded job, not an intelligent assistant

“Add AI” is not a useful specification. We need a task with an identifiable input, an expected output, and a clear definition of failure.

Drafting a product description from approved attributes is bounded. Answering any question about an entire business is not. Suggesting categories for an incoming message is easier to evaluate than letting a model decide what should happen next across several systems.

We start by writing down what the feature must not do. A drafting tool must not invent specifications. A support helper must not promise refunds. A summarizer must not silently turn uncertain statements into established facts.

This boundary determines the interface, evaluation process, and fallback. It also gives us a sensible first release: assistance within an existing workflow rather than autonomous control over it.

Treat the prompt as versioned application logic

A prompt is not merely copy. It encodes instructions, examples, assumptions, and formatting conventions that affect application behavior.

We keep prompt templates in version control alongside the code that assembles and interprets them. Each production request should be traceable to a template version, model choice, and generation settings. Otherwise, a seemingly harmless wording change can create a regression that is difficult to reproduce.

A useful template separates the instruction, reference material, user input, and expected output. Delimiters help make those boundaries legible, although they do not enforce security isolation.

We also choose examples deliberately. If every example is polished and complete, the model receives little guidance about missing information. We include cases where the desired response is an empty field, a request for clarification, or an explicit statement that the supplied material is insufficient.

Choose settings through evaluation, not intuition

The API offers models with different capability, speed, and pricing characteristics, including Ada, Babbage, Curie, and Davinci. We should not assume that the most capable option is necessary for every step.

For a constrained classification task, we can evaluate a smaller model before selecting a larger one. For nuanced drafting, a more capable model may justify its cost. We make that decision against representative examples rather than a handful of impressive demonstrations.

Generation settings belong in the same evaluation. Lower temperature generally suits consistency-oriented tasks; higher temperature may be useful when variation is part of the product. Neither setting establishes factual correctness, and even a low-temperature configuration should not be treated as a guarantee of identical results.

We set completion limits and stop sequences carefully, then test whether they truncate legitimate answers or allow unwanted extra text.

Keep the integration behind our own backend

We never put the provider API key in browser code, a mobile application bundle, or firmware distributed to customers. Requests pass through a backend that authenticates the caller and applies application-specific limits.

With Google Cloud as our preferred platform, Cloud Run is a practical place for that integration layer, with credentials held in Secret Manager. We can use Cloud Tasks or Pub/Sub when the work should happen asynchronously, depending on the delivery and processing requirements.

The architecture matters more than the brand names. We can apply the same separation on AWS or Azure: clients call our service, our service controls model access, and durable application state lives outside the model interaction.

We keep the provider adapter small. Prompt assembly, authorization, validation, and business rules should not become inseparable from a single vendor’s request format.

Design for latency and uncertain outcomes

A model request is a network operation with variable execution time. We need explicit timeouts, bounded retries, and a user interface that does not suggest an immediate result when none is guaranteed.

We retry transient failures and rate-limit responses with backoff and jitter, while avoiding retries for invalid requests. A queue can absorb bursts, but it also needs a maximum waiting time and a way to abandon stale work.

A timeout deserves special care: our service may stop waiting even though the upstream request has already been processed. Retrying can therefore create extra work and charges. We track application-level request identifiers and prevent duplicate results from triggering repeated downstream actions, without assuming that this guarantees a single upstream generation.

For longer drafting tasks, we prefer a visible job state: queued, processing, ready, or failed. A recoverable failure is better than an indefinite spinner.

Validate outputs before they touch business logic

GPT-3 produces text. Asking for JSON does not turn that text into a guaranteed schema-compliant response.

We parse structured outputs defensively and validate required fields, types, length limits, and allowed values. If a field must contain one of our internal category identifiers, we check it against that set. If the model returns a URL, we do not automatically trust or fetch it.

A repair attempt can sometimes recover malformed output, but repair requests need a strict limit. After that, we fall back to a safe result or human review rather than looping until something parses.

Validation must also cover meaning. A syntactically valid description can still invent a feature. We compare claims with authoritative input wherever practical and avoid treating valid formatting as evidence of truth.

Most importantly, generated text must never become executable code, a database query, or an authorized action merely because the model produced it.

Supply facts instead of expecting the model to know

We should not treat GPT-3 as a current, authoritative database. It can generate plausible statements that are unsupported, outdated, or simply wrong.

For knowledge-dependent tasks, we retrieve relevant material from sources we control and include an appropriately sized selection in the prompt. That might mean a database lookup, a keyword search over documentation, or a curated set of product records. We do not need a complicated retrieval system to begin.

The prompt should make the intended boundary explicit: answer from the supplied material and acknowledge missing information. We still test whether the model respects that instruction.

Where sources matter, we attach source identifiers in our application and verify any returned references. We should not display a model-generated citation as though it proves that a document exists or supports a claim.

Minimize data and separate trust boundaries

Before sending any input to an external API, we identify what it contains and whether it is necessary. Credentials, payment details, and irrelevant personal information have no place in a prompt.

We review the provider’s current terms and data-handling arrangements rather than assuming our own cloud settings also govern the external service. For sensitive workloads, that review may change the feature’s scope or rule out the integration entirely.

Our own logging needs equal attention. Full prompts and completions are convenient for debugging, but they can create a second, poorly controlled store of confidential information. We prefer minimal metadata, restricted access, defined retention, and carefully redacted diagnostic samples.

Untrusted text introduces another boundary. A document or user message can contain instructions that conflict with our template. Delimiters and warnings may help, but authorization must remain in ordinary application code, not in the model’s willingness to follow our prompt.

Build an evaluation set before polishing the demo

A production decision needs more than a successful live example. We assemble a small, representative evaluation set from material we are permitted to use, with synthetic cases added for sensitive or unusual scenarios.

That set should cover normal inputs, incomplete inputs, ambiguous requests, unusually long text, conflicting instructions, and content outside the feature’s intended scope. We define what counts as acceptable before comparing configurations.

Some checks are mechanical: valid structure, permitted categories, or absence of prohibited fields. Others require human judgment, including faithfulness, usefulness, and tone. We keep those judgments separate so that a fluent answer does not hide a factual failure.

Every meaningful prompt or model change runs against the same baseline. During rollout, we inspect failures and expand the evaluation set rather than relying only on positive feedback from enthusiastic early users.

Make cost and observability part of the feature

Usage depends on tokens, not simply the number of requests. Prompt examples, retrieved context, and generated output all contribute to the budget. Longer prompts also leave less room within the model’s context limit.

We set input and output bounds, enforce quotas per account, and monitor usage by feature. We record latency, errors, retries, token counts, validation failures, and whether users accept or substantially edit suggestions. These signals help distinguish an expensive feature from an expensive feature that actually saves effort.

Caching can help when identical inputs genuinely deserve identical outputs. Cache keys need to include the relevant model settings and prompt version, and cached content must respect tenant boundaries and data-retention rules.

We also provide an operational off switch. If quality degrades or usage becomes abnormal, we should be able to disable generation without disabling the rest of the product.

Give people control over consequential results

The interface should make the model’s role clear. We label a draft as a draft, preserve the original input, and let people edit or reject suggestions without fighting the workflow.

For consequential actions, we require confirmation after showing the actual proposed result. Approving a general request is not the same as approving the particular message, record update, or transaction the system later generates.

This distinction matters especially when connecting AI to hardware or Web3 systems, where actions may affect physical devices or be difficult to reverse. We keep permissions, transaction construction, and safety constraints outside the language model.

Human review is not a substitute for good engineering. It is an additional control that works best when reviewers have enough context and time to notice mistakes.

Our short pre-launch checklist

• Define the bounded task, prohibited behavior, and fallback.
• Version prompts, settings, and output schemas.
• Keep credentials server-side and minimize submitted data.
• Add timeouts, bounded retries, quotas, and an off switch.
• Validate structure, references, and consequential claims.
• Evaluate realistic failures as well as successful examples.
• Review applicable API requirements and confirm human approval paths.

GPT-3 gives us a powerful text-generation component, not a replacement for product design or dependable software. We get further by narrowing the task, measuring the result, and keeping control in the surrounding system. If we can help turn an AI idea into a practical integration, start a project with Kordspace, or explore our related services pages to see how we bring software, AI, infrastructure, and design together.