Go microservices for e-commerce middleware at scale

By Marco Lopes · Published 2022-01-10 · Topic: Web Development

E-commerce middleware lives between systems that rarely agree. A storefront speaks in carts and customers; a warehouse speaks in stock movements; a payment provider speaks in authorisations and settlements. Our job is to turn those differences into reliable workflows without making checkout depend on every integration being healthy.

At Kordspace, we work as a remote collaborative studio across software, AI, Web3, hardware, and design. For commerce infrastructure, we favour practical engineering over architectural fashion. Go is a strong option when we need concurrent integration workers, straightforward deployment, and services that remain understandable as a platform grows.

Here is how we approach Go microservices for e-commerce middleware, with Google Cloud as our preferred platform and room for AWS or Azure when the surrounding infrastructure calls for them.

Start with workflows, not service counts

Before choosing containers or message brokers, we map the business journey: accept an order, confirm payment, reserve inventory, request fulfilment, and communicate progress. Then we identify who owns each fact and what happens when a step fails.

A successful payment does not guarantee available stock. An accepted warehouse request does not mean a parcel has shipped. Middleware must preserve those distinctions instead of compressing everything into a single “success” flag.

We also question whether microservices are necessary. A modular Go application with clear internal boundaries may be the best starting point. We split services when independent deployment, scaling, ownership, or failure isolation provides a concrete benefit, not because each database table deserves an API.

Draw boundaries around business capabilities

Useful boundaries often include order orchestration, inventory integration, payment integration, catalogue synchronisation, and fulfilment adapters. Each service should own a coherent responsibility and publish an explicit contract.

We keep external-provider details inside adapters. A warehouse’s unusual status codes should not become vocabulary that every internal service must understand. Instead, we translate them into a small internal model while retaining the original payload where audit and troubleshooting requirements justify it.

Data ownership matters just as much. We avoid letting services update one another’s tables. Sharing a database instance can be a reasonable early compromise, but separate schemas and restricted database roles help preserve boundaries. Direct cross-service writes create coupling that becomes painful during migrations and incidents.

Use Go for clarity as well as concurrency

Go gives us useful building blocks: a capable HTTP standard library, explicit error handling, built-in testing, and lightweight goroutines. We can package a service as a compiled executable and keep its runtime container small.

However, goroutines are not a substitute for capacity planning. If every incoming request launches unbounded background work, we can exhaust database connections or overwhelm a provider long before CPU becomes our bottleneck.

We use bounded worker pools, pass context.Context through request paths, and set deadlines on outbound calls. Shared HTTP clients reuse connections; explicit transport settings make resource use predictable. We also run the race detector in appropriate tests and design shutdown paths that stop intake before waiting for in-flight work.

Separate checkout from slower integration work

Checkout needs a short, understandable synchronous path. We validate the request, perform the decisions that genuinely require an immediate answer, persist the accepted state, and move suitable downstream work onto a queue.

We do not make order acceptance wait for an analytics endpoint, an email service, and an enterprise resource planning import. Those dependencies can proceed asynchronously, with visible status and recovery paths.

That does not mean every operation belongs in the background. Payment authorisation or inventory reservation may need to happen before confirmation, depending on the business rules. We make that policy explicit. The API response should distinguish an accepted order from one that is paid, reserved, or ready for dispatch.

Treat messages as contracts

On Google Cloud, Pub/Sub is a natural option for distributing integration events. We design consumers for redelivery and avoid assuming global ordering. When ordering matters, we define the entity boundary and evaluate whether ordering keys or application-level version checks fit the workflow.

An event envelope usually includes an event ID, event type, schema version, creation time, entity ID, correlation ID, and payload. We document whether the payload represents a historical fact or a request to perform work.

We favour additive schema changes and tolerant readers. Removing a field or changing its meaning needs a migration plan. Producer and consumer deployments will not happen simultaneously, especially across a remote team operating independently deployable services.

Close the database-to-broker gap

A common failure appears between two innocent-looking operations: save the order, then publish an event. If the database commit succeeds but publishing fails, the order exists while downstream systems know nothing about it.

We address this with a transactional outbox. The order change and an outbox record are committed in the same database transaction. A separate publisher reads pending records, publishes messages, and marks its progress.

This still permits duplicates: the publisher might crash after publishing but before recording success. That is why the outbox and idempotent consumers belong together. We monitor outbox age and backlog, not just publisher errors, so a stalled integration is visible even when the storefront continues accepting orders.

Make retries safe before making them automatic

Retries are essential, but an uncertain payment response must not become a second charge. We define idempotency at the business-operation level rather than treating it as a queue feature.

For incoming API commands, we can store an idempotency key alongside a request fingerprint and the recorded result. A repeated key with different input should be rejected. For event consumers, a unique constraint on the processed event ID can prevent duplicate local changes when recorded within the same transaction.

External side effects require additional care. We use provider-supported idempotency keys where available and reconciliation where outcomes are uncertain. We retry transient failures with exponential backoff, jitter, and limits. Invalid input goes to a repair path rather than an endless retry loop.

Model workflows as recoverable state transitions

We avoid pretending that one database transaction can cover a payment gateway, an inventory system, and a shipping provider. Instead, we model explicit states and compensating actions.

For example, an order might move through payment authorised, inventory reserved, and fulfilment requested. If reservation fails, the appropriate compensation may be to void the authorisation. If money has already been captured, refund rules and timing become part of the workflow.

A compensation is a new operation, not a magical rollback. It can fail too. We retain enough history to explain what happened, retry safely, and support manual intervention. This is where a modest internal operations screen can be more valuable than another layer of automation.

Deploy around the workload on Google Cloud

For stateless HTTP APIs and bounded Pub/Sub push handlers, we consider Cloud Run. It reduces infrastructure management and suits services that can complete work within request limits. We do not rely on background goroutines continuing after a response; durable work belongs in a queue and an appropriate worker environment.

For long-running pull consumers or workloads needing more control, GKE can be a better fit. Cloud SQL for PostgreSQL is a practical transactional store, while Secret Manager keeps credentials out of images and repositories.

Autoscaling needs guardrails. We cap instance counts, tune concurrency, and budget database connections across the fleet. On AWS or Azure, equivalent components can support the architecture, but delivery semantics and operational limits need fresh validation rather than a mechanical product substitution.

Scale the constrained resource

“More instances” is not always the right response to increasing load. A provider rate limit, a hot database row, or a serial inventory update may be the actual constraint.

We measure queue depth, oldest-message age, processing duration, database contention, and downstream response times. Those signals tell us whether to increase consumers, reduce concurrency, batch requests, or change the data access pattern.

Backpressure protects the whole platform. We place limits around each provider and, in multi-tenant systems, prevent one merchant’s catalogue import from consuming all available workers. Bulk synchronisation and urgent order events often deserve separate queues or worker allocations so background volume does not delay time-sensitive operations.

Build observability into the contract

We use structured logs with service names, event IDs, order references, and correlation IDs. We propagate tracing context across HTTP and messaging boundaries where practical, using tools such as OpenTelemetry alongside Cloud Logging and Cloud Monitoring.

Technical health is only half the picture. We also track business-facing signals: orders awaiting reservation, payments awaiting reconciliation, and fulfilment requests without acknowledgement. A service can return healthy responses while the commerce workflow is stuck.

We keep sensitive data out of logs. Payment details, access tokens, and unnecessary customer information should not become debugging material. Alerts should point toward an actionable condition and a runbook, not merely announce that a container restarted.

Test failures, secure boundaries, and release gradually

Unit tests cover mappings and state transitions; integration tests exercise database transactions, deduplication, and outbox behaviour. Contract tests help us detect provider changes before deployment. We explicitly test duplicate webhooks, delayed events, timeouts, expired credentials, and messages arriving in an unexpected order.

For security, we verify webhook signatures against the raw request body, apply replay checks where supported, and scope service identities with least-privilege IAM. We minimise payment-data handling and agree retention rules for stored provider payloads.

Our delivery pipeline runs tests, builds a versioned image, and promotes the same artifact through environments. We use backward-compatible database changes and gradual rollouts. Replaying failed messages is an operational capability we test, with safeguards against repeating external side effects.

A short checklist before production

Before we launch, we confirm the essentials:

• Each service has an owner, a clear boundary, and a documented contract.
• Duplicate requests and messages cannot silently repeat business actions.
• Database changes and event publication have a recovery strategy.
• Timeouts, retries, concurrency, and connection pools have explicit limits.
• Failed workflows have reconciliation and manual repair paths.
• Dashboards reveal business backlogs as well as infrastructure health.
• Secrets, webhook verification, and log redaction are checked.
• Deployment, rollback, and message replay procedures are documented.

Build for the next constraint, not every possible future

Scalable middleware is less about the number of services and more about how clearly we handle ownership, uncertainty, and failure. Go helps us build focused components, but reliable commerce comes from explicit contracts, durable state, bounded concurrency, and operations that people can understand. We start with the smallest architecture that protects those properties, then let measured constraints guide the next split.

At Kordspace, we bring engineering and design together to make integrations usable as well as resilient. To explore an e-commerce middleware platform with us, start a project, or read our related services pages to see how we approach software delivery and collaboration.