Twilio and SendGrid: setting up reliable notification relays

By Marco Lopes · March 10, 2022 · Cloud & DevOps

A notification can look like a tiny feature: send an email after a purchase, text someone when a device needs attention, or tell a distributed team that a deployment has finished. The difficulty begins when a provider times out, a worker restarts, or a delivery callback arrives twice.

At Kordspace, we work remotely across collaborative software, AI, Web3, hardware, and design. Across those disciplines, we treat notifications as a small distributed system rather than a final API call bolted onto an application.

Twilio and SendGrid give us useful delivery infrastructure, but reliability still depends on how we connect them to our products. Here is the approach we use to design notification relays, with Google Cloud as our preferred foundation and equivalent options on AWS or Azure.

Start with a relay, not a direct send

We separate the business event from the delivery attempt. An application should record that something happened without waiting for an SMS network or receiving mail server to cooperate.

Our basic flow is:

Application → durable event → queue → delivery worker → Twilio or SendGrid

A second flow closes the loop:

Provider callback → authenticated webhook → durable event → notification state

This separation lets us retry temporary failures, control throughput, and investigate missing messages without replaying the original business action.

We also distinguish several meanings of “sent.” A queue accepting work is not the same as a provider accepting a request. Provider acceptance is not proof of delivery. Delivery is not proof that a person read or acted on a message. Our interfaces and dashboards should preserve those distinctions.

Define the notification contract

Before wiring up credentials, we define a small internal message format. We usually include:

• A stable notification ID and originating business event ID.
• The channel, recipient reference, and notification purpose.
• A template identifier and version, plus validated template data.
• Creation time, expiration time, and a correlation ID.
• A delivery state and separate records for individual attempts.

We prefer recipient references over copying personal information into every queue payload. The worker can resolve the necessary address or phone number using appropriately scoped access.

We keep provider identifiers separate from our notification ID. A Twilio Message SID belongs to a provider submission; our internal ID represents the intended notification. Multiple attempts must remain traceable without pretending they are the same network operation.

Expiration matters, too. A password reset or device alert can become useless, or confusing, after a delay. We check validity immediately before sending, not only when enqueueing.

Make the database-to-queue handoff durable

One of the easiest failures to miss happens before Twilio or SendGrid enters the picture. We commit a business transaction, then fail to publish its notification. Reversing the order creates another problem: a message may announce a transaction that never committed.

For applications with a suitable transactional database, we use an outbox. The business update and an outbox record are written in the same transaction. A dispatcher later publishes pending records and marks their progress.

The dispatcher can crash after publishing but before recording success, so duplicate queue messages remain possible. We make consumers idempotent rather than assuming the outbox guarantees exactly one execution.

With Cloud SQL, the outbox can live alongside application tables. With Firestore, we can use an atomic write or transaction where the data model permits it. In either case, we monitor undispatched records and give the dispatcher a safe recovery path.

Choose queue semantics deliberately

On Google Cloud, we commonly combine Cloud Run workers with either Pub/Sub or Cloud Tasks.

We reach for Pub/Sub when a business event should fan out to independent consumers. We reach for Cloud Tasks when we want explicitly scheduled HTTP work, configurable dispatch limits, and controlled retries toward a particular handler. The choice follows the delivery model, not a preference for adding more services.

We acknowledge work only after its result, or enough information to recover it, is durably recorded. For long-running workflows, we split the work into stages rather than holding one request open indefinitely.

We use dedicated service accounts and authenticated requests between services. Public provider webhooks need their own verification mechanism; an internal worker should not become public just because a callback endpoint must be reachable.

On AWS, SQS with Lambda or container workers can fill similar roles. On Azure, Service Bus with Functions or containers is another option. We still review each platform’s retry and acknowledgment semantics independently.

Configure the channels before testing reliability

For Twilio SMS, we provision an appropriate sender and, where useful, group senders through a Messaging Service. We normalize destination numbers to E.164, while remembering that valid formatting does not prove a number exists or can receive SMS.

We review destination countries, sender requirements, throughput constraints, and consent before enabling traffic. We also test message length and encoding: Unicode characters can change segmentation, affecting both cost and the recipient experience.

For SendGrid, we authenticate the sending domain and configure SPF and DKIM using the provider’s DNS instructions. We align our DMARC policy with the domain’s broader mail setup rather than introducing a policy that breaks other legitimate senders.

We keep transactional and marketing purposes distinct, respect suppression and unsubscribe rules, and provide required opt-out mechanisms. A reliable relay must not repeatedly force messages toward recipients who have opted out or addresses that have permanently failed.

Treat retries as a policy, not a loop

We classify failures before deciding whether to retry. Invalid addresses, malformed payloads, and authorization failures usually require correction. Rate limits, selected server errors, and network failures may justify another attempt.

For transient failures, we use exponential backoff with jitter, a maximum attempt count, and an overall expiration deadline. Where the provider supplies applicable retry guidance, we respect it. We also keep queue concurrency below what our sender configuration and account can support.

The difficult case is an ambiguous timeout. A provider may have accepted the request even though our worker never received the response. Retrying can create a second message; refusing to retry can leave the recipient without one.

We do not assume either send API offers a universal idempotency guarantee. Instead, we record attempts before submission, persist returned identifiers promptly, and define an ambiguity policy by notification type. Low-risk updates and sensitive one-time messages may need different handling.

An application-side deduplication key helps with repeated events, but it cannot eliminate every crash window around an external API call.

Verify callbacks and expect duplicates

Callbacks are inputs from outside our trust boundary. We authenticate them before using their contents to change state.

For Twilio, we validate the request signature with an official helper library and the appropriate auth token. Validation depends on the request details, including the URL Twilio called. Reverse proxies and URL rewriting can therefore cause legitimate requests to fail validation if we reconstruct the external URL incorrectly.

For SendGrid, we enable signed Event Webhook requests and verify the signature using the configured public key. We preserve the raw request body needed for verification rather than parsing and serializing it first.

After verification, we validate the payload, durably accept the events, and return promptly. We do not wait for unrelated application work before responding.

Both delivery paths must tolerate duplicates and out-of-order events. We deduplicate SendGrid events using their event identifiers where available, and design Twilio status processing around the Message SID and callback data rather than assuming every request is new.

Model delivery as evidence

We keep an append-only history of provider events alongside a materialized current state. That gives us an explanation when a support question arrives, while keeping routine application reads simple.

Our internal states might include pending, submitting, accepted, delivered, failed, and expired. We map provider statuses explicitly, preserving the original status and error details for diagnosis.

For email, delivery generally means the receiving mail server accepted the message, not that it reached the inbox or was read. Open tracking is also not a dependable acknowledgment; image loading and privacy features affect what it tells us.

For SMS, the availability and meaning of delivery receipts depend on the route and carrier. We avoid presenting a stronger guarantee than the evidence supports.

We also prevent late intermediate callbacks from blindly overwriting later outcomes. Where events conflict, we retain the evidence and apply channel-specific transition rules instead of sorting everything by arrival time.

Secure the relay and limit its blast radius

We store credentials in Secret Manager rather than source code, container images, or queue payloads. SendGrid keys receive only the permissions they need. We use restricted Twilio credentials where supported, and isolate environments so development cannot accidentally reach production recipients.

Provider secrets, webhook verification material, and internal service identities have different jobs. We document rotation for each and test how deployments pick up replacements.

We redact message bodies, phone numbers, email addresses, and reset links from routine logs. Correlation IDs and provider IDs usually give us enough operational visibility without creating a second database of sensitive content.

We apply retention limits to event histories and restrict access to replay tools. A manual resend is still an outbound communication, so it needs authorization, an audit trail, and the same consent and expiration checks as an automatic send.

Operate for recovery, not just success

We monitor queue age, outbox backlog, provider acceptance, delivery failures, callback verification failures, and exhausted retries. Queue age is especially useful: workers can be healthy while recipients wait behind a growing backlog.

We separate dashboards by channel, environment, and notification purpose. A marketing burst should not hide a failing password-reset path. Separate queues or dispatch limits can also protect urgent traffic from bulk work.

For Pub/Sub, we can configure a dead-letter topic. For Cloud Tasks, we maintain our own terminal-failure records and recovery process rather than assuming an equivalent dead-letter queue exists. In both cases, replay preserves the original notification identity and checks whether delivery is still appropriate.

We test duplicate callbacks, reordered events, timeouts, invalid signatures, and worker crashes after provider acceptance. Provider test facilities help with request handling, but they do not replace controlled end-to-end checks of real delivery paths.

A short launch checklist

Before we enable production traffic, we confirm that:

• Business events survive a failed queue publish.
• Duplicate work and ambiguous submissions have documented handling.
• Sender configuration, consent, and domain authentication are ready.
• Callbacks are verified, persisted, and processed idempotently.
• Retry budgets, expiration, and throughput limits are explicit.
• Alerts, terminal-failure records, and audited replay tools work.
• Logs and payloads expose only necessary recipient information.

Reliable notification relays come from clear boundaries and recoverable failures, not from assuming every API call succeeds. At Kordspace, we connect cloud infrastructure, software, hardware, and product design so those boundaries make sense for the people using the system. We welcome the chance to build yours: start a project, or explore our related services to see how we can work together.