An integration that has been broken for three weeks and an integration that is working look identical from the outside. Both systems are up, no one has filed a ticket, and the dashboard, if there is one, is green. The difference surfaces at month end, when finance finds four hundred orders that never reached the accounting system and no one can say when they stopped arriving. Integrations rarely fail loudly. They fail quietly, in a small number of predictable ways, and the failures are predictable enough to design against.
This is not architecture advice about queues versus webhooks. That choice matters far less than whether the connection has an owner, a written data contract, defined failure behavior, and something that checks whether the two systems still agree.
The root cause is that nobody owns it
Integrations are built during projects and orphaned when projects end. The consultant who wrote it left. The internal engineer who understood it moved teams. The vendor considers it your side of the boundary and you consider it theirs. So the integration has no name attached to it, no on-call rotation, and no place in anyone's roadmap, which means no one notices when the volume drops by half.
This is an organizational fact with technical consequences. Unowned code does not get its dependencies updated, its credentials rotated, or its assumptions revisited when the business changes. Assigning an owner is the cheapest reliability work available, and it is the step most often skipped because it is not a build task.
- A named owner for each integration, not a team alias with no rotation
- A documented business consequence of failure, so severity is decided before the incident, not during it
- A stated recovery expectation: how far behind the data may fall before someone is paged
- A known contact on the other side of every external connection, refreshed when people leave
- A place in a backlog where contract changes and dependency updates can actually be scheduled
Write down the data contract
Most integrations encode their contract implicitly: whatever the source system happened to send on the day it was built. Fields are read positionally or by name, types are inferred, and nothing checks that today's payload resembles yesterday's. Then someone on the other side adds a field, widens an enum, changes a date format, or starts sending null where a value was always present. Nothing throws. The record is written with a missing customer reference, and the damage is discovered downstream, weeks later, by an accountant.
The fix is to validate at the boundary and refuse anything that does not match. Define the schema explicitly, version it, and treat an unexpected shape as a failure rather than something to work around with a default value. Rejecting a record loudly is nearly always better than accepting a half-populated one, because a rejected record is visible and a corrupted one is not.
Expect the contract to change without notice. Vendors ship on their schedule, not yours, and the notification email goes to whoever signed the original agreement. Assume you will discover the change from your own validation errors, and design so that discovery is the first thing that happens rather than the last.
Retries make things worse without idempotency
Every durable messaging system you are likely to use delivers at least once. That is the correct default, and it means duplicates are not an edge case. They are guaranteed. A timeout does not tell you whether the other system processed the request; it only tells you that you did not hear back. Retry that request and you may have created a second invoice, a second shipment, or a second payment.
Idempotency is what makes retries safe. Give every message a stable key derived from the business event, not from the moment of transmission, and record which keys you have already processed. On a repeat, return the original result instead of doing the work again. This is a small amount of code and a small table, and it is the difference between a retry policy that heals the system and one that quietly doubles the general ledger.
Poison messages, backpressure, and the queue that eats itself
One malformed record can stop everything behind it. A message that fails deterministically will fail on every retry, and if the queue preserves order or the worker retries indefinitely, that single record blocks the flow while the backlog grows. Bound the retries. After a small number of attempts, move the message to a dead-letter queue, alert on it, and let the rest of the traffic through.
A dead-letter queue is only useful if someone can replay from it. That means the payload is stored intact with the failure reason and enough context to reprocess, and that a documented replay procedure exists and has been run at least once in a non-production environment. A dead-letter queue no one has ever drained is a landfill.
Backpressure is the other half. When the downstream system slows down or starts returning rate-limit responses, an integration that keeps pushing at full speed converts a slowdown into an outage, and often into blocked credentials. Respect the documented limits, back off exponentially with jitter so your workers do not synchronize into a thundering herd, cap concurrency, and open a circuit breaker when the error rate stays high. Slowing down is a feature.
The failure modes and the controls that address them
| Failure mode | How it presents | Control |
|---|---|---|
| No owner | The break is reported by a customer or found at month end, not by a system | A named owner, a documented failure consequence, and a recovery expectation agreed in advance |
| Contract drift | A field changes type, format, or disappears; records write successfully with missing data | Explicit versioned schemas validated at the boundary, rejecting anything that does not match |
| Duplicate delivery | Two invoices, two shipments, doubled totals after a timeout or a redeploy | Idempotency keys derived from the business event, plus a processed-key store with a retention window |
| Poison message | One record retries forever while the backlog behind it grows | Bounded retries, then a dead-letter queue with a replay procedure that has actually been tested |
| Rate limiting and retry storms | Rising 429s, growing latency, then blocked credentials | Exponential backoff with jitter, a concurrency cap, and a circuit breaker on sustained errors |
| Partial failure | One system committed the change and the other did not; totals disagree | An outbox in the source system and explicit compensating actions, never a distributed transaction across vendors |
| Silent divergence | No errors anywhere, both systems healthy, and the two datasets disagree | Scheduled reconciliation producing a variance report that a named person reviews |
| Alert fatigue | The alert channel is muted and a real outage sits unread | Alerts on business symptoms and thresholds rather than on every exception, each with an owner and an action |
Alerts people act on, and reconciliation as proof
There are two kinds of alert. One tells a specific person that a specific thing needs a specific action. The other fires on every handled exception, arrives in a channel with four hundred unread messages, and trains everyone to ignore it. The second kind is worse than no alerting, because it creates the belief that the system is monitored.
Alert on symptoms the business would recognize: orders that have not synced within the expected window, a queue depth that keeps climbing, a dead-letter count above zero, a success rate below its normal band. Every alert needs a documented action. If the action is to look at it and close it, delete the alert and keep the log.
None of that proves the two systems agree. Health checks confirm that a process is running, not that yesterday's two thousand transactions exist on both sides with the same values. Reconciliation is the only thing that proves it: on a schedule, compare counts and totals for a defined window, report the variances, and give a person the job of clearing them. Most teams discover real, long-standing gaps the first time they run it, which is the argument for running it.
- 1
Pick the window and the key: a business day, matched on the source system's record identifier.
- 2
Compare record counts on both sides first; a count mismatch is cheaper to detect than a field mismatch.
- 3
Compare the two or three values that carry financial or operational weight, not every field.
- 4
Publish variances to a place a person reviews, with the source record identifier included so it can be traced.
- 5
Track how long a variance stays open. A number that keeps growing is a design problem, not a data-entry problem.
A trades services business ran dispatch, field updates, and invoicing across fragile, undocumented integrations. Failures were invisible until they showed up as delayed invoices, which made the connections themselves a business constraint rather than a technical detail.
Make it supportable by someone who did not build it
An integration is supportable when someone who has never seen the code can diagnose it at two in the morning. That is a documentation problem, and it takes about a day per integration. It needs to exist somewhere other than the author's memory.
- What moves, in which direction, triggered by what business event, and at what volume
- Which system is authoritative for each field when the two disagree
- The data contract and its current version, with a note on how changes are communicated
- What happens on failure: retry counts, backoff, where dead-lettered messages land, and how to replay them
- Where the credentials live, who can rotate them, and when they expire
- The reconciliation job, its schedule, and who reads the output
- Two or three known failure scenarios with the exact first step to take
This is also the honest test of whether an integration is finished. If the only way to support it is to call the person who wrote it, it is not finished. It is a dependency on an individual, and it will break for good the week they change jobs.
Common questions
- Our integration works fine for weeks and then fails suddenly. Why?
- Almost always because something on the other side changed (a field, a format, a rate limit, an expired credential) and nothing validates the payload before it is processed. The code did not change, so the code is not where to look. Add schema validation at the boundary and you will usually see the cause on the next occurrence rather than at month end.
- Will an iPaaS platform make our integrations reliable?
- It gives you connectors, retries, and logging without writing them, which is genuine value. It does not give you an owner, a validated data contract, idempotent operations, or reconciliation, and those are what actually break. A managed platform with none of those controls fails the same way a script does, just with a nicer interface.
- How do we stop duplicate records when we retry?
- Derive a stable idempotency key from the business event. Use an order identifier, not a timestamp or a random value generated at send time. Record processed keys and return the original result when a key repeats. If the downstream system supports idempotency keys natively, use it; if it does not, keep the dedupe table on your side.
- How often should we reconcile?
- Match the cadence to the business consequence. Financial flows generally warrant a daily reconciliation with a monthly deeper comparison; a low-stakes reference data sync may only need a weekly count check. The important part is that the variance report has a named reader, because an unread reconciliation is the same as no reconciliation.
- How much does it cost to make an existing integration reliable?
- Retrofitting validation, idempotency, dead-lettering, alerting, and reconciliation onto one existing integration is usually a few weeks of work, and less if the flow is simple. The larger cost is inventorying what you have, since most organizations find integrations nobody remembered. That inventory is worth doing on its own, regardless of what you fix afterward.
- Should we rebuild a fragile integration or repair it?
- Repair it if the data contract is understood and the flow is roughly correct; adding the controls described here is cheaper than a rewrite and carries less risk. Rebuild when nobody can state what it does, when it writes directly to another system's database, or when the failure behavior cannot be reasoned about at all. Rewriting an integration you do not understand reproduces the bugs you have not found yet.