API integration development: connecting systems without letting data drift apart

A Monday morning in a business with three systems: in the order program, a customer appears with the new delivery address, in accounting still with the old one, and in the web shop he exists twice because he once registered with a different e-mail address. Nobody made a mistake. Each system was correct on its own, and that is precisely the problem.

In this scenario we assume a mid-sized service provider that has bought specialist software over the years. Every program has an export function, a few have an API, and one delivers only CSV files by e-mail. Between the systems works one person who opens files, moves columns and copies values. That works as long as this person is there and the column order stays the same.

Anyone who commissions an interface therefore does not buy a connection but a behaviour: what happens when the same order arrives twice, when the target system is not responding, or when someone renames a field? The answers to these questions determine whether the integration holds. That two programs can exchange data is the easy half of the task.

On top of that, manual exchange hides knowledge. The person between the systems knows that certain customers must never be created automatically, that a column sometimes contains comments instead of numbers, and that the export takes longer at month end. This experience is written down in no documentation. Anyone who does not ask about it when automating builds an interface that handles exactly these special cases wrongly.

Most interface problems initially look like chance. An order is missing in the target system, another is there twice, a call fails on Monday but not on Tuesday. Behind these symptoms lie a few recurring causes, and they can be named before the first line of code is written.

The first is time. Credentials expire, tokens are renewed, certificates lose their validity. The second is volume: third-party systems limit requests per minute and typically respond with status code 429 when the limit is exceeded. The third is meaning: the field "status" has the same name in both systems but different values. The fourth is change, because the provider of the third-party API updates its software without knowing your maintenance plan.

On top of that, networks are unreliable. A call can reach the recipient, be processed, and the confirmation gets lost on the way. The sender sees a timeout and retries. Without precautions, this produces exactly the duplicate order that nobody can explain. This pattern is not an exception but a normal case in distributed systems.

A fifth cause is often underestimated: the silent partial transfer. A reconciliation processes 400 of 500 records, aborts at one faulty entry and reports nothing because the process formally ended. The next day 100 records are missing, and nobody knows which. Whether an integration isolates individual errors or aborts the whole run is therefore a deliberate design decision.

Before REST or webhooks are discussed, there is a question of responsibility: which system is the leading source for which data item? If customer addresses may be edited in both the order program and accounting, every synchronisation is a dispute that an algorithm cannot decide. With clear data ownership, a two-way reconciliation usually becomes a one-way flow with defined exceptions.

The second analysis concerns the trigger. Should the target system know immediately what happened, or is a reconciliation every few hours enough? Immediately means webhooks or events. Regularly means polling by modification time. Both have costs. Webhooks deliver quickly, but sometimes they do not arrive at all, arrive twice or in the wrong order. Polling is sluggish, but easier to repair, because you can ask at any time what has changed since yesterday.

Third, we look at the interface form of the third-party system. A well-documented REST API with a sandbox is the comfortable situation. More often there is a half-finished API that does not cover all fields, or only a file import via CSV or XML. In that case the file interface is not a stopgap but a format that can be made robust with checksums, fixed file names and receipt files.

Finally, the question of damage belongs in the analysis: what does it cost if a transfer is down for an hour, a day or a week? A newsletter contact arriving late is tolerable, a missing release for a delivery is not. The answer determines how much effort for monitoring and recovery is justified.

The tempting route is the point-to-point connection: system A calls system B directly. With two systems that is acceptable. With five systems, up to ten connections arise, each with its own formats, its own error handling and its own credentials. Every change to one system affects several places. We would therefore suggest a central integration layer from the third connected system onwards.

This layer is unspectacular: a small service that receives data, translates it into an internal model, places it in a queue and distributes it to the target systems. It collects credentials, logs and retry logic in one place. The price is an additional component that someone has to run. With two systems and stable interfaces that would be overkill, and we would say so.

An alternative is ready-made integration platforms and no-code tools. They are quick to set up and cover standard cases well, such as "new enquiry in the form, new entry in the CRM". Their limit lies where data is to be converted, reconciled or deliberately retried on errors, and in dependence on the vendor. For simple flows this is an honest option; for processes on which invoicing or delivery depend, we weigh things more carefully.

One idea is decisive: the integration is part of the software, not beside it. It has versions, tests, monitoring and an owner. Anyone who treats it as a by-product ends up with a script that an employee eventually no longer dares to change.

The central design step is the data model between the systems. For each entity, such as customer, order or item, we define which fields exist, which are mandatory, which values are allowed and which system is the master. This includes a mapping table: how does the value "open" in one system become the value "in progress" in the other? Such tables look trivial, but they settle the questions that later cost weeks.

A stable identifier is especially important. If two systems mean the same customer, they need a key known to both. Names and e-mail addresses are poor choices because they change. It is common for the integration to store the third-party system's identifier alongside its own. Without such a key, only a fuzzy match by similarity remains, and that produces mismatches that are noticed only when the invoice goes out.

Further details are inconspicuous and treacherous: time zones and timestamps, decimal separators, rounding of amounts, character sets for umlauts, empty values versus fields that were not sent. A field that is not transferred can mean "unchanged" or "deleted". This distinction has to be defined explicitly, otherwise an update overwrites complete records with gaps.

The model is described as a schema, for example with JSON Schema, and every incoming message is validated against it. Invalid messages do not end up silently in the target but in a list that a person can see. This turns a data quality issue into a visible process.

Let’s start with access. Third-party APIs usually require an API key or OAuth 2.0 with short-lived access tokens and a refresh token. Keys are never kept in the source code but in a protected store on the server, and they receive only the permissions the integration needs. When a token expires, the service must renew it itself and log the event. A calendar reminder for a certificate’s expiry is not a luxury.

For webhooks the rule is: acknowledge quickly, process later. The receiver checks the message’s signature, puts it in the queue and responds with success right away. The actual processing runs separately. That way a slow target system does not block the sender, and a message is not lost just because processing failed. Because senders repeat their messages on failure, the receiver must be able to cope with duplicates.

This is where idempotency comes in. An operation is idempotent if executing it several times has the same result as executing it once. In practice, the sender generates a unique identifier for each transfer, and the receiver remembers which ones it has already processed. If the same identifier arrives again, it is acknowledged but not executed again. Some APIs offer an idempotency key in the request header for this; where they don’t, you build the check yourself.

For errors, the integration distinguishes two classes. Transient errors such as timeouts, server errors and status code 429 are retried, with increasing intervals and a bit of randomness so that many retries do not arrive at the same time. Permanent errors such as invalid data or missing permission are not retried but parked and reported. Retrying everything blindly burdens the third-party system and hides the actual cause.

Finally comes the receipt: only when the target system confirms success and returns an identifier of its own is the transfer considered complete. This identifier is stored and permanently links the two records.

In integration projects, special cases are the rule. The first is catch-up processing: after an outage of one day, the data left behind is to be transferred, and the third-party API allows only a limited number of requests per minute. A rate limiter in your own service helps here: it spreads the calls evenly, evaluates the API’s response headers and respects the specified wait time on a 429. Catch-up processing must not displace normal operation.

The second special case is ordering. Webhooks can arrive out of order: the change “order canceled” comes before “order created”. The service must either detect the order using a version counter or timestamp and discard older messages, or query the current state directly from the sender when a message arrives. The latter is more robust but more expensive in terms of API calls.

Third, versioning of the third-party API. Reputable providers maintain versions and announce the end of old ones. The integration therefore fixes the version it uses explicitly instead of relying on a default version, and reads deprecation notices from response headers and change announcements. Tolerance when reading matters as well: new, unknown fields in a response must not cause a crash. If required fields are missing, on the other hand, the integration should fail loudly instead of quietly writing wrong data.

Fourth, connecting legacy systems that have neither an API nor webhooks. A lab system or an older ERP may deliver a file to a directory every night. This too can be solved cleanly: files are processed only once a completion marker is present, each line gets an identifier, and a receipt file reports back what was accepted. Such connections should not be seen as inferior but as a slow yet easily verifiable interface.

The benefit of an integration shows where people have so far worked as a data bridge. Less retyping means fewer typos, faster handover and time freed up. In this study we do not cite figures, because they depend on your starting point: how many records move each day, how often do errors occur, how long does correction take? These quantities should be measured before the project, otherwise there is nothing to compare against later.

Success can be measured with a few metrics: share of messages transferred successfully, number and age of parked cases, time from event to arrival in the target system, reconciliation differences in a regular sample, and hours of rework per month. A dashboard with these values is less impressive than an architecture diagram but more useful in operation.

The limit of the commitment is just as important. An integration can only be as reliable as the weakest connected system. If the third-party API is down for a day, the integration will not make up for that, but will catch up on the data once the system is back. Quality defects in the source data do not disappear either. An interface makes errors visible and transfers bad data quickly if you do not fix it at the source.

A realistic claim is therefore that transfers are traceable, errors are reported rather than swallowed, and an outage does not lead to data loss. That is less than “always works”, but considerably more than many grown-over scripts deliver.

Testing starts against the third-party provider’s sandbox, if there is one. It lets you trigger error cases safely: invalid data, expired tokens, limits reached. Where no sandbox exists, a custom replica of the third-party API as a simple test service that returns recorded responses helps. It does not replace real-world testing, but it makes retry logic and idempotency verifiable in an automated way. Production data does not belong in tests.

Observability in operation is just as important as the code. Each transfer gets a correlation identifier that appears in all log entries, so a record’s path can be traced. Alerts should report few but meaningful states: credentials are about to expire, the queue is growing, cases have been parked longer than a defined time. An alert that fires daily with no consequence gets ignored.

The documentation should be written so that a person with no prior knowledge can run the integration. This includes the data flows, the mapping model, the procedure in case of an outage, the management of credentials, the contact point at the third-party provider and a list of known quirks. The latter are often the most valuable part, because they record what is written nowhere in the official documentation.

Maintenance is not an appendix but part of the contract with yourself. Third-party APIs change, certificates expire, libraries become outdated. A fixed rhythm for dependency updates and a review of the third-party providers’ change announcements cost little compared with an emergency repair at month-end. Anyone who commissions only the development and leaves operation unresolved will later get the script nobody dares to touch.

Anyone having an API interface developed should not ask whether the data arrives, but what happens when it doesn’t. This perspective separates robust integrations from scripts that work in the normal case. Credentials, limits, retries, versioning and documentation are not extras here but the core of the work.

A sensible next step is a small one. For two of your systems, note which one leads for customers, orders and articles respectively, and where people today carry data from one to the other. This list alone usually shows which connection causes the most rework and which one is worth connecting first. Often it is not the biggest one but the one with the most errors.

We are happy to look at your system landscape together with you and tell you clearly where custom development makes sense and where a ready-made tool is enough. For the first flow we name a range after we have seen the third-party interfaces, not before. Commitments without that look would be guesses.

We also plan the timing of the switchover with you. A parallel run, in which the old and new transfer run side by side for a few weeks and are reconciled regularly, takes some effort but builds trust. Only when the differences are explained is the manual bridge switched off, and the person who has carried it so far stays involved as the expert on the special cases.