Skip to content
stateless.co · Engineering notes from the request/response layer
statelessThe engineering desk

A publication about the machinery under everyday software: the contracts between services, the queries behind a page, and the failures that only show up in production.

—Archive

Designing Resilient Payment Orchestration Engines

14 November 2026

A variable recurring payment in UK Open Banking involves two separate API calls, and an orchestration engine has to treat them as different things. The VRP API profile v3.1.11 defines a consent step in which the payment service user (PSU) sets up a VRP consent. Only afterwards does the third party initiate individual payments against it. Version 4.0 of the specifications reflects the same split: POST /domestic-vrp-consents creates the consent resource, and POST /domestic-vrps initiates a payment. An engine that collapses those into a single "pay" action will struggle the first time a network call fails halfway through.

Dynamic gateway routing and payment method configuration

The orchestration layer sits between the merchant's checkout or billing system and the providers that move the money. Its job is to decide, per request, which payment service provider (PSP) or bank-facing route to use, and which payment methods to offer in the first place. That decision should live in configuration and rules rather than in application code, so that a route can be disabled, a method switched on for one market, or traffic shifted between providers without a deployment. Teams that hard-code provider choice into checkout services usually discover the cost during an incident, when the only way to reroute is a release.

Routing engines evaluate payload metadata dynamically to select appropriate payment rails across regional acquirers. The inputs are usually mundane: currency, amount band, customer country, whether the setup involves corporate billing or online casino deposit options, and the current health of downstream routes. For Open Banking, one of the first questions is whether the customer has a live VRP consent. If so, the engine can initiate a payment against it. If not, it must send the customer through the consent journey before anything can be collected.

The decision itself is as valuable as the outcome. One payment orchestration example, published by Philip Scheer, recommends storing a full decision trace for every authorization, for audit and for future model training. In practice that means recording which rules were evaluated, which inputs they saw, which candidate routes were considered and why the chosen one won. When a merchant asks why a particular payment went through a more expensive route, or a regulator asks why a customer was offered a given method, the trace answers the question without anyone reconstructing it from logs.

Payment-method configuration deserves the same discipline. Treat each method as a versioned record with explicit conditions for availability, and keep a history of changes. A method that quietly vanished from checkout last Tuesday is far easier to explain when the configuration change is dated and attributed.

Idempotency keys and duplicate-safe retries

The most dangerous moment in a payment flow is a timeout. The client sent a POST, nothing came back, and it is unclear whether the bank received the instruction. Retrying blindly risks a second payment; not retrying risks a customer who paid nothing or a merchant who never gets paid. Idempotency keys exist to take that dilemma away.

Open Banking 4.0.1 uses idempotency keys on the relevant POST endpoints. It also says that, wherever possible, the third party should use the GET endpoint to check payment status before submitting a duplicate. That ordering matters. A retry should begin with a read, not another write: ask the bank what it knows about the resource, and only resubmit if the answer shows nothing was created. The idempotency key is the safety net for the cases where that read is inconclusive or the retry races with the original.

Generating the key is where teams often go wrong. The key must be tied to the business intent, such as "collect this invoice from this mandate", and stored before the first attempt is sent. If the key is generated inside the retry loop, every attempt looks like a new instruction and the protection is gone. Persist the key and the intended request together, then let any worker that picks up the job reuse them.

A production payment engine guide published by Faru goes a step further and recommends combining idempotency keys with consumer deduplication and a database uniqueness constraint. The layers cover different failures. The key protects the bank-facing call, deduplication in consumers protects against a message being processed twice from a queue, and the uniqueness constraint is the final backstop if both of those are bypassed by a bug or a manual replay. No single layer is enough on its own.

State machines for consent, submission, and settlement

A payment is not a boolean. Between "customer agreed" and "money has arrived" there are several distinct states, and an orchestration engine that models them explicitly is far easier to operate than one that stores a single status string and hopes for the best. The lifecycle for VRP starts with the consent, moves to initiation of a payment under that consent, and ends with evidence of settlement.

Consent has its own states, independent of any payment. A consent can be awaiting the customer's authorisation, authorised, rejected or revoked, and each payment submitted under it inherits those constraints. Keeping consent and payment as separate state machines means a revoked consent can block new submissions without rewriting the history of payments that already completed.

Final settlement evidence is where precision pays off. Open Banking payment-status guidance, as set out in a write-up of the payment state machine from Asima, distinguishes ACSC as settlement completed on the debtor account from ACCC as the beneficiary account credited. These are not synonyms. If the engine treats ACSC as "the merchant has the money", it will mark orders as paid before the beneficiary side confirms. Decide in advance which status releases goods, which one triggers ledger entries, and which one merely moves the payment along.

Model transitions as a closed set: each state lists the events that may move it onward, and anything else is rejected and logged. Late or out-of-order notifications are then handled by rule rather than by accident, and a status that arrives after a later one has already been recorded cannot drag the payment backwards.

Webhook delivery, callbacks, and recovery

Polling every pending payment is wasteful, and relying only on callbacks is fragile. Open Banking 4.0.1 gives both options. An account servicing payment service provider (ASPSP) can send a signed real-time event notification to a registered callback URL, and the Event Notification API supports aggregated polling for anything missed. A resilient engine uses the first for speed and the second for certainty.

The callback endpoint should do very little before it responds. Verify the signature, persist the raw event, acknowledge it, and hand the rest to asynchronous workers. Doing business logic inline ties the bank's delivery attempt to the speed of your database and invites timeouts, which in turn invite redelivery.

The following controls cover the usual failure modes for event handling:

  • Verify the signature on every inbound event before parsing or trusting its contents.
  • Store the raw event first, then acknowledge, then process asynchronously.
  • Deduplicate on the event identifier so a redelivered notification changes nothing.
  • Run aggregated polling on a schedule to catch events that never arrived.
  • Reconcile against the GET status of any payment that has stayed pending beyond its expected window.

Reconciliation is the control teams skip until they need it. A scheduled job that compares pending payments against what the bank reports will turn a missed callback from a customer complaint into a routine correction. It also provides a clear answer to the question of what the system believes versus what the bank holds.

Auditability and operational controls

Resilience is partly about surviving failure and partly about being able to explain it afterwards. The decision trace described earlier should be append-only: written once when the routing decision is made, never edited, and linked to the payment by a stable identifier. If a rule is later corrected, the old trace still shows what the engine did at the time, which is what an auditor or a dispute handler actually needs.

The uniqueness constraint recommended in the production guidance belongs in the schema, not just in application checks. A constraint on the idempotency key, or on the combination of consent and business reference, means that two workers racing each other cannot both succeed. The loser receives an error and reads the winner's record. That is a far better failure than two payments leaving the building.

Finally, keep transport state separate from business state. Whether a request was sent, timed out, or received a response describes the network conversation. Whether the payment is pending, settled or failed describes the business outcome. Mixing the two produces the classic bug where a timeout is recorded as a failure and the order is cancelled, even though the bank went on to complete the payment. Store the attempts as their own records, derive business state from confirmed evidence such as ACSC and ACCC, and make every change of state traceable to the event that caused it.