In many large enterprises, an API layer is being built on top of the existing application landscape via an ESB and an API gateway. The implementation of the APIs relies on the existing backend systems (ERP systems, cloud-based solutions, custom databases and other legacy systems) integrated via the ESB.
This contrasts with the green-field architectures that can be adopted by a start-up, in which there are no legacy systems or heavyweight ERPs to deal with. As a result, these architectures can be built from the ground up using microservices and APIs.
This two-part article discusses two patterns by which we can decouple a REST API from the backend systems that underlie its implementation.
The first pattern is the subject of this blog post (part 1), the second pattern builds on the first and will be described in part 2.
Typical API enablement architecture
A common realization of an API layer on top of enterprise backend systems is shown in the figure below. |
| Figure 1: Typical high-level architecture for API enablement |
- The API Gateway deals with API policies (authentication, usage monitoring, etc.)
- The ESB provides the API implementation by orchestrating interactions with one or more backend, and mapping between backend data formats and the API data format (i.e the resource representations since we are talking about REST API)
In sophisticated API architectures, the ESB implements up to three tiers or layers of APIs:
- "System" APIs, exposing raw backend data resource representations (e.g., JSON or XML representations of DB record sets)
- "Process" APIs, exposing "business domain" resources using some kind of backend-agnostic, "canonical" representation
- "Experience" APIs, exposing customized views of the Process APIs for specific types of clients; these API's are typically just a "projection" of the Process APIs
It is a fact that the gap between legacy systems and our desired resource representations can be quite wide, especially if we want our API to follow the RESTful HATEOAS principle and expose resource representations following JSON API or HAL, for example.
But ... we still have coupling!
Despite its many advantages, an architecture for REST APIs as described above can still possess a fairly high degree of coupling.There are in my view two main sources of residual coupling between API clients and our (theoretically hidden) backends:
- Time coupling associated with simple synchronous request-response
- Data coupling in resource URIs and resource representations
Quite clearly, the presence of a synchronous invocation scope that spans all the way into the backend systems couples API clients with the backends that are part of the API implementation.
This can be problematic for a number or reasons:
- The backend systems needed by the API implementation need to be highly available in order for the API to meet its agreed availability SLA.
- The performance of a synchronous chain of invocations can be compromised by a single "weak link". In many cases, backends can end up being the weak link. A slow response from a single backend can cause every API based on it to beach its response-time SLA.
- Sometimes synchronous interactions with systems of record are not encouraged, especially for create/update operations. For example, it is common for SAP teams to limit or forbid usage of synchronous BAPIs to post update transactions, demanding instead the use of IDocs (which are normally processed asynchronously) for this purpose.
- Some legacy backends may not even support synchronous request-response interactions (using staging DB tables or even files to exchange data).
Asynchronous update APIs
Here I describe an asynchronous approach when using APIs to update backend resources (above all for POST and PUT operations, but possibly also for DELETE operations).This is more complex than the simple synchronous approach but gains us the precious advantage of decoupling between API clients and backend systems.
For query (GET) operations it is usually sufficient to appropriately implement paging to cope with queries that are long-running queries and may yield a large resultset. However if the backend endpoint does not support paging or does not support a synchronous interaction, then the asynchronous approach I am explaining below may apply as well.
Behavior of an async update API
The essence of the proposed approach is that the API implementation submits the request message to the backend(s) via asynchronous, one-way protocols and immediately returns to the client an URI representing a "handle" or "promise" that allows to retrieve the API invocation result later (in a subsequent request).This "promise" pattern loosely takes after the Promise concept in Javascript/Node.js, where a Promise object encapsulates the result of an asynchronous operation that will be completed in the future.
Sticking to the REST update scenario, I can illustrate the proposed mode of operation with the following example. Against the following request:
POST /api/billing/v1/vendorinvoices HTTP/1.1
Host: mycompany.com
Content-Type: application/json
{Data}
The response may look like the following (omitting some "customary" response headers such as Date etc.):
HTTP/1.1 202 Accepted
Location: https://mycompany.com/api/billing/v1/vendorinvoices/f894eb7a-f2fc-4803-9a7d-644a0261010f
Where f894eb7a-f2fc-4803-9a7d-644a0261010f is a UUID generated by the API infrastructure and stored in a high-speed correlation store. After some time (possibly suggested to the client in the API response via a Retry-After header), the client should check for the results like this:
GET /api/billing/v1/vendorinvoices/f894eb7a-f2fc-4803-9a7d-644a0261010f HTTP/1.1
Host: mycompany.com
On receiving this GET request, the ESB would look up the correlation store and check whether a response has been posted there in the meantime (after being asynchronously mapped from backend data). There are three possible outcomes:
1. Result still pending
This should generate the same response as to the original invocation (with HTTP status 202).
2. Success
Response should be either:
HTTP/1.1 201 Accepted
Location: https://mycompany.com/api/billing/v1/vendorinvoices/5105600101
{Optional Data}
Or:
HTTP/1.1 303 See Other
Location: https://mycompany.com/api/billing/v1/vendorinvoices/5105600101
{Optional Data}
The important point here is that the "handle" or "promise" URI gets exchanged with the actual URI of the created resource, which contains the document ID generated by the target backend (i.e. 5105600101)A subsequent GET request to the resource URI would be not go through the correlation store anymore as standard ESB logic would recognize the URI as an actual resource URI.
3. Error
The correlation store needs to record any error response produced as a result of the execution of the API against a backend system. This error information would be returned in the response body with an appropriate HTTP error status code in the 4xx or 5xx range.
In case no result (positive or negative) gets recorded against the "handle" within a given timeout period, an error is returned to the API client.
Implementation
The figure below show how this "promise" pattern can be implemented (the "API implementation block" makes use of the ESB and the API Gateway): |
| Figure 2: Design for asynchronous API (note: dashed arrows express asynchronous flows) |
The original client request (1.1) generates a UUID.
The UUID is mapped as part of the asynchronous update message to the backend (1.2), for correlation purposes.
A correlation entry in created in the correlation store for the "handle" URI, incorporating the UUID (1.3).
The 201 Accepted response is returned to the client with the "handle" URI in the Location header.
When the new resource is created within the backend system, an asynchronous notification message is sent out (2.1) from which the actual resource URI can be derived. KEY POINT: This message MUST incorporate the UUID so it can be correlated with an existing correlation store entry. Any posting errors resulting from the processing of the request within the backend must also be notified via the same channel.
The actual resource URI (in case of success) or alternatively the error information is saved in the correlation store (2.2).
When the client makes a GET request with the "handle" URI (3.1), this is used to look up the correlation store (3.2), and the actual URI or the error information is returned to the client (3.3) via the customary Location header.
If the 3.1 request comes after a certain configured timeout after the creation timestamp of the correlation entry and the correlation entry still has a "null" result (i.e., no actual resource URL and no error info), then an error can be returned in step 3.3. Retention periods in the correlation store are discussed below.
In case of success, the client now holds the actual resource URI and can then query it (steps 4.1 to 4.4).
"Hybrid" implementation with synchronous backend-facing logic
At first sight, the complexity of this design seems to reside in the use of the correlation store, but this is not really the case: a sophisticated ESB implementation will normally anyway leverage a reliable, high-speed, high-available data store for caching, idempotency, and other purposes that require storing state.The most critical point is actually getting out a notification from the backend system (point 2.1), containing the correlation UUID that was injected via 1.2. There may be technological constraints within the backend that make this difficult.
In cases where the backend system does allow a synchronous request-response interaction, the best option is to take advantage of the best practice that demands that the API implementation be layered into a "system" API layer and a "business" API layer, and avoid propagating the UUID through the backend altogether.
This is shown in the figure below, where the System ("backend-facing") API interacts with the backend synchronously but communicates with the upper Business API layer asynchronously.
 |
| Figure 3: Decoupling within API implementation |
The System API implementation is triggered by message A and is completely decoupled from the interaction between the API client and the Business API layer. The backend-specific integration logic in the System API can update the backend synchronously (even if it has to wait relatively long for the backend response), and can automatically retry the invocation until successful in case the backend is temporarily unavailable. All this does not affect the API client in any way: the client holds the "handle" URI, i.e. the "promise" to retrieve the eventual outcome of the operation.
Once the System API has a "final" (positive or negative) response from the backend (C), it forwards this response asynchronously (again, typically, via a JMS queue), including the UUID that was in the System API synchronous context. This is message D, which triggers the Business API to update the correlation store with the result.
Retention period in the Correlation Store
Under this design, correlation store entries are not meant to be kept long-term, but should be automatically evicted from the store after a configurable retention period.The retention period must be guaranteed to be longer than both the following time intervals:
- The longest possible interval between the asynchronous backend request and the asynchronous backend response (i.e., between events 1.2 and 2.1 in Figure 2 above).
- The longest possible interval from the original client request to the latest possible request to exchange the "handle" URI with the effective resource URI (i.e., between events 1.1 and 3.1 in Figure 2 above). This second interval is the most binding one as it normally needs to be longer than the first.
If the API receives GET request (3.1 in Figure 2) with a "handle" URI that no longer exists in the correlation store, the API client receives an error. In any case, after retrieving the actual resource URI, a clients is supposed to keep hold of it and not query for the "handle" URI anymore.
Conclusion
The biggest disadvantage of such a design (besides of course the added complexity) is that API clients need multiple API calls to get hold of the actual resource.If the client is too "eager" to obtain the final result of the operation, it may poll the API too often while the API result is not yet available in the correlation store.
A REST API still relies on the HTTP(S) protocol which unlike a WebSocket protocol does not allow the server to notify the client of an asynchronous event (in this case: the API result is ready).
Nevertheless, it is so important to decouple API clients from the internal IT landscape of the enterprise that this pattern should be adopted more widely.
Part 2 of the article, with the aim of addressing the issue of data representation dependencies, will build on this pattern showing how what we called the "handle" or "promise" URI may well become the effective, permanent resource URI.
