What is the Structured Integration approach?

I have called my blog Structured Integration because I believe that even in today's digital economy where agility is paramount, we should still approach the integration of applications in a structured, methodical way,  rather than in a casual, careless way.

Technology and methodology trends come and go, but in our particular domain there  is no shortage of experience: people have been interfacing separate IT systems and applications for many years, via file transfers, EDI translators, message oriented middleware, message buses, ESBs, ETL tools, Managed File Transfer tools, and more recently via API gateways and iPaaS solutions.

The EAI movement of the late 1990's and early 2000's originated from the need to address the anarchy of point-to-point ("spaghetti") interfaces.
Subsequently, the SOA paradigm brought much needed governance to Systems Integration, but on the flip side WS-* standards often proved complex to deploy, and centralized service governance proved too heavy and slow.

The emergence of REST APIs  has opened up many new channels of integration, mainly from mashups and  mobile applications, and the REST paradigm is currently the most popular to expose device-generated (IoT) data.  In the years 2011-2013 there has been a dramatic move away from SOAP services towards REST APIs and this trend is continuing.

Let us keep some governance, please!

However, the agile, "startup" mindset that allows us to seize the opportunities of the digital economy should not bring us back to the days of no governance and spaghetti interfaces.  The risks, especially for established enterprises that have much at stake, would be too big. 

First of all, on the backend side not every application has migrated to the Cloud (by any means) or exposes REST APIs, so we still have the challenges of traditional EAI.  Conclusion: require a solid ESB platform with capable adapters/connectors.

Secondly, even at the API layer, we still need a structured top-down approach to API design that is reasonably consistent both in itself and with regards to the backend service layer.  Using a pure bottom-up ("build first") approach, we likely end up with APIs that are inflexible and inconsistent, and with a bigger gap between the APIs and the backend services, requiring more complex mediation to bridge the two layers.

Put differently: are SOA Patterns still relevant in the API world?  Absolutely, just the technical details are changed. 

And, for that matter, even all the old dear EAI Patterns are still relevant for whoever is tasked with implementing backend services.  We should prevent the hype of new technologies from making us forget and throw away  established concepts, techniques, and methods that are still valid, and  just need some adaptation.

The role of BPM

Considering the bigger picture, the one that comprises the whole integration landscape, we need to recognize that integrations at every level are there to support some kind of Business Process.   An approach the leverages Business Process Managment techniques allows us to:

• clarify integration requirements (at which point of a business process an integration should be carried out)
• recognize those integration requirements that are suitable to implementation via orchestration by a BPMS
• conceptually model the interaction among business events and service/API invocations even when it is not suitable to orchestration by a BPMS.

That is why Processes sit on top of the "integration lozenge" that summarizes my approach:

 

Services, APIs, and Events 

Just below the Processes element, we find the two elements  Services and Events.

A service is normally either a SOAP or a REST service, but in general it can be anything that implements a well-defined request-response interface, even if based on proprietary technology.
The description of services has evolved from "ancient" beginnings (think CORBA IDE, for those who are old enough...), to the WSDL 1.1 and 2.0 standards, though the adoption failure of WADL, and finally to the newer notations for the descriptions of REST services (such as Swagger, RAML, and API Blueprint).

On the other hand, events (which were all the rage at the end of the 90's) have taken a back seat to services (of which the modern REST APIs are just the latest incarnation) for the last decade.

Still, the event driven dimension is an essential one.  Asynchronous events having business relevance and daily life relevance are all around us.    Even if it possible to consider an event as an invocation of a one-way service (with no response message),  this point of view is often an awkward one.  
Besides, polling intelligent devices and appliances via request-response calls is hugely inefficient:  we need an efficient push model (like pub-sub MQTT messaging) rather than a pull model here.   Also in the IoT arena, the combination of lightweight asynchronous messaging with lightweight synchronous services interfaces seems to be a necessary one.

Thus, when doing integration architecture and design, it is valuable to treat events a first class entities, which have a well-defined data content and semantic significance.  Events are normally the trigger to start automated processes and they can be of course sent/published by service implementations.

In conclusion, integration work definitely calls for the service oriented and the event driven perspectives to be combined.

Data

Data is modeled in many IT contexts, and modeling in the context of integration is crucial to understand clearly:

• what information is held in instances of business process models
• what information is required and returned by our services/APIs
• what information is disseminated though instances of our event definitions

Broad-range, enterprise-wide data modeling initiatives, which aim to capture and harmonize the data models of most applications across the (extended) enterprise to define a true Common Data Model, are very demanding and expensive, and not many businesses are willing to invest in this.

However, a certain degree of data governance is essential, as ambiguity and inconsistency carry significant risks and costs.  As minimum, a version-aware repository of schemas is necessary, along with custom standards about how to define, extend, and use the schemas.  Schema Centralization is still a valid pattern to avoid inconsistencies and duplication of the same data definitions across different service / API contracts.

For XML, XML Schema has been around for a long time and has extensive tooling support.  The lack of an equivalent for JSON has proven a liability for REST service governance for some time, but this gap has been filled in the last years by JSON Schema.

The clear definition of messages and events according to custom modeling standards (and there are many ways to do this) is a cornerstone of Structured Integration.  Ideally, such standardization should extend to data semantics.

Conclusion

My "integration lozenge" is a very simple and high-level depiction of the interrelated aspects that every Systems Integration professional should care about.  Of course, there are many details of protocol, format, and logic (process logic, business rules, data mapping, etc.) that sit underneath.
The effort here is to conceptualize and address the relevant aspects of this domain in a way that is systematic and yet simple, without the burden of heavyweight methodologies.

I hope my blog will be able to meet this goal.