Systems integration
Predictable and avoidable problems

Real projects (with real numbers)

These are three actual systems integration projects taking place in similar organizations, with wildly different outcomes.

  • 1. Expensive learning experience

    After 18 months, an integration project is $70 million over budget and must be restarted from scratch.

    Of the three projects in this list, this one has the biggest team.

  • 2. Elastic schedule

    A project is $2 million over budget, and the integration platform migration is 3% completed.

    • 5 enterprise systems
    • 4 external partners
    • Message-based integration
    • MuleSoft platform

    Of the three projects in this list, this one has the most advanced technology stack.

  • 3. Going above and beyond

    The vendor of one of the core enterprise systems lowers estimates by 20% when they see the integration architecture, and yet the project is delivered early, under budget.

    • 14 enterprise systems total (3 core)
    • 25+ external partners
    • Pub-sub integration
    • Biztalk platform

    Of the three projects in this list, this one has the most dated technology stack, the smallest team and the largest number of interfaces.

Three different projects, three different outcomes, one conclusion: money and technology never hurt, but they're not the key ingredient of a successful systems integration project.

The problem with systems integration projects

Most Enterprise Architecture frameworks in use today (such as TOGAF) derive in some way from the Business Systems Planning methodology created at IBM in the 1970s. Over the decades, there's been a lot of studies and whitepapers documenting the flaws of those frameworks and the way they tend to create a disconnect between IT and business stakeholders, and yet most EAI projects are still based on such frameworks. (See criticisms in the Wikipedia article.)

This is not new information. A paper from Goodhue, Quillard & Wybo published in 1992 already outlined the main cause of failures in EAI projects: the wide gap between what the enterprise architects design and the actual requirements of business stakeholders. In 2003 it was reported that 70% of integration projects fail.

Projects don't fail because of the selected platform, or because of insufficient budgets, or because of incompetent developers. They fail because IT loses sight of what the business actually needs.

Here's how it usually happens:

  • Early in the project, a target architecture is prepared, with varying degrees of participation of business stakeholders.
  • Implementing this target becomes the goal of the project, without further validation from the business along the way.
  • The architecture takes a life of its own, generating an ungodly amount of "write-only" documentation and frameworks that serves no purpose other than reinforcing the authority of the Target.
  • The stakeholders who could raise red flags about the project are smothered by a huge stack of As-Is To-Be roadmaps, BPMN swimlanes and endless UML diagrams, preventing them from seeing the monstrous Rube Goldberg machine being slowly assembled by independent teams who all lack a clear understanding of the business.
  • By the time the failure is obvious, "insufficient requirements" is identified as the root cause of the fiasco.

After this failure, the organization either embarks on a costly do-over, or outsources the whole thing to a third-party that typically will follow the same pattern and deliver a half-baked solution at an obscene cost. Since too much money has been sunk in the project already, someone declares victory and IT moves on while business stakeholders have to make do with poorly integrated systems.

The hard truth

When it comes to systems integration, there will never be a situation where all business requirements can be thoroughly defined in advance. And this applies to all possible scenarios:

  • rolling out a brand new integration platform for brand new enteprise systems
  • integrating a new enterprise system with existing ones
  • integrating a new version of an enterprise system with existing ones
  • replacing an existing integration platform
  • replatforming one or many enterprise systems without changing the integration platform
  • etc.

There are three main reasons why it's impossible to define thorough business requirements:

  1. Organizations are always in flux, with endless changes at both the micro and macro levels
  2. Most organizations have limited or no formally documented business processes, and those that do are deluding themselves in believing that they are accurate
  3. Integrating enterprise systems always generate a flurry of emergent requirements, of which there are an exponentially larger number as the number of enterprise systems increase

Rather than accepting the unknowable nature of human organizations and building processes accordingly, IT architects tend to create (or rely on) flawed models upon which systems are designed and built; from there it's usually only a matter of time and money before failure is observed.

Mitigating the risk of failure

Systems integration projects tend to fail. This places any organization starting an integration project in the undesirable position of having to defeat the odds. Rather than roll the dice or to dispute statistics built over decades of IT history, the leaders of a new integration project should take these risks into account and do everything in their power to preemptively address the most frequent source of failure, which has been established as a disconnect between IT and business.

This can be done by following a few rules of thumb.

  • Bake the need to provide evidence of success to business stakeholders at regular interval into the project roadmap. This can be done by implementing a project management structure inspired by the Incremental Commitment Spiral Model, which is both thorough and easy to implement.
  • Minimize the amount of time spent on "write-only" documentation, whether they are formal or agile. A backlog of several hundreds user stories is no more useful than a network volume filled with Business Architecture Requirement Documents and other forms of dross. Energy is better spent in implementing a smaller set of value-added features that won't become dead wood if the project is suddenly interrupted.
  • Rather than ask for directions from vendors or advisory firms, ask for feedback once internal resources with a solid knowledge of the organization have completed a first draft of the architecture and design. Otherwise, the solution will be a composite of current trends in the market masquerading as best practices.
  • Shield enterprise systems from each other to maximize resilience. This can be achieved by at the very minimum implementing an API gateway or an Enterprise Service Bus to abstract interactions with other enterprise systems.
  • Define each enterprise system according to its interaces - how it talks to the rest of the organization - and engage relevant business stakeholders in policing the development, maintenance and usage of those interaces.

What about technology?

For a long time, the EAI industry has been focused on automating business processes and minimizing coding. Modern cloud-based integration platforms (aka IPaaS) offer the same kind of features.

Can you guess what bleeding edge platform is described below?

"The platform is comprehensive because of its complete top-to-bottom implementation capabilities. This begins with a visual modeling environment that supports graphical design of business processes, to link systems and interfaces together and generate a process in the underlying integration platform without doing any programming. The process models are more than diagrams; they link directly to the applications to be integrated, so that deploying a process is a simple single-click operation."

Answer: this is actually a quote from a marketing brochure for Webmethods published in 2003. Webmethods was once named by Deloitte as the fastest-growing software company in North America but has since been acquired by Software AG and currently has a market share of 1.8% according to datanyze.

This doesn't mean that Webmethods or any other specific platform is bad; it just shows that if technology was the answer, the endless waves of failed integration projects would have resorbed by now.

When it comes to systems integration, there are no shortcuts. No cloud product, no magical AI platform, no visual design environment will address the root cause of failure, which is the disconnect between IT and business. If anything, adding layers of automation and software will only make things more difficult.

Questions?

Learn about us Read the FAQ