Enterprise Architecture in a nutshell
Enterprise architecture is a fairly complex discipline that requires a thorough understanding of the organization's business as well as a solid grasp of technology.
In the absence of an experienced team of architects, an organization can still progress towards a mature enterprise architecture practice by carefully mapping business activities with relevant IT systems.
The first step is to document business functions. If this task appears daunting, one can simply rely on the wisdom of Peter Drucker, who stated that organizations have fundamentally only two functions (marketing and innovation), and that everything else is support:
Once the business functions have been identified, high-level tasks performed in the context of these business functions can be documented. Those tasks are called business processes:
A word of caution
At this stage, it can be tempting to invest heavily in documenting business processes. In fact, most enterprise architecture frameworks will even recommend it. But this would be a mistake. Most enterprise architecture frameworks derive in some way from the Business Systems Planning methodology created at IBM in the 1970s, which has a terrible track record. Those frameworks are designed in the abstract, assuming that organizations can successfully follow a top-down approach, which is rarely the case.
If your organization is new to formal enterprise architecture, it's best at first to keep things very, very simple. There will be opportunities later to refine the list of business processes.
Once basic business processes have been identified, the relevant enterprise systems can be added to the big picture. At this point, it's best to focus on core systems - those that are badly needed to perform the most important business processes.
An organization will often have two or more applications with overlapping capabilities, either because different departments couldn't agree on a single product, or because some kind of migration is happening. To make the overall enterprise architecture a little more future-proof, it can be convenient to group similar applications in abstract systems. In fact, it's usually a good idea to only have abstract systems in relation with business processes, for reasons that will become clear later in this article:
And that's it! You now have the main components of enterprise architecture: business functions, business processes, and core enterprise systems.
The plot thickens
Enterprise systems are often managed like independent silos (which is by itself a problem), but reality doesn't fit in neat little boxes. There's some chatter going on between applications:
Making applications talk to each other directly is called point-to-point integration, and is usually a bad idea. In a point-to-point architecture, the code required to make one application talk to another is heavily dependent on the specific product (and version). For instance, there's no universal way for a CRM to let a payroll system know that Bob has earned a 5% commission; someone has to write code to make data flow from the CRM application (such as Maximizer*) to the payroll application (such as SAP SuccessFactors*). If either Maximizer or SAP SF is upgraded or replaced, this integration will no longer work, and it won't be easy to give Bob his 5% commission.
* please note that we use those products as examples, there are many other competing products offering similar features
ETL to the rescue? Not so fast.
Basic point-to-point integration is often done using an ETL process. While this approach may work with very simple applications, it's almost never a good long-term solution, especially with COTS applications where the organization has limited control over how data is stored.
Biggest pitfalls:
- Data found in the database may or may not represent the actual state. For instance, some applications may work extensively in memory and use a proprietary destaging algorithm. In such case, ETL processes would pick up incorrect data and there would be no easy way to detect the problem.
- Most enterprise applications use a software-defined database model, often powered by an ORM library such as Hibernate, or by a scaffolding framework like Play. This means that the database model can change drastically between releases of the application, breaking existing ETL processes, which in turn can delay upgrades.
- Most enterprise applications vendors advise against accessing the underlying database directly; they instead provide custom frameworks, API or utilities to interact with their product programmatically.
- As products increase in scale and complexity, sophisticated designs based on microservices are becoming the norm. This type of application is often designed for large scale computing, and may introduce new paradigms such as eventual consistency, making traditional ETL processes unreliable at best.
- Enterprise applications often rely on distributed transactions that can last from a few seconds to a few hours, and that may rollback at any time. An ETL process picking up data while a distributed transaction is in flight maybe provide incorrect data to the destination system; this is a textbook case of a race condition, always a red flag.
- ETL processes are usually heavily biased towards incremental changes; the destination will be informed of new or updated records, but may not be informed when previously submitted records are no longer present in the source. This leads to the apparition of orphaned records and can negatively impact aggregate operations (such as sums or averages).
Avoiding point-to-point integrations (and especially database-driven ones) is the first architectural pattern that an organization should embrace.
To avoid building a brittle integration layer, a simple rule can be followed: core enterprise systems should never talk directly to each other.
Instead, systems should interact with "the organization" itself:
This can be achieved using one of the following approaches:
- Build a traditional SOA platform, such as an enterprise service bus
- Rely on an API gateway and require system vendors or integrators to publish their API
- Rely on a cloud-based solution (IPaaS)
While those solutions are conceptually similar, they have different pros and cons.
| Approach | Pros | Cons |
|---|---|---|
| Enterprise service bus (ESB) |
|
|
| API gateway |
|
|
| IPaaS |
|
|
Which one is best?
Each organization is unique so there is no universal "best" approach for enterprise systems integration. Simple rule of thumbs:
- If your organization cannot invest heavily (in time and money), forget the ESB approach.
- If your organization relies on software created in-house for important business processes, be very careful before adopting an IPaaS approach.
- If your organization relies on legacy platforms (such as HP-3000 or OS/390) or on midrange or mainframe computers (such as AS/400 or zSeries) for important business processes, forget the IPaaS approach.
- In doubt, use an API gateway and leave the heavy lifting of API crafting to the enterprise systems vendors.