Robertkarel_2Jameskobielus_2 By Rob Karel and James Kobielus

The integration software market is a busy place, with technologies such as ETL (extract, transform, and load), ESB (enterprise service bus), CDC (change data capture) and IC-BPMS (integration-centric business process management suites) crowding the landscape.  The technologies within this "integration acronym buffet" address a common requirement: they either physically or virtually deliver information from point A to point B with integrity.  (For more information on the variety of integration options available, see "2009 Update: Evaluating Integration Alternatives.") Some approaches, such as ETL and ESB, have developed into multi-billion dollar stand-alone markets.   But other integration alternatives, while valuable, haven’t survived as well as a standalone market segment.   EII (enterprise information integration) is one example of this. 

In a nutshell, EII — also often referred to as data federation — is an umbrella term that arches over a collection of technologies and best practices for providing custom views into multiple data sources as a way of integrating data and content for real-time read and write access by applications.   Though integration professionals have been using EII for many years in niche applications, it has struggled to find the ideal scenarios in which it has a clear advantage over other integration approaches.   

Where enterprise data warehousing (EDW) is concerned, EII provides an on-demand, decentralized "virtual" alternative, but has not dislodged ETL’s role in the centralized, batch-oriented EDW architectures that predominate throughout the corporate world. For real-time and operational business intelligence (BI) requirements, EII supports low-latency queries across distributed data environments, but it competes against other approaches such as trickle-feed ETL and CDC.  And for real-time read/write access across disparate databases, EII competes against more versatile approaches such as ESB and information as a service (IaaS).

In more recent years, EII has found adoption as an adjunct to enterprises’ core EDW, BI, and transactional computing integration strategies–not as the main approach.  When they implement EII, information architects generally use this technique sparingly, and primarily for tactical, project-based requirements — not as an enterprise standard.  So it’s no surprise that a stand-alone EII tools marketplace has failed to develop, or that most EII vendors have slowly disappeared via acquisition into larger BI, EDW, data integration, or database platforms such as Business Objects’s, Red Hat’s, and Sybase’s acquisitions of Medience, MetaMatrix, and Avaki, respectively.   Meanwhile, the remaining EII pure-play vendors such as Composite Software have wisely shifted their solutions and messaging towards a broader focus on SOA (service-oriented architecture) and IaaS.   

Ironically, as the EII-branded market continues to consolidate and merge with other data management markets and essentially disappear as a recognized standalone segment, EII’s earliest goal of enabling virtual data warehousing may now be poised to become reality.   As the complexities of today’s business operations increasingly requires real-time decision making, real-time data warehousing options are once again being discussing by data architects across industries.  As Forrester analyst James Kobielus discussed in his research, "Federation: Sharpen Your Focus On Vast Constellations Of Data," multiple use cases have arisen where leveraging EII tools and techniques may be the best way to deliver information insights in a complex, heterogeneous environment.  And federated use cases go beyond data warehousing and business intelligence.  Architects designing solutions to manage complex enterprise content management (ECM) and master data management (MDM) strategies also consider federation as a legitimate means to bridge information siloes.

Does this mean the standalone EII market is ready for a comeback?  Not quite.  What it does mean is that EII may have finally found its place in the software world.  Vendors building data management solutions focusing on areas such as IaaS, SOA, BI, data integration, data warehousing, and MDM should continue to invest in seamlessly integrating EII-like federation capabilities as an embedded integration technique available to their customers.  An executive at a data integration software company recently shared with us his perception of how their customers have evolved to think about EII: "What was once called EII are really just a set of integrated access, abstraction, federation, and delivery capabilities behind (a) Virtual Data Federation as a complementary data integration tool in the DI project toolbox and (b) IaaS as an enterprise data virtualization architecture approach for the enterprise." So integration architects should take a step back and consider all of the options within the integration acronym buffet not as competing tools, but complementary techniques that must build the foundation for your organization’s cross-enterprise integration competency.