Take The Confusion Out Of Data Mesh
Confused by data mesh? You’re not alone. While the domain-centric and data-governance-by-design principles feel simple and intuitive, the big question is, “How do you execute on data mesh?”
OK, breathe. Data mesh principles are not new. Organizations already model data, stand up data warehouses, master their data, and ensure data quality. Data governance artifacts to define and establish policies, check! Utilization of modern flexible data warehouses and knowledge graphs to navigate data and data relationships for insight, check! Ontologies, taxonomies, and data catalogs created and populated by data subject-matter experts, check!
What data mesh does is shift the data strategy from predominantly analytic visualization to data in motion and real-time solutions. Data development and application development collide as data is set in motion for real-time, distributed, and internet-of-things applications. Data mesh models data as a twin of the business in the language of the business. This makes it possible to work with data in a declarative fashion and simplify integration of data components with application components. Think how rapidly you could create and scale insights-driven solutions with a common framework.
Consider the logistics sector. Trucking, rail, and shipping are deeply connected to supply chain operations and customers. The data ecosystem relies on several domains internally and under control, as well as external data with sharing and consent policies. Data mesh addresses the foundation of interoperability by applying standards, definitions, and protocols specific to the handoff points for each decision and step in the process. When there is a 20-minute backup on a highway, data from the highway infrastructure can be picked up in real time and used to optimize a truck route to keep deliveries on time. Infrastructure and truck coordinate and communicate in a common language for the right outcome.
The five factors that shape the application of data mesh to evolve from watching the world to influencing the world through data-driven value are:
- Semantics. Expand logical domain definitions and models to represent semantic views and understanding. By applying the business language in the form of relationships, classifications, labels, and tags, working with data becomes declarative. In the no-code/low-code application development environments, semantics improves and speeds up the mapping between the right data and what is needed in a business process. Better interoperability between data and application results.
- Data products. Applications rely on services and APIs to access data sources and pipelines. These elements or components are data products. Data products output a data source, event, query, schema, control, insight, etc. They are designed to match the data requirement of the application and take on the heavy lifting of handling complex data logic to simplify application process routing, or they deliver a service to balance and optimize the cost to performance for production payloads.
- Portfolio management. The number one challenge for data engineering, according to the Forrester Analytics Business Technographics® Data And Analytics Survey, 2021, is data product management. As data products are defined at a more granular level, portfolio management is crucial to maintaining order and ensuring alignment, speed, and reuse of capabilities. Power leveling of data product portfolio management comes with harmonizing data development with the broader solution and business digital portfolio. Thus, data comes into alignment by capability, priority, and defined value and outcomes.
- DataOps. Rather than executing data development and engineering for monolithic deployments, DataOps takes on the agile and continuous integration and delivery of data products. Architects at the enterprise and line-of-business level provide patterns and blueprints as starting points that offset potential technical debt. Data engineers own the products they develop, meaning DataOps takes on responsibility of the quality, speed, and outcomes for data provisioning and through ongoing optimization and lifecycle management. Thus, data is governed by design and not an afterthought.
- Federation. Circling back to the semantics of the data, connections to the subject matter experts must be strong and innate. Large enterprises and global organizations are building organization and operation models to cover centralized data and governance foundations and shared artifacts while also pushing data development up into solutions teams in lines of business. In lines of business, data engineers are elevated to members of the overall application development team. They then take responsibility to provide their products and domain-centric knowledge back to the centralized data services environment.
Up your business outcomes with data-mesh-driven data decisioning. Assess your competency in the five data mesh factors for success across data management, engineering, governance, and consumption practices. Ensure that these competency centers are coordinated and intertwined. Remember, data mesh is not just about the data; it is about making data work for a resilient, competitive business.