The Four Data Mesh Principles

Data is rapidly revolutionizing our society. From e-commerce and healthcare to finance systems, nearly every industry utilizes data to drive their functions. So, how a company manages and organizes data significantly affects its operational efficiency. In the data mesh architecture, data is managed by individual domain teams, offering more flexibility and collaboration.

In this article, we’ll explore the key data mesh principles, its benefits, and strategies to overcome common challenges during adoption. 

What is Data Mesh?

Having numerous data sources means the same data can be present in different formats across tables, leading to data inconsistency. That’s when centralized data architecture emerged as a solution, gathering all the data into a single central repository and utilizing it. 

Later, ThoughtWorks thought that data would become more efficient if business domains closest to the specific data own and manage it rather than a single team handling the entire data.

Zhamak Dehghani at ThoughtWorks founded the concept of data mesh in 2018. Though the idea is pretty new, many large enterprises can get the most out of their data with decentralized architecture. Let’s kick things off with the definition. 

Data mesh is an enterprise data management architecture that distributes the data ownership to relevant business domains. It allows data producers and consumers to collect, analyze, manage, and own the data they use, without relying on the centralized data team.

Key Principles of Data Mesh

Successfully integrating data mesh into your business is no easy feat.

It requires careful planning and implementation of four key principles. They are discussed below. 

Domain ownership

Decentralization through domain-driven ownership is the first principle of data mesh. This means domain teams own and serve the data that belongs to them. 

For example, customer service teams take the ownership of customer data, responsible for its storage, accessibility, and maintenance. While the marketing team is responsible for the storage, security, and accessibility of the marketing data. 

This allows other internal teams to use and analyze a particular domain’s data without having to query a large centralized data lake. The next principle, “data as a product,” deals with this concept more closely. 

Data as a product

Typically, data is used to make products. For example, a visualization dashboard containing population distributed by age is the product, while population data is used to create it. 

However, with data mesh, data itself is the product, and it is treated with the same care and attention as any other business product.

The domain owners are responsible for creating and maintaining high-quality data products, keeping end users in mind. The primary goal is to provide end users with easily accessible and quality data.

For example, the sales domain manages sales data and the marketing team manages promotions data. They treat their data like a product, ensuring it’s easily accessible, well-organized, well-documented, and of high-quality. 

In case the marketing team wants to analyze sales data for better promotions, they can easily access and use it. Here, the marketing team is the end user, while the sales team is the producer. 

Data products need not always be datasets or tables. They can be delivered via APIs, files, storage systems like Apache Hive, real-time streaming platforms like Kafka, and more, depending on the end users’ convenience and the producers’ policies. 

However, each data product is expected to have actual data, its metadata, clear documentation, and infrastructure to run it. 

Self-service data platform 

In the data mesh architecture, each domain not only owns and manages the data but also produces data products. However, not every business team has the technical proficiency to do that. 

The central IT team equips domain teams with a self-service data platform that simplifies creating and managing high-quality data products. The idea of a self-service data platform is to empower domain teams to create data products without needing much technical expertise. 

The underlying complexities of this platform are handled by the IT team. The self-service data platform should ideally have a low learning curve and be easily adaptable to business domain teams. 

Federated governance 

While decentralization spreads the data ownership, they still need to follow some global standard policies to maintain consistency across the organization. 

Each data domain team should register their data products with the central data governance platform. These platforms or teams run automated data governance policies that verifies all data products adhere to standard data quality and compliance. 

What are the Benefits of Implementing a Data Mesh?

• Enhanced responsibility: With the decentralized architecture, the domain teams own the data that is closest to them, making them accountable for the data they manage. 
• Increased collaboration: The second principle, “data as a product,” ensures that data from a specific domain is easily accessible to other domains. This promotes better sharing and collaboration across teams.
• Granular governance: Governance rules are applied at both central and local levels, ensuring higher data integrity and consistent quality standards.
• Scalability: As data grows, data mesh doesn’t pressurize the central repository; instead, it distributes the load to respective domain teams. 
• Reduced data silos: Data mesh avoids data silos by establishing the concept of data as a product, promoting collaboration across the organization.
• Discoverability: Implementing a data catalog in data mesh allows required data to be located through a simple user interface, enhancing data discoverability. 
• Flexibility: Data mesh decentralization promotes agility by providing teams the flexibility to respond quickly to changing trends. 

Data Mesh Challenges and How to Resolve them

Implementing data mesh architecture effectively within a business requires strategic planning and thoughtful measures to overcome common challenges. To ensure a successful integration, you must address these challenges proactively. 

Cultural change

Data mesh architecture distributes ownership and the responsibility that comes with it to the domain teams. However, not every domain team wants to take on the responsibility and extra work.

Moreover, if your data is managed by a single central team, adopting a decentralized data mesh can make them feel undervalued and they may stand against this transition. 

So, discuss the need for data mesh within your teams and provide them with proper training and resources to confidently take up data ownership.

Learning curve

Domain teams rely on the self-service data platform to create and maintain their data products. However, not everyone, especially non-technical users, will enjoy the learning curve associated with adopting the data platform. 

So, the data platform should be simple to use, with a user-friendly interface and easy navigation. 

Governance 

Different domain teams maintain multiple data products, which can lead to inconsistencies when not formatted properly. Moreover, as the number of data products grows, varying formats and different quality standards make it really tough for other teams to use them.

Therefore, the global and local governance rules to maintain data integrity and quality across domains should be defined. And every domain should follow these common data quality, security, and access control policies. 

Maintaining data discoverability 

Data catalogs are often used to make data easily discoverable to everyone within the organization. They act as an inventory of data products with metadata information that allows users to easily search and locate a particular data product. 

If any new products or deleted products information isn’t updated in the catalog, it can misguide users across the organization. To ensure catalogs are always up to date, implement a docs-as-code scheme where the catalog automatically updates with every pull request made. 

Practical Use Cases of Data Mesh

Let’s explore some use cases to illustrate how different industries can leverage the power of data mesh architecture.

Logistics and supply chain

Within the supply chain, where data consistency, efficiency, and real-time visibility are paramount, implementing data mesh architecture can reshape its key functions. 

Traditionally, inventory management is a challenge due to data-siloed systems. This led to inaccurate inventory level projections and increased carrying costs. However, with data mesh, a dedicated stock management domain team sources the inventory data and ensures its accuracy. 

Another key application of data mesh in logistics is transport management. By treating transport data as a product, the domain team integrates data from GPS trackers, maps, and weather forecasts to analyze and optimize delivery routes and reduce fuel costs.

E-commerce 

Large industries like e-commerce, with numerous domains operating different business functions, such as sales, promotions, supply chain, pricing, store space optimization, and vendor management greatly benefit from decentralized architecture. 

Moreover, these businesses should be highly responsive in a competitive market. Say Amazon drops a product price by 6%, Walmart should quickly respond—perhaps within an hour—by cutting the price by 7% to attract customers. 

This quick and responsive decision-making is only possible when the pricing data is owned and managed by the relevant domain team. 

Customer service operations

Customer service is always a top priority for any business. When a specific domain team owns and provides the customer data as a product, other teams, like the support team, can easily access it and reduce average ticket resolution time, while the marketing team can pull the required customer data and create personalized offers. 

Conclusion

Undoubtedly, data mesh is a revolutionary technology; large enterprises are already adopting it to improve data accessibility and make quick decisions. 

For smooth and effective adoption, companies should follow all four data mesh principles when designing the architecture. Understanding these principles is crucial before deciding whether to choose a central repository, a data mesh, or a hybrid architecture, where domain teams own their data, and each team implements a data lake to manage it.

FAQs

1. What are the 4 principles of data mesh?

Domain-driven ownership, data as a product, self-service data platform, and federated computational governance are the four data mesh principles. 

2. What is the concept of data mesh?

Data Mesh distributes data across different teams within an organization rather than storing it in a single location managed by a central team. The idea is to spread responsibility, improve accessibility, and enhance collaboration, enabling quicker and more informed decision-making.

3. Is data mesh the future?

Data mesh removes the data silos issue that is commonly found in traditional centralized approaches. With increased agility, improved data quality, and clearly defined ownership, data mesh can be the future, especially for large enterprises.