====== Data Mesh ====== Resources, personal insights and notes. ===== Resources ===== * [[https://martinfowler.com/articles/data-mesh-principles.html|Data Mesh Principles and Logical Architecture – Zhamak Dehghani (2020-12-03, martinfowler.com)]] * [[https://martinfowler.com/articles/data-monolith-to-mesh.html|How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh – Zhamak Dehghani (2019-05-20, martinfowler.com)]] * [[https://www.infoq.com/presentations/data-mesh-concepts/|Data Mesh: an Architectural Deep Dive – Zhamak Dehghani (2021-11-26, infoq.com)]] * [[https://www.thoughtworks.com/de-de/what-we-do/data-and-ai/data-mesh|Data Mesh (Thoughtworks)]] * [[https://www.datamesh-architecture.com/|Data Mesh Architecture]] * [[https://www.datamesh-governance.com/|Data Mesh Governance]] * [[https://datameshlearning.com/|Data Mesh Learning]] * [[https://datameshlearning.substack.com/|Data Mesh Learning Newsletter]] * [[https://www.thoughtworks.com/insights/articles/data-mesh-in-practice-getting-off-to-the-right-start|Data Mesh in practice: Getting off to the right start (Part I)]] * [[https://www.thoughtworks.com/insights/articles/data-mesh-in-practice-organizational-operating-model|Data Mesh in practice: Organizational operating model Data Mesh in practice: Organizational operating model (Part II)]] * [[https://www.thoughtworks.com/insights/articles/data-mesh-in-practice-product-thinking-and-development|Data Mesh in practice: Product thinking and development (Part III)]] * [[https://www.thoughtworks.com/insights/articles/data-mesh-in-practice-technology-and-the-architecture|Data Mesh in practice: Technology and the architecture (Part IV)]] * [[https://www.innoq.com/en/blog/thoughts-on-a-canonical-data-model/|Why You Should Avoid a Canonical Data Model]] * [[https://www.innoq.com/de/articles/2023/07/what-is-a-data-product/|What is a data product?]] * [[https://www.thoughtworks.com/insights/articles/recommendations-for-a-successful-data-mesh-implementation|10 recommendations for a successful enterprise Data Mesh implementation]] ===== How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh ===== [[https://martinfowler.com/articles/data-monolith-to-mesh.html|How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh – Zhamak Dehghani (2019-05-20, martinfowler.com)]] * **The data mesh is a technical and organizational architecture pattern** to solve the organizational problem of needing multiple teams to develop multiple components and applications due to the size of the data analytics application landscape. The pattern describes how to break a data analytics landscape into services which can be owned by business and technical teams. * **View data as a product** * **The business department is responsible for the data product** * **The product needs to have a certain quality to be attractive for others** * Discoverable * Addressable * Trustworthy and truthful * Self-describing * Inter-operable and governed by global standards * Secure and governed by a global access control * Built by a cross-functional domain data team with data product owner and data engineers * **The data product must support** * General * Product evolution and product versioning * Unified data access control and data access logging * Pipeline implementation and orchestration * Data discovery * Data catalogue registration * Publishing * Data lineage * Monitoring, alerting, logging * Product quality metrics: definition, evaluation, collection and sharing of the metrics * Use-case specific * Caching * Data de-identification * Data pseudonymization ===== Data Mesh Principles and Logical Architecture ===== [[https://martinfowler.com/articles/data-mesh-principles.html|Data Mesh Principles and Logical Architecture – Zhamak Dehghani (2020-12-03, martinfowler.com)]] * **Define data consumer roles**: e.g. data analyst, data scientist * Design **interfaces** to meet the data **consumer roles' needs** * It is more important to have a **sovereign data product** than to trade infrastructure synergies for dependencies. * **Domain-oriented teams own decentralized data products** * Data product consists of **code, data, metadata, and infrastructure** * Data products are **discoverable, secure, explorable, understandable and trustworthy** * The **quality** of data products **is measured in data quality, lead time to data consumption, data user satisfaction** * **Data infrastructure enables teams to be autonomous** * Teams can **autonomously build, deploy, execute, monitor and access their data product** * **Data products can be used in combination** * **The emergent graph of data products must be managed** * Data products have **independent lifecycles, builts, deploys and teams** * Data products must **interoperate to get value in forms of higher order datasets and insights** ===== Designing data products ===== [[https://martinfowler.com/articles/designing-data-products.html|Designing data products – Kiran Prakash (2024-12-10 martinfowler.com)]] * "Data products **package structured, semi-structured or unstructured analytical data for effective consumption** and data driven decision making, keeping in mind specific user groups and their consumption pattern for these analytical data." * "One of the most prevalent misunderstandings is **conflating data products with data-driven applications**. Data products are natively **designed for programmatic access and composability**, whereas data-driven applications are primarily intended for human interaction and are not inherently composable." * "**No single data product should be owned by multiple domains**, as this can lead to confusion and finger-pointing over quality issues." * "While it's important to consider domain ownership early, **it’s often more efficient to have a single team develop all the necessary data products** to realise the use case at the start of your data mesh journey. Splitting the work among multiple teams too early can increase coordination overhead, which is best delayed. Our recommendation is to begin with a small, cohesive team that handles all data products for the use case. As you progress, use “team cognitive load” as a guide for when to split into specific domain teams." * "**Having a consistent blueprints for all data products will make this transition of ownership easier when the time comes**. The new team can focus solely on the business logic encapsulated within the data products, while the organization-wide knowledge of how data products are built and operated is carried forward." * "**For structured data, this usually means a single denormalized table**, and for semi-structured or unstructured data, a single dataset. Anything larger is likely trying to do too much, making it harder to explain its purpose in a clear, concise sentence and reducing its composability and reusability." * "In our experience, **it's beneficial for each data product identified earlier to have its own source control repository and associated deployment pipeline**, allowing for independent management of its lifecycle." ===== Data Mesh: an Architectural Deep Dive ===== [[https://www.infoq.com/presentations/data-mesh-concepts/|Data Mesh: an Architectural Deep Dive – Zhamak Dehghani (2021-11-26, infoq.com)]] * **Data products should not be monolithic** as data warehouses and data lakes. * **Follow domain-driven design**: Decouple architecture, data and data ownership by business domain * **Decomposition by business domain** necessary to manage business and technical complexity. * As with **microservices** one team should have a **slice of business functionality and data** while maintaining the full stack of their product. * **Create interfaces to the domain, capabilities and APIs** * **Data products offer a baseline usability** * Analytical APIs * **Provide historical data over time** * **Polyglot data** means that you might have many different spatial technologies of that time variant data. * Events for app developers * SQL for reporting * Columnar files for data scientists * **Design your API to be only read-only and be temporal time variant**. Data needs to be immutable, otherwise it would have cascading effects. * **A data product is an architectural quantum** consisting of * Code for * Transformations * Serving data * Embedding policies * Data * Actual data * Metadata * Storage * SLOs * Schema * Syntax and semantics * Documentation * Policy configuration * One operational system can have one or more data products * **Data products can be based on other data products creating a cascade** * **Multi-Plane data platform** * Utility plane * Orchestation and provisioning of Spark jobs * Creating a Kafka topic * Provisioning service accounts and storage accounts * Provisioning and management of the data product lifecycle * Declaratively create them * Read data products and versions * Consume * Product API * Schema * Semantics links to other data products they are related to * Browse and search data products * **Three layers of interconnectivity** * Flow of data between input and output ports, the lineage * Links between schemas, semantic links between the schema so that you can join this thing * Link of the data ===== Governing data products using fitness functions ===== [[https://martinfowler.com/articles/fitness-functions-data-products.html|Governing data products using fitness functions – Kiran Prakash (2024-09-05, martinfowler.com)]] * Decentralization requires governance at scale * Small autonomous teams need more governance than a central team * "The term “data product“ has unfortunately taken on various self-serving meanings [...]" * Data product: [[https://www.oreilly.com/library/view/building-evolutionary-architectures/9781491986356/ch04.html|architectural quantum]] * "Smallest valuable unit of analytical data, sourced from data streams, operational systems, or other external sources and also other data products, packaged specifically in a way to deliver meaningful business value. It includes all the necessary machinery to efficiently achieve its stated goal using automation." * Fitness function * Test used to evaluate how close a given implementation is to its stated design objectives * Shift left on governance: identify potential governance issues early to enable proactive improvement * Prioritization * Goverance by rule over Governance by inspection * Empowering teams to discover problems over Independent audits * Continuous governance over Dedicated audit phase * Organization wide data catalog to index data products with rich metadata * Assertions within the catalog: Tools like Collibra and Datahub provide hooks * Assertions outside of the catalog * Assessing metadata with LLMs * Reporting of results ===== You Can't Buy Integration ===== [[https://martinfowler.com/articles/cant-buy-integration.html|You Can't Buy Integration – Brandon Byars (2021-12-14, martinfowler.com)]]. * You Can't Buy Integration argues that the **focus on integration architecture must be put on the interfaces to abstract capabilities and hide implementation details**. One must not build the integration architecture from bottom up, i.e. from tools to interfaces, but top-down from interfaces to tools. This is the application of general application principles like [[https://en.wikipedia.org/wiki/SOLID|SOLID]] and [[https://en.wikipedia.org/wiki/Package_principles|package principles]] to integration. * You Can't Buy Integration argues that **integration is not just arrows between systems but that it is systems itself which need to follow system design principles like abstraction and reuse**. Integrating two systems means abstracting the system details of the first system to generalize it's capabilities to have a reusable interface on that system. The second step is then to integrate the second system with this generalized capabilities interface. It's basically creating two adapters, one abstracting the first system and another to use this abstraction with the details of the second system. * **Architect responsibility**: Create a concept of the capabilities interfaces for integration implementation. * **Don't be afraid of channel APIs** like backends for frontends who's purpose it is to translate one interface into another interface better suiting the needs of the user because it separates system and channel which change for different reasons at different times and speed. * **Focus on interfaces, interfaces are more important than implementation** * **Interfaces abstract capabilities not systems** * **Every interaction between two processes/components/systems is through interfaces** * If a part of a data analytics application grants access to multiple down-stream components, then this is done via an interfaces and this interface must abstract the capabilities and not the systems. This means that the **interface must add value and not just pass through data**.