Personal insights and notes.

How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh – Zhamak Dehghani (2019-05-20, martinfowler.com)

• The data mesh is a technical and organizational architecture pattern to solve the organizational problem of needing multiple teams to develop multiple components and applications due to the size of the data analytics application landscape. The pattern describes how to break a data analytics landscape into services which can be owned by business and technical teams.
• View data as a product
• The business department is responsible for the data product
• The product needs to have a certain quality to be attractive for others
  • Discoverable
  • Addressable
  • Trustworthy and truthful
  • Self-describing
  • Inter-operable and governed by global standards
  • Secure and governed by a global access control
  • Built by a cross-functional domain data team with data product owner and data engineers
• The data product must support
  • General
    • Product evolution and product versioning
    • Unified data access control and data access logging
    • Pipeline implementation and orchestration
    • Data discovery
    • Data catalogue registration
    • Publishing
    • Data lineage
    • Monitoring, alerting, logging
    • Product quality metrics: definition, evaluation, collection and sharing of the metrics
  • Use-case specific
    • Caching
    • Data de-identification
    • Data pseudonymization

Data Mesh Principles and Logical Architecture – Zhamak Dehghani (2020-12-03, martinfowler.com)

Data Mesh: an Architectural Deep Dive – Zhamak Dehghani (2021-11-26, infoq.com)

You Can't Buy Integration – Brandon Byars (2021-12-14, martinfowler.com).

• You Can't Buy Integration argues that the focus on integration architecture must be put on the interfaces to abstract capabilities and hide implementation details. One must not build the integration architecture from bottom up, i.e. from tools to interfaces, but top-down from interfaces to tools. This is the application of general application principles like SOLID and package principles to integration.
• You Can't Buy Integration argues that integration is not just arrows between systems but that it is systems itself which need to follow system design principles like abstraction and reuse. Integrating two systems means abstracting the system details of the first system to generalize it's capabilities to have a reusable interface on that system. The second step is then to integrate the second system with this generalized capabilities interface. It's basically creating two adapters, one abstracting the first system and another to use this abstraction with the details of the second system.
• Architect responsibility: Create a concept of the capabilities interfaces for integration implementation.
• Don't be afraid of channel APIs like backends for frontends who's purpose it is to translate one interface into another interface better suiting the needs of the user because it separates system and channel which change for different reasons at different times and speed.
• Focus on interfaces, interfaces are more important than implementation
• Interfaces abstract capabilities not systems
• Every interaction between two processes/components/systems is through interfaces
• If a part of a data analytics application grants access to multiple down-stream components, then this is done via an interfaces and this interface must abstract the capabilities and not the systems. This means that the interface must add value and not just pass through data.