data-mesh
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| data-mesh [2023-08-08 10:45] – mrks | data-mesh [2025-01-13 11:59] (current) – mrks | ||
|---|---|---|---|
| Line 20: | Line 20: | ||
| * [[https:// | * [[https:// | ||
| * [[https:// | * [[https:// | ||
| + | * [[https:// | ||
| Line 73: | Line 74: | ||
| * Data products must **interoperate to get value in forms of higher order datasets and insights** | * Data products must **interoperate to get value in forms of higher order datasets and insights** | ||
| + | |||
| + | ===== Designing data products ===== | ||
| + | |||
| + | |||
| + | [[https:// | ||
| + | |||
| + | * "Data products **package structured, semi-structured or unstructured analytical data for effective consumption** and data driven decision making, keeping in mind specific user groups and their consumption pattern for these analytical data." | ||
| + | * "One of the most prevalent misunderstandings is **conflating data products with data-driven applications**. Data products are natively **designed for programmatic access and composability**, | ||
| + | * "**No single data product should be owned by multiple domains**, as this can lead to confusion and finger-pointing over quality issues." | ||
| + | * "While it's important to consider domain ownership early, **it’s often more efficient to have a single team develop all the necessary data products** to realise the use case at the start of your data mesh journey. Splitting the work among multiple teams too early can increase coordination overhead, which is best delayed. Our recommendation is to begin with a small, cohesive team that handles all data products for the use case. As you progress, use “team cognitive load” as a guide for when to split into specific domain teams." | ||
| + | * " | ||
| + | * "**For structured data, this usually means a single denormalized table**, and for semi-structured or unstructured data, a single dataset. Anything larger is likely trying to do too much, making it harder to explain its purpose in a clear, concise sentence and reducing its composability and reusability." | ||
| + | * "In our experience, **it's beneficial for each data product identified earlier to have its own source control repository and associated deployment pipeline**, allowing for independent management of its lifecycle." | ||
| Line 125: | Line 139: | ||
| * Links between schemas, semantic links between the schema so that you can join this thing | * Links between schemas, semantic links between the schema so that you can join this thing | ||
| * Link of the data | * Link of the data | ||
| + | |||
| + | |||
| + | |||
| + | ===== Governing data products using fitness functions ===== | ||
| + | |||
| + | [[https:// | ||
| + | |||
| + | * Decentralization requires governance at scale | ||
| + | * Small autonomous teams need more governance than a central team | ||
| + | * "The term “data product“ has unfortunately taken on various self-serving meanings [...]" | ||
| + | * Data product: [[https:// | ||
| + | * " | ||
| + | * Fitness function | ||
| + | * Test used to evaluate how close a given implementation is to its stated design objectives | ||
| + | * Shift left on governance: identify potential governance issues early to enable proactive improvement | ||
| + | * Prioritization | ||
| + | * Goverance by rule over Governance by inspection | ||
| + | * Empowering teams to discover problems over Independent audits | ||
| + | * Continuous governance over Dedicated audit phase | ||
| + | * Organization wide data catalog to index data products with rich metadata | ||
| + | * Assertions within the catalog: Tools like Collibra and Datahub provide hooks | ||
| + | * Assertions outside of the catalog | ||
| + | * Assessing metadata with LLMs | ||
| + | * Reporting of results | ||
data-mesh.1691484302.txt.gz · Last modified: by mrks