Overview
As CEO and Product Manager at Zeenea, a next-generation data catalog vendor, I often get to discuss metadata management expectations with our customers, prospects and partners.
In all our discussions, one topic comes up time and time again: Data Lineage.
As a concept, Data Lineage seems universal: whatever the sector of activity, any stakeholder in a data-driven organization needs to know the origin (upstream lineage) and the destination (downstream lineage) of the data they are handling or interpreting. And this need has important underlying motives.
For a Data Catalog vendor, the ability to manage Data Lineage is an essential component in its offer. As is often the case however, behind a simple and universal question lies a world of complexity that is difficult to grasp.
I believe there are several reasons for this complexity
💡 The first reason is that while the question is always the same, the answers will vary substantially from one interlocutor to another,
💡 The second reason is more operational - How can we capture, maintain, and update the huge mass of information that lineage represents?
💡 The third reason involves ergonomy. How can such a volume of information be rendered in a digestible and relevant way?
It is these different aspects that I will elaborate on in this document, specifying throughout the approaches that we favor at Zeenea.
