Hello all, does anyone know the initiative “Egeria - Open Metadata and Governance” https://egeria.odpi.org I just got to know the initiative last week and am trying to classify it or evaluate whether it is worth going deeper into it. Many greetings Michael

Egeria - Open Metadata and Governance | Best practices

arthurburkhardt

·3 years ago · Edited

Fully agreed that we should have better out of the box connectivity!
But we should also think about the context and audience. Collibra focuses on data analysts, whereas open lineage focuses more on the “dataops” persona.
Knowing which ETL processes run, when and when they crash is great, but too much irrelevant details for the typical data analyst. Collibra made a great three-tiered audience for data: Data use (data marketplace), data understanding (data catalog), data awareness (metadata lake).

There is quite a gap between an Apache Atlas and a Collibra, and I don’t think the focus is on dataops engineers just yet.

arthurburkhardt

·3 years ago · Edited

Fully agreed that we should have better out of the box connectivity!
But we should also think about the context and audience. Collibra focuses on data analysts, whereas open lineage focuses more on the “dataops” persona.
Knowing which ETL processes run, when and when they crash is great, but too much irrelevant details for the typical data analyst. Collibra made a great three-tiered audience for data: Data use (data marketplace), data understanding (data catalog), data awareness (metadata lake).

There is quite a gap between an Apache Atlas and a Collibra, and I don’t think the focus is on dataops engineers just yet.

ludovicjanssens

·3 years ago · Edited

There’s definitely truth in what you’re saying. Collibra has indeed a focus on data use, data awareness and data understanding with a focus on business stakeholders and business analysts.

However with the introduction of more Connectors, Data Observability/Data Quality and Technical Data Lineage and the lack of more visual (UML) modeling components, Collibra has clearly showed they also focus on the usage of the catalog for more technical people.

Looking at the latter, I would expect Collibra to do support at least those technologies that are mostly used or at least foresee metadata connectors to the technologies that do allow this capturing. This is a major frustration of many of my clients.

I agree that it would generate a lot of detail, but I believe a combination of metadata and AI may be helpful to make abstraction of metadata layers that are irrelevant to certain personas. This would be a feature I’d love Collibra to invest in

ludovicjanssens

·3 years ago · Edited

I would love to reopen this discussion, as I noticed we are having trouble capturing metadata that is easily found in open source tooling such as Amundsen and Marquez. Although Collibra has a much broader use case, it is harder to sell now with these competitors. DG should start at business, but starts mostly at IT who sees the benefit immediately. Being IT, they require good integration and broad compatibility with common technologies, especially Python and Spark. Both these technologies are well covered by these open standards such as Egeria (mentioned in the discussion and OpenLineage ( Home | OpenLineage). Advancing on this level would make Collibra a much easier sell, as its wide support for proprietary tools isn’t matched by many other competitors.

Ann Wuyts

·4 years ago · Edited

We have an Ideation item around this from 2019: Consider supporting/participating in the new Open Metadata Standard proposed by ODPI. Any thoughts?

michaelkolb

OP4 years ago · Edited

Thank you very much for your feedback. I feel confirmed in my first impression with your answers. The feedback from my network outside our company also showed a similar picture. Nevertheless, we will take another look at the activity in our team and continue to monitor the initiative. If there is anything new, I will post it here.

bryangoodliffe

·4 years ago · Edited

I have experience with Egeria from a prior role at a different company whose Research & Development group was a major contributor to Egeria.

Some of the major contributors felt that the enterprise metadata problem was too big, too daunting, and outside of the scope of what they could control in their little vendor ecosystem. As more and more different data sources and systems started to emerge they felt that there was no way to effectively catalog and govern the wide variety of systems. Thus, rather than one specific vendor solving the problem they felt that it should be crowdsourced.

The original goal of the project was to establish a metadata exchange standard so all the various systems could exchange information between each other. Egeria was intended to be the go-between of these systems. However in order for Egeria to do anything it must be fed metadata from some system(s) or catalog(s), which requires that system to be set up to import/export based on the Egeria standard. This is combined with all the functionality that needed to be built out on the Egeria platform which had to be crowdsourced.

In the last few years since it’s establishment it hasn’t gotten much traction, because each catalog vendor and customer views the world differently, and the number and type of catalog(s) has continued to balloon. Without an agreement between all these vendors there isn’t much use for Egeria. Given that a company like Collibra has already solved the enterprise metadata problem there isn’t much need to outsource the solution to Egeria.

Ronald Schaareman

·4 years ago · Edited

At our company they use (?) Egeria, but for now it is far from finished. It doesn’t have proper UI, nor can it show horizontal data lineage (source to target table). It is open source, so you can go at it for free, but don’t forget that you need development team(s) to make it work and make it usable for business users. It is a bleak tool compared to Collibra, Alation and even OEMM / MIMM. Like the latter Egeria/OMDP is foremost a technically oriented tool.

Ronald Schaareman

·4 years ago · Edited

At our company they use (?) Egeria, but for now it is far from finished. It doesn’t have proper UI, nor can it show horizontal data lineage (source to target table). It is open source, so you can go at it for free, but don’t forget that you need development team(s) to make it work and make it usable for business users. It is a bleak tool compared to Collibra, Alation and even OEMM / MIMM. Like the latter Egeria/OMDP is foremost a technically oriented tool.

michaelkolb

OP4 years ago · Edited

As far as I know, ING is also one of the drivers/initiators of Egeria.

Ronald Schaareman

·4 years ago · Edited

That is correct.

Note that in the 2020 Gartner magic quadrant for metadata management solutions you can read that IBM claims “working within the Open Data Platform initiative (ODPi) to enable metadata exchange via open metadata standards (like Egeria)”.

An addition to my previous post: Egeria is working the other way around compared to most tools; it expects the metadata to be sent to the tool (tool centric) whereas most other tools (at least that I know of!) go out and capture (harvest) the data lineage from the business applications (application centric). Both approaches come with pros and cons.

arthurburkhardt

·4 years ago · Edited

Meh… Nice concept and idea, but I never found any substantial content to reuse.
For example, I was looking for guidance on how to define data domains in a Pharma company (there are many examples for Banking/insurance, less so for pharma). I was happy to find something on egeria, but it’s not very formalized/standardized, it’s more of an example. data-governance (odpi.org)

I’d love for this type of initiative to develop and expand, but it does not seem to get a lot of traction.