Hello all, Wanted to check on how others are dealing with generating lineage in Collibra from databricks pipelines that are written using Spark Languages like Scala, Python. I see the lineage harvester has an out of the box connectivity for Spark SQL. Converting the pipeline code written in Python or Scala to Spark SQL files (not sure if possible, but I read in databricks documentation that there is Spark data frames concept behind the scenes when we execute an SQL query using Spark SQL), and fe...

Spark Languages Lineage | Product Q&A

Spark Languages Lineage | Product Q&A | Collibra Community