Spark Languages Lineage

Hello all,

Wanted to check on how others are dealing with generating lineage in Collibra from databricks pipelines that are written using Spark Languages like Scala, Python.

I see the lineage harvester has an out of the box connectivity for Spark SQL. Converting the pipeline code written in Python or Scala to Spark SQL files (not sure if possible, but I read in databricks documentation that there is Spark data frames concept behind the scenes when we execute an SQL query using Spark SQL), and feeding those files to the lineage harvester could work?

Appreciate the mindshare…

noor

Posted 4 years ago · Edited 1 year ago·Last reply 4 years ago

1 comment

arthurburkhardt

·4 years ago · Edited

No, not possible to convert pyspark/spark (scala) code into Spark SQL. Those are different constructs and with different capabilities.
Technical lineage on spark is very difficult to do. Maybe it would be more efficient to do lineage at script level rather than at data level.