Collibra Data Lineage

In order to fully access INFORMATION_SCHEMA it is required by Snowflake to use elevated user permissions (admin). Please elevate the permissions for the user which you have specified in Lineage Harvester for Snowflake data source.

If you cannot elevate the permissions you might use to the admin user to create a view on top of INFORMATION_SCHEMA and grant access to that other non-elevated user. This should overcome the permissions issues. In such case you have to follow up and modify also SQL statements in Lineage Harvester /sql/snowflake/ so it is reading from that new view instead.

Hi Martin,

Can you please guide on all necessary pre-requisites for snowflake metadata ingestion? The below article talks about running snowcd command from the jobserver and whitelisting the snowflake hosts.

https://support.collibra.com/hc/en-us/articles/360014046437-Snowflake-connectivity-issue

Do we need to do this always? Do we need to whitelist only the main end point type :DEPLOYMENT" or other endpoints of type “STAGE”?

also, any certificated need to be imported into the jobserver?

Br,
Noor.

Hi Noor, I recommend you reach out to Collibra support with this question.

Hi @karanpreet.singh , lineage harvester officially supports only user/pass authentication. However since LH is using JDBC driver from Microsoft you can try to make it work by passing custom properties. This is possible by modifying lineage-harvester.conf as outlined bellow. However I cannot offer further guidance on this.

“customConnectionProperties” : “instanceName=my-instance;databaseName=my-database;integratedSecurity=true;domain=my-domain;authenticationscheme=NTLM;user=my-user;password=my-password”

We plan to add support for NTLM later this year once Lineage Harvester is ported to a new component called Edge.

Hi Noor. You can create technical lineage manually by creating a custom JSON file. Instructions to do so are in our documentation here:

https://productresources.collibra.com/docs/cloud-user/latest/#CollibraDataLineage/TechnicalLineage/InstallationAndConfiguration/ta_prepare-a-json-file-for-technical-lineage.htm

Hey mate,

I think you’re referring to a traceability diagram here and not tech lineage (slight differences in product offerings).

Answering your question quickly, you can add the relation to the Data Model as a Boxed/Boxing Node and there’s an option to set the Node as collapsed by default (see below).

Shout if you have any issues

Thanks, Alvin,

Certainly helped clear up the ‘mess’. Still not quite right for what my management want to see but I’m new to Collibra so will do some more digging.

Assuming we are talking about importing transformation details into Collibra Data Lineage then this has to be done via JSON file and ingestion via Lineage Harvester and not via Catalog APIs.

We have been struggling with the same issues and I’m curious if this remains unresolved for you

Harvester is unable to complete the master batch update and can run for 24 hours or more before we must manually kill the process. Support for Data Lineage has been anemic for the past 1.5 months which makes it incredibly challenging to successfully deploy this product. Our efforts have been full-stopped because we can no longer use Harvester to update the TechLin servers.

Since this morning, we have begun receiving the ‘410 Gone’ response code that terminates the master batch process. Is there anyone from Collibra paying attention to this thread that can address the issues of two customers who are having the exact same problem?

@martin.masarik.collibra.com
@paulo.taylor

My open case #72103

Hi Peyton,

first of all I am sorry to hear about you having troubles with Lineage harvester. We have indeed worked with Adobe and resolved the reported issue. I had a quick peak at your ticket and it is not related to the same problem.

I am not sure whether you raised more Support tickets but the one you linked was reported 6 working days ago. Engineering already provided partial fix earlier today which indeed you reported back is not solving the issue in full. I am going to work with the team to understand the issue in full and provide an update on this before end of this week via Support ticket.

Martin

@kristen.freer please reach out to Chandra internally and point him towards that post. Thank you.

Hi @noor.shaik , supported are SparkSQL statements. You can upload plain SQL statements as files via Lineage Harvester or use JDBC driver which is shipped with Lineage Harvester. That driver allows connection only to AWS host in regard to SparkSQL.

@martin.masarik.collibra.com
May I know what is expected in those SparkSQL statements/SQL files?

Currently, I have seen a colleague who feeds a Snowflake .SQL file to Lineage Harvester. The SQL file contains a SELECT query on INFORMATION SCHEMA that gets the view names & their source code (just 2 columns in select clause). I believe lineage harvester uses the OUTPUT of this SQL to link views with underlying tables (?). The same Snowflake schema has a stored procedure that moves data to another schema. I guess then the link of data movement between these 2 schemas cannot be established as lineage harvester cannot parse yet stored procedures yet. So, the .SQL file in this case then can contain links/middle man (stored procedure) information so lineage is established between the tables in these 2 schemas?

Hi @balasubrahmanyam.estamsetty, unfortunately this is not something I can help with. I recommend you raise a Support ticket for this and see if you get can help with this custom properties option. Thank you.