21 Messages
DBT to Collibra Integration(v1.2.2)
Hello @spring-team.collibra.com,
I have update the latest version(v1.2.2) of spring boot application. After setting up the values for properties file and updating other dependencies, we executed the code but it failed with below error message.
e[2m2022-12-27T17:55:44.963-05:00e[0;39m e[32m INFOe[0;39m e[35m8900e[0;39m e[2m—e[0;39m e[2m[0.0-8181-exec-1]e[0;39m e[36mc.c.dbt.component.DBTLineageTransformer e[0;39m e[2m:e[0;39m Transforms DBT tests performing
e[2m2022-12-27T17:55:44.973-05:00e[0;39m e[32m INFOe[0;39m e[35m8900e[0;39m e[2m—e[0;39m e[2m[0.0-8181-exec-1]e[0;39m e[36mc.c.dbt.component.DBTLineageTransformer e[0;39m e[2m:e[0;39m Test assets transformer operation has been completed
e[2m2022-12-27T17:55:49.517-05:00e[0;39m e[31mERRORe[0;39m e[35m8900e[0;39m e[2m—e[0;39m e[2m[0.0-8181-exec-1]e[0;39m e[36mc.c.d.exception.GlobalExceptionHandler e[0;39m e[2m:e[0;39m Null pointer exception has been occurred
java.lang.NullPointerException: Cannot invoke “java.util.UUID.toString()” because the return value of “com.collibra.dbt.component.CollibraOperations.findDomain(String)” is null
Please let me know if I need to provide domain UUID in configuration file?
Thanks
Sumit
springboot_team
368 Messages
2 years ago
Hi @SumitTiwari,
Please check that you have migrated all the properties to the new version, as the error seems to be coming from a domain ID set to null. Kindly follow the documentation to ensure that all properties are set correctly.
Thanks
1
0
sumittiwari
21 Messages
2 years ago
DBTLineageTransformer .txt (23.1 KB)
Hi @spring-team.collibra.com,
Below are few updates needed to the code:
With further investigation, we found that the issue was with domain name created by this application. The DBName (SystemName > DBName > Schema) in domain was in lower case and at the time of assets insertion, the application was looking for domain name with DBName in UpperCase.
There were few other modification required where DBName, Schema and Table Names are getting created in lower case. Although its not throwing an error but to be consistent with EDGE server imports, I believe everything should be in Upcase.
In this section, there are few attributes getting populated for exposures i.e .addAttributeValue(CollibraConstants.AttributeType.URL, exposures.getUrl()). There is no validation applied in case of null or empty value. @ralph.rimorin.collibra.com helped me fixed that and applied same validation for other attributes too.
Attaching the DBTLineageTransformer file with updated code. Please review it and update the main code.
As per conversation with @antonio.castelo, this package should not create any duplicate assets. The configuration should consider EDGE and only upsert the assets to avoid any duplicate entry. Unfortunately this new version is creating duplicate assets and creating new domains for each schema. This was not the case with earlier version.
Thanks
Sumit
@ralph.rimorin.collibra.com @antonio.castelo @aaron.parry
1
0