Anyone use Google Cloud BigQuery Bidirectional Integration from Marketplace?
Hello! We are starting to look at this Marketplace Integration running in Spring Boot. We have NOT set it up yet, but I do have a few questions for anyone that has installed and used it.
If you have tried or are using the integration, please let me know. I am curious as to how the tool handles edit collisions? Or is it run it BQ > Collibra or Collibra > BQ and it is all or nothing???
Thank you!
Brian Lloyd
Posted 3 years ago · Edited 2 years ago·Last reply 3 years ago
5 comments
stefan_busch
·3 years ago · EditedHello,
I did set it up once for testing. But i do not fully understand your question.
Where would the collision from your question appear? In Collibra or in BigQuery?
You can choose if you want to use the direction from BQ to Collibra, Collibra to BQ or both directions.
If you can specify your question a bit more I may be able to answer it.
stefan_busch
·3 years ago · EditedHi,
There are no rules that can be set regarding this kind of collisions. I did not try it, but the way in which the integration is working the first one to write should be the one who wins. So if BQ > Collibra is executed first then the BQ description would win, if Collibra > BQ is executed first then the Collibra description would win.
The reason for that is, that as far as I could tell from the code when I looked at it, the two directions do not communicate with each other.
Right now in the integration there is only one cron expression for both directions, but the direction Collibra > BQ is executed first (as long as they do not run in parallel this direction should win, but I did not test that).
What you can do, to be sure that Collibra > BQ wins, is to configure the integration so that only Collibra > BQ is executed regularly and that BQ > Collibra is started manuel (even though I don’t know if this is feasible for your use case).
stefan_busch
·3 years ago · EditedHello,
I did set it up once for testing. But i do not fully understand your question.
Where would the collision from your question appear? In Collibra or in BigQuery?
You can choose if you want to use the direction from BQ to Collibra, Collibra to BQ or both directions.
If you can specify your question a bit more I may be able to answer it.
Brian Lloyd
OP3 years ago · EditedCertainly - what I am unaware of is if we have Data Engineers setting metadata in Big Query and Data Stewards editing metadata in Collibra and what happens if we choose bi-directional sync? If Person A edits the description of a column inside of Big Query and Person B edits the same column’s description in Collibra - and the two systems are synchronized - which edit wins? Last edit wins? Or can you select Collibra as the winner for edits? or some other scenario??
I assume that is we choose Big Query to Collibra as the direction then any edits in Big Query overwrite the same attribute in Collibra. And vise versa… But for bi-directional sync what are the rules and can they be configured?
Thanks!!
Brian Lloyd
OP3 years ago · EditedThank you for the information! I suspected that the integration was operating like this, your note cleared things up.