Refresh Conflict Notifications
Has anyone set up a workflow to notify the owners of a schema of the schema’s refresh conflicts?
ashleyodonnell
Posted 6 years ago · Edited 1 year ago·Last reply 4 years ago
18 comments
Has anyone set up a workflow to notify the owners of a schema of the schema’s refresh conflicts?
Former User
·5 years ago · EditedThis is definitely the most painful part of Collibra to date (1.5 years in now). I have several DB teams that are doing major updates and I get literally thousands of conflicts. The big problem is that I would have to drop the schema/data dictionary and reload it to clear it, but that will lose all of the manual additions, such as annotations and relations. Support suggested creating a workflow, but I am only one person doing DG in a 1400 person company so I don’t have the time. Can anyone actually help me? BTW, there is chatter in Collibra about a release that makes Conflicts a warning, and not a sync failure. It can’t come quickly enough.
arthurburkhardt
·5 years ago · Edited@slefferts.signifyhealth.com This should not be too hard to implement a workflow to clear refresh conflicts. Would you consider reaching out to collibra coaching for this?
I don’t have refresh conflicts to test on , but here’s how it might look like:
noor
·4 years ago · Edited@arthur.burkhardt, so just giving the domain/community as input, this piece of code should do the wonder right?..I got 183 refresh conflicts today in PROD env due to some changes in the source
Thinking for now, to delete the attribute manually or develop a workflow…
arthurburkhardt
·4 years ago · EditedHum, no. There is no method to filter on domain/community at the attribute level.
So either you can use the output module (recommended); or you can query attributes and assets separately and compute the intersection.
bokichena
·5 years ago · EditedBy deleting only “Refresh Conflict attributes” it is working but some of them are keep coming back. Will keep working with Support. Thanks for your help Tracy
tracywilliams
·5 years ago · EditedI have a workflow to clear out Refresh conflicts with the touch of a button.
bokichena
·5 years ago · EditedTracy, I guess you are kidding. Having table and column conflict issues for a while and an trying to resolve them gracefully. Would you mind sharing workflow it is real.
tracywilliams
·5 years ago · EditedActually I am not kidding
In our case, columns are never deleted from our data warehouse tables so we don’t get refresh conflicts on columns. We do get them on tables though, and after an application upgrade we tend to get a LOT of them - in the thousands. I scheduled a Collibra coaching session with Pranay to discuss how to handle them and he provided me with the workflow and helped me test it out. I would suggest talking to him - I don’t feel comfortable sharing his workflow directly.
bokichena
·5 years ago · EditedThanks Tracy. Yes, most of my conflicts, if not all of them, are on tables. I have opened a support ticket and they are still investigating on why it is happening on 80-90% of the tables even though nothing changed in the source. I will check with Pranay.
tracywilliams
·5 years ago · EditedI found that it happens when a table (or tables) are added and deleted from the Schema between refreshes. It can’t handle that case so it sets them all to Refresh Conflicts. I was manually deleting the deleted tables, then removing the Refresh Conflicts on the added tables, then refreshing. Now I just remove the refresh conflicts and rerun the ingestion.
bokichena
·5 years ago · EditedI understand if it happens when tables gets deleted. But, in my case source is well guarded production environment and no one delete tables. Also, my case gets complicated as we have mappings (relationships) between physical layer (columns) and business terms. I have to find out all the mappings, delete the tables and re-establish the mappings. Thanks for replying to my posts
tracywilliams
·5 years ago · EditedHave you tried deleting the Refresh Conflict attributes only and then refreshing the schema?
bokichena
·5 years ago · EditedMost of my assets have conflicts. I think I am better off deleting all of them and re-loading again
tracywilliams
·5 years ago · EditedI have a workflow to clear out Refresh conflicts with the touch of a button.
ashleyodonnell
OP5 years ago · EditedThat would be amazing!
georgecheung
·5 years ago · EditedRefresh Conflicts are stored as attributes on the Columns. Refresh Conflicts are added when a refresh of the ingestion happens and there are new or removed columns that were identified during the refresh.
It is possible to have a workflow run periodically (for example daily) that searches for ingested columns (in the OOTB Schemas Community) that have been last modified within the last day that has attributes for ‘Refresh Conflict’ and from there retrieve the Table and then the Schema.
Afterwards send out an email listing the columns that have Refresh Conflicts attributes.
edwintugade
·5 years ago · EditedI agree, this will be a great additional feature. Capturing schema refresh conflict is good but admins/stewards won’t know unless they check in Collibra. Notifications via out of the box workflow or similar will improve this process.
Tom Friesen
·6 years ago · EditedThis is something on our wish list, but we haven’t yet got around to implementing it. Interested in hearing from others on this one.