Start Workflow when Asset Removed
Has anyone managed to get a workflow to start when an asset is removed? There is a workflow start event of “Asset Removed”, but that doesn’t seem to be working.
I created a workflow set to start when a Column asset is added to or removed from a specific community. The workflow does run when a column asset is added. It doesn’t run when the very same column is deleted. I have tried deleting via the Collibra website and the REST API. Neither triggers the workflow. My Collibra version is 5.7.3.
Example workflow configuration:
sameary
Posted 5 years ago · Edited 1 year ago·Last reply 5 years ago
27 comments
rostislav
·5 years ago · EditedI’m happy to let you know that as of the 2021.07 release, our engineers have fixed the bug preventing the Asset Removed start event from working as expected.
https://productresources.collibra.com/docs/collibra/2021.07/Content/ReleaseNotes/to_latest-release-cdic-cloud.htm#Fixes202107
noor
·5 years ago · Edited@rostislav.andriciuc.collibra.com
Will this work if the catalog schema refresh ingestion process has identified a delete on source system side & therefore deleted the asset in the catalog?
marieaudemagarshack
·5 years ago · EditedHello, from our end, the main issue with asset removal is the deletion of all the relations that the removed asset was a part of. We need a warning about which other assets lost a relation. We have not investigated possible solutions, but this kind of change on existing assets could be used as a triggering event, correct?
arthurburkhardt
·5 years ago · EditedI think so, but that would trigger every time you remove a relation, which might not be optimal.
marieaudemagarshack
·5 years ago · EditedHello @arthur.burkhardt, yes you are right, thank you for the heads-up. We do not have a large volume of assets yet in the DGC, but this is worth keeping in mind for the future.
Best regards
Former User
·5 years ago · EditedNice one, looks like a decent amount has change since we last looked into this.
In all honesty, looking back on this retrospectively, the workflow we built and gave our users probably wasn’t the best approach to addressing the issue we had. The workflow is still in production for a select few but we found that disabling user’s permissions to delete and creating recycling bin domains our users could move their obsolete assets to worked a whole lot better.
Former User
·5 years ago · EditedAh right, so for my use case I also needed to find out additional information on top of the deletion. Maybe this will help you with your use case.
As the asset was deleted there was no business item for the asset to go off of but you can find out certain things about the asset using the activityStream API endpoint.
The last deletion activity by the user in our use case was always the asset being deleted (thus triggering the workflow). You’d also have a jobId for the deletion which you could query further to get more insights into how the asset got deleted.
You could really go to town on the activity API to reverse engineer loads of information on the asset being deleted. Biggest issue we faced is the performance of that API at the time was very shocking. Perhaps things have changed since then?
arthurburkhardt
·5 years ago · EditedHum… Yeah… That does not cut it for me, I really needed to prevent deletion.
Both times when I had to restore assets, I ended up taking a backup and going directly into the postgres database to get the info.
asset = pd.read_sql(f""" select * from representations r where id = '{asset_id}' """ , con) attributes = pd.read_sql(f""" select * from "attributes" a where "owner" = '{asset_id}' """, con) rels = pd.read_sql(f"""select * from relations r where "source" = '{asset_id}' or target = '{asset_id}'""", con) comments = pd.read_sql(f"""select * from "comments" c where base_resource_rid = '{asset_id}'""", con) responsibilities = pd.read_sql(f"""select * from members where associated_resource_id = '{asset_id}'""", con)For your information, the activities API performs great, no issues with it, as long as you use the right filters. You can already experience this by looking the history section in the asset page or in the admin panel. Filtering by assetId or by startdate returns very quickly…
start = time.time() #get last 24h of deletion r = c.get_resource('activities', params={'activityType': 'DELETE', 'startDate':int(time.time())*1000-(1*86400000)}) print("Execution took {:.2f}".format(time.time()-start)) 2021-05-27 07:57:37,665 - INFO - Retrieved 56 activities Execution took 0.13arthurburkhardt
·5 years ago · EditedAhaha, thanks @sam.eary and @alvin.useree.lloydsbanking.com, I appreciate it!
I don’t know what I did wrong the first time around, but it’s working great now: the workflow is triggered when an asset is deleted and throwing a WorkflowException prevents the asset from being deleted, but the error message is not displayed to the user
Ultimately, I intend to add a little more logic, for example to be able to check for specific assets/asset types, users, etc.
noor
·5 years ago · Edited@arthur.burkhardt
Can we apply this for asset deletion by the catalog schema refresh process via the OOTB drivers? I guess the refresh process would fail. In my case, I do not have concerns that users would delete assets but I need to prevent deletion by the refresh process, and mark the asset status as “obsolete”.
Br,
Noor.
arthurburkhardt
·5 years ago · EditedI don’t have sufficient experience with the OOTB drivers, but that’s not how they work, do they? They don’t delete assets I think, but identify a schema conflict.
noor
·5 years ago · Edited@arthur.burkhardt
Here is how they work:
noor
·5 years ago · Edited@arthur.burkhardt
I created a workflow with the code snippet shared by you, and here are my inferences:
“An unexpected error occurred while importing the ingested assets from data source …”
I unassigned the workflow from the Physical Data Dictionary domain type, and triggered the schema refresh once again. It went through. So, conclusion is the OOTB refresh process fails as expected.
noor
·5 years ago · Edited2021-08-20 08:01:50.474 [JobServer Procedure Manager] WARN c.c.c.r.c.s.m.i.DataSetMatcherEventListener - Listener invoked with no events to process [ehcache-txid=27625319]
2021-08-20 08:01:50.588 [JobServer Procedure Manager] INFO c.c.d.c.w.i.WorkflowEventServiceImpl - Starting workflow ‘Prevent Asset Deletion’ for business item id ‘6c761ec0-29e6-47a2-abc4-48d0d191344a’ and start event ‘TERM_REMOVED’ [ehcache-txid=27625319]
2021-08-20 08:01:50.607 [JobServer Procedure Manager] INFO c.c.d.c.w.a.e.DGCScriptTaskActivityBehavior - Starting execution of workflow script task ‘Script Task’ [ehcache-txid=27625319]
2021-08-20 08:01:50.620 [JobServer Procedure Manager] INFO c.c.d.c.api.component.LoggerApiImpl - Workflow triggering inside condition [ehcache-txid=27625319, trace_id=df77e3c28de72e5629f9e0e0c70994bc, trace_flags=01, span_id=771d48ad7f0a7f7e]
2021-08-20 08:01:50.620 [JobServer Procedure Manager] WARN o.f.e.i.b.b.ScriptTaskActivityBehavior - Exception while executing scripttask1 : problem evaluating script: javax.script.ScriptException: com.collibra.dgc.workflow.api.exception.WorkflowException: Exception description. [ehcache-txid=27625319]
2021-08-20 08:01:50.621 [JobServer Procedure Manager] ERROR c.c.d.c.w.a.WorkflowExceptionHandler - Error starting workflow with id: PreventAssetDeletion [ehcache-txid=27625319]
org.flowable.common.engine.api.FlowableException: problem evaluating script: javax.script.ScriptException: com.collibra.dgc.workflow.api.exception.WorkflowException: Exception description.
georgecheung
·5 years ago · EditedIt is possible to have a workflow start upon the Event of Asset Removed.
It depends on how you have the workflow configured and how you delete the asset.
If you have the workflow set up for the Asset level (Applies To: Asset) then it would obviously not work as a workflow requires a resource (in this case an asset) to run on. So it is not possible to delete the asset and have it run on that same deleted asset.
If you have the workflow set up for the Domain level (Applies To: Domain) then it is possible because the domain would be the resource that the workflow is running on. And in order to trigger this workflow to run - you will need to delete the asset from the asset page, and NOT from the domain table view of assets.
You can also set the workflow up such that it Applies to: Global, and when the asset is deleted from the Asset page, it will trigger the workflow on the domain that the asset was on.
sameary
OP5 years ago · EditedI did get a workflow to start on Asset Removal with the workflow set at domain level. I was unable to get it to start at the global and community levels. Why is there such a specific way to start the deletion? Since a deletion can originate from the domain’s asset table, that deletion should be able to start a workflow. It seems like there is a bug preventing this feature from working correctly.
Is there any option in which the same workflow would start upon asset add and asset removal? When the workflow applies to asset or globally, it will start when an asset is added, but not when removed. When it applies to domain, I’m only getting the removal start condition to work. " Applies to Community" isn’t doing anything.
arthurburkhardt
·5 years ago · EditedI’m starting to have the need for such a trigger, and finding this thread is quite depressing…
I won’t even open a support ticket, just to be answered “This is not a bug, this is a feature. You can open an idea on the ideation board” (aka the trash can). (￢︿̫̿￢☆)
@sam.eary I could not replicate your findings on workflow at domain level, can you please explain how it works?
This is my config
But I can still remove assets from a “Process Register” domain type in this community without triggering the workflow.
There is one way to prevent asset deletion with a workflow: just add syntax errors in your script. Magically, it will throw an error and leave the asset intact.(づ￣ 3￣)づ
But well, it’s not really useful, because it’s still easier to just remove the delete permission from all roles.
sameary
OP5 years ago · Edited@arthur.burkhardt That configuration is pretty much what I did. I had the start event be Asset Removed. I did some quick testing with my original workflow attempt on version 2021.05.1-17 it seems to be triggering upon deletion from both the Asset list page in a domain and the delete button on the Asset page. However, I never did find a way to prevent the deletion and convert the asset to obsolete as originally intended. Also, I’m not sure what Sync API, Import API or Core API initiated deletions would do.
Here is my configuration:
Here is the actual workflow:
Former User
·5 years ago · EditedI had a similar use case about a year back, I imagine this still work as we still have this workflow in production and haven’t had any of our users raise a bug on it.
If you throw a workflow error using the workflowException method it should stop the deletion mid flight.
Ronald Schaareman
·5 years ago · EditedHi @chuck.grissom,
To be frank, no. Although it has been one of the starting events of a workflow once (still in version 3 or 4), but was never triggered. I quickly tried something on version 5 and indeed this does not trigger anything.
This seems like a defect to me. Did you log a ticket with support for it?
Regards, Ronald
sameary
OP5 years ago · EditedI just created a support ticket and will post back if support has any insights.
Ronald Schaareman
·5 years ago · EditedHi @chuck.grissom,
To be frank, no. Although it has been one of the starting events of a workflow once (still in version 3 or 4), but was never triggered. I quickly tried something on version 5 and indeed this does not trigger anything.
This seems like a defect to me. Did you log a ticket with support for it?
Regards, Ronald
rostislav
·5 years ago · Edited@sam.eary, thank you for your patience and for the support ticket . I will discuss this with the workflows product manager.
Ronald Schaareman
·5 years ago · EditedIn answer to @chuck.grissom I would like to bring forward the use case of keeping multiple models in sync. If you use Collibra to govern changes to any type of model (be it a list of business terms, a data model) you would surely want to trigger an event on all changes, also when an asset is deleted.
In the area of governance you might also want to notify the business steward if any (unwanted?) change occurs in one of this person’s domains.
Kind regards,
Ronald
chuckgrissom
·5 years ago · EditedHI Ronald,
Thanks for providing that use case. Are you able to trigger a workflow or notification when as asset is deleted? In reading Rostislav’s response, I got the impression that it was not possible because the asset was deleted.
Thanks.
Chuck
chuckgrissom
·5 years ago · EditedHi Rostislav,
We were trying to figure out the same thing. Since "“Asset Removed” is listed under possible start events, what is an example where deleting an asset does start a workflow?
Thanks.
Chuck
rostislav
·5 years ago · EditedHi Sam,
Starting a workflow when an asset is removed is not possible because the trigger removes the asset the workflow is supposed to run for.
Kind regards,
Rostislav