groovy-lib and workflow best practices
We develop quite a few workflows that serve a basic need: forms to document assets.
Those are quite repetitive tasks, with the same pattern:
- One workflow to create, one workflow to manage
- Load the characteristics values into memory
- Present the user with forms to update information
- Save the input data back to the database
Best practices for workflow development
What are the best ways to write code for loading and saving characteristics?
Here’s what I like, personally.
def attrs = attributeApi.findAttributes(
FindAttributesRequest.builder().assetId(dataset.id).build()
).results.collectEntries{[(uuid2String(it.type.id)): it.value]}
execution.setVariable("attrs", attrs)
execution.setVariable("location", attrs.get(locationTypeId))
execution.setVariable("description", attrs.get(descriptionTypeId))
execution.setVariable("statusId", uuid2String(dataset.status.id))
def attributes = [
[type: aliasTypeId, value: alias?.trim(), isMandatory: false],
[type: drugDevelopmentStageTypeId, value: utility.toList(drugDevelopmentStage), isMandatory: true],
[type: primaryUseCaseTypeId, value: primaryUseCase?.trim(), isMandatory: true],
[type: restrictionsTypeId, value: restrictions?.trim(), isMandatory: true],
[type: licenseTypeId, value: license?.trim(), isMandatory: true],
[type: natureOfDataTypeId, value: natureOfData?.trim(), isMandatory: true],
[type: sourceOfDataTypeId, value: sourceOfData?.trim(), isMandatory: false],
[type: sizeOfAssetTypeId, value: sizeOfAsset?.trim(), isMandatory: false],
[type: lastUpdateTypeId, value: lastUpdate? Date.getMillisOf(lastUpdate):null, isMandatory: false],
[type: refreshFrequencyTypeId, value: refreshFrequency?.trim(), isMandatory: false],
]
def missingAttributes = attributes.findAll{ it.isMandatory && [null,'',[]].contains(it.value)}
if (missingAttributes) {
def missingAttributeNames = missingAttributes.collect{attributeTypeApi.getAttributeType(string2Uuid(it.type)).name}.join(', ')
def dgcError = new WorkflowException("Mandatory attributes: ${missingAttributeNames}")
dgcError.setTitleMessage ("Missing Mandatory Attributes");
throw dgcError;
}
attributes.each{
assetApi.setAssetAttributes(SetAssetAttributesRequest.builder()
.assetId(dataset.id)
.typeId(string2Uuid(it.type))
.values(([null,'',[]].contains(it.value)?[]:[it.value]))
.build()
)
}
What do you think about it? How do you typically handle such tasks?
groovy-lib
the use of the OOTB APIs is quite cumbersome and multiple developers have requested to start using groovy-lib functions to avoid copy/pasting code all the time, and avoid inconsistencies and bugs.
Cons
- Risk on performance impact: https://developer.collibra.com/rest/slow-compiling-groovy-script/
- I disklike the idea of having 50 functions such as
addStringAttributeToAssetorgetSourceRelationsFromAssetwhich I believe would lead to too much complexity. - Requires restarting the environment every time (why can’t we use the
customizationApialready???)
Pros
- Better developer productivity
- Better code quality
- Some Collibra APIs have suboptimal implementations, such as the very useful “assetApi.setAssetAttributes” => Instead of comparing old and new values, then UPDATING existing values, it always deletes and create. This leads to poor diff in the activities (history audit).
Recently, I’ve been considering the creation of a WorkflowAsset class to manage very repetitive behavior that could work like:
scriptTask: init variables
scriptTask: save variables
if (!asset.attrs.location.value.contains("//")) {
asset.exception("Location must be a fully-qualified URI")
}
asset.saveAttributes(["description", "location"])
asset.rels.groupedByAsset.value = asset.rels.groupedByAsset.value.findAll{ it.id != asset.id }
asset.saveRelations(["groupedByAsset", "represents"])
And we could define the characteristics in the startForm just like this
Feedback?
So, what are your workflow best practices and what do you think could be improved?
Personally, I dream of the possibility to get rid of workflows entirely and be able to develop business rules directly into the collibra UI, so that we would not need workflow to document data sets and other assets.
akashgoel1
·4 years ago · EditedHi All,
The discussion here is really inspiring and I gained a lot of insight from here.
alvinuseree
·4 years ago · EditedSome sort of data citizen open source - like initiative is definitely something I’d get behind and contribute to!
Workflows, Dashboards, EmailTemplates and Integration Templates jump to mind .
I think a lot of this already exists (right?) but having it community driven really opens up additional opportunities for what can be shared and iterated on
**Edit: just saw the small print of what you wrote @ann.wuyts ! **
I think the beauty of community driven templates is less of an expectation to support and maintain what is socialised
Ann Wuyts
·4 years ago · EditedI’m taking the liberty to assume you don’t just want to make them more beautiful, but also more coherent, intuitive and user-friendly as a whole. On that note, I’d also ask for examples of why you’d want to include a table. For most use cases there must exist valid alternatives.
Opportunity-wise, there are options:
Timing-wise, I would say that for workflow templates these will be most valuable if they make use of all the new form capabilities in the Workflow Designer which will soon be in public beta. Similar for any updated dashboard templates. I would focus there on how dashboards can supplement the new Homepage, and take into account new navigation patterns.
For the email template, I suggest you support and add to
This idea was created by the Data Office (@shiroshana.tissera) based on customer feedback and our own experience when using Collibra internally. Branding & looks are included in this, but we definitly identified other areas to improve as well.
* I learned with the Dashboard Design Kit that publishing a collection of templates is one thing. Maintaining them is a larger challenge.
arthurburkhardt
OP4 years ago · EditedAny ideas how to make it work?
The marketplace seem way too rigid for efficient collaboration and datacitizen is too “free text” and temporal.
What platform would be best suited for collaborative curation?
Tom Friesen
·4 years ago · EditedThere could be a lot of benefit to the DCI… I’d definitely be interested in collaborating!
alvinuseree
·4 years ago · EditedAs always, a great post Arthur!
Here are some of my best practises:
Feature flags:
This has proven especially useful for OATs and piloting workflows; it’s essentially toggling certain features of your workflow for certain division as you roll out the workflow.
Groovy Linting:
I recommend this library for anyone that uses VSCode. It’s a very useful library that enforces some basic best practises for your Groovy code. Also useful to run a Groovy Linting check when deploying code.
Reusable CSS in Workflows:
I found myself adding loads of inline css to my workflow forms, much of which is duplicate css but maintained in various workflows. This didn’t feel right so I like to add a custom.css and upload it to the cloud and replace my workflow inline css with something more robust and maintained in one place.
Test Driven Development:
This is more a mindset and way of working but given the colleague facing nature of workflow development, I’ve found a TDD approach to work quite well. It enables good collaboration between Business and Development Teams; I’ve also found that it gives the Business a snippet into the art of the possible for workflows as well as the Development Teams more of a snippet around the context of why the Business have asked for a workflow.
The added bonus of mitigating the risk of a workflow being built that the Business haven’t quite asked for also helps!
Once again - great post
arthurburkhardt
OP4 years ago · EditedI love it!
I’ve always been terrible at making things look nice, and this is especially true when fighting against the collibra css.
@ann.wuyts Is there any opportunity to develop a library of templates to make workflows, email notifications and dashboards more beautiful?
I would say that one of the most useful would be how to display beautiful tables in workflow and emails. What do you think?
I’m already very happy with my default mailtask template. It’s simple and basic, but at least it’s not too ugly.
Tom Friesen
·4 years ago · EditedHey @arthur.burkhardt…
With regard to code reuse, I end up having a “library” of functions from which I copy and paste into scriptTasks as needed.
But I was wondering, based on what I’ve seen you mention in your posts, whether it might not be possible to do something like this:
There would still be a challenge if one were to re-import the workflows into Eclipse, but I could live with that. How difficult would the third bullet be? Worth it?
arthurburkhardt
OP4 years ago · EditedYes, I have a small python script that integrates with Eclipse and automatically deploys workflows for me.
There’s a small function in there that allows to automatically inject reusable functions into scripts
https://datacitizens.collibra.com/forum/t/workflow-variables-data-types/276/11?u=arthur.burkhardt
So in my code, I just need to write:
//INSERT src/main/java/lib/textarea.groovyand it appends the file.
Tom Friesen
·4 years ago · EditedHey all,
Thanks for sharing, @arthur.burkhardt !
I’ve been struggling with the redundancy in script code as well, but have only used the groovy-lib for workflow logging. That said, I like the generic-ness of your proposed WorkflowAsset class. That would be something I would include.
I noticed that you included “formType” as a form value for the characteristic. Are you somehow dynamically creating form fields in userTasks? The closest thing I’ve done in this regards would be to create a userTask with multiple fields/elements, which I then dynamically hide, depending on certain variables/conditions in the workflow.
But I do dislike manually entering information on startForms! So couldn’t one identify the characteristic ids and names through the assignmentApi.getAssignmentsForAssetType(), and even infer the formType based on resourceType/stringType (overriding it in the initial scriptTask where necessary)? If the minimumOccurrences doesn’t describe the “required-ness”, we could maybe simply the start form to have requiredFieldNames (eg. “description,location”)?
Workflows: Smaller, Abstracted, and Event-Driven
One of the other things I’ve been doing is breaking up the workflows that manage the asset lifecycle based on common function, status, and/or location (for those assets that move around), and having them triggered through events. This not only simplifies workflows, but it reduces the chances of having to kill and restart a whole bunch of workflows/tasks when something goes wrong, and if it does, it’s just way easier to manage.
An example of this would be our Policy Management lifecycle. It includes numerous workflows: some policy-specific, and some generic subprocesses:
Anyone else have some best practices to share?
arthurburkhardt
OP4 years ago · EditedThanks for the feedback, @tom.friesen!
No, I thought about a “workflow generator” workflow that would enable the user to select attribute type and generate the xml code (using most probably form property types), but I didn’t pursue this.
Right now, I’m using python and copy/paste into the bpmn directly.
arthurburkhardt
OP4 years ago · EditedSharing one more tip here: I used the python code to generate all the possible values in one go.
with open("FormProperties.txt", "w+", encoding='utf-8') as f: for k, v in {'assetTypes':'AssetType', 'attributeTypes':'Type', 'communities':'Community', 'complexRelationTypes':'CRType', 'dataQualityRules':'DQRule', 'domainTypes':'DomainType', 'domains':'Domain', 'relationTypes':'RelationType', 'roles':'Role', 'statuses':'Status', 'userGroups': 'UserGroup', 'workflowDefinitions':'Workflow'}.items(): for kk, vv in c._cache[k].names.items(): var_name = tokenize(vv) + v + "Id" f.write(f"""<activiti:formProperty id="{var_name}" name="{var_name}" type="string" default="{kk}" """ f"""readable="false" required="true"></activiti:formProperty>\n""")This created a 3000 line long file that looks like:
<activiti:formProperty id="acronymAssetTypeId" name="acronymAssetTypeId" type="string" default="00000000-0000-0000-0000-000000011003" readable="false" required="true"></activiti:formProperty> <activiti:formProperty id="analyticsProjectAssetTypeId" name="analyticsProjectAssetTypeId" type="string" default="8214f868-9600-487a-a2fd-c071c7946aab" readable="false" required="true"></activiti:formProperty> <activiti:formProperty id="assessmentReviewAssetTypeId" name="assessmentReviewAssetTypeId" type="string" default="00000000-0000-0000-0000-000000031305" readable="false" required="true"></activiti:formProperty> ....So now, when I need to add new default properties in the start event, I can just open the .bpmn with the text editor, ctrl + f in my FormProperties.txt file and copy paste the values over. So far it’s proven quite efficient.