Experience with onboarding bulk data requests into a business glossary

HI,

Interested in understanding experience in providing the business an option to bulk upload a high volume of business assets in one action. At the moment we have utilised the OOTB workflows for adding new BT but this has only been configured for singular BTs.

The requirement for bulk upload mainly centres on,

The data been available for approval/review/rejection etc.
We have determined a number of mandatory attributes that we would need populated as part of this process

We currently utilise the bulk import however this does not allow either of the above requirements. Also we have restricted this function to admin users.

Ideally we would like to develop a workflow that takes an CSV/Excel file similar to the import function and within that workflow check that we have the required attributes and be queued for review in the data flow.

Interested in understanding if any citizens have developed a similar workflow or perhaps went with a different approach to achieve this in Collibra.

Kind Regards
Stephen

stephenmc_donald

Posted 4 years ago · Edited 1 year ago·Last reply 3 years ago

5 comments

arthurburkhardt

·3 years ago · Edited

sure

Code	Description
1	One
2	Two
3	Three

chumitalukdar1

·3 years ago · Edited

@arthur.burkhardt: can you please provide the reference of the excel file used in bulk upload of the assets

stephenmc_donald

OP4 years ago · Edited

This is very helpful - thank you.

arthurburkhardt

·4 years ago · Edited

FYI, here’s a sample code to batch import data from an excel file in a workflow. (it’s for reference data: code set/code value).
From there, you can customize to add any validation constraints, start other approval workflows, etc.

importCodeSets.groovy

import org.apache.poi.ss.usermodel.DataFormatter
import org.apache.poi.xssf.usermodel.XSSFWorkbook
import com.fasterxml.jackson.databind.ObjectMapper
import com.collibra.dgc.core.api.dto.file.AddFileRequest
import com.collibra.dgc.core.api.dto.instance.asset.AddAssetRequest
import com.collibra.dgc.core.api.dto.instance.asset.SetAssetAttributesRequest
import com.collibra.dgc.core.api.dto.instance.domain.AddDomainRequest
import com.collibra.dgc.importer.api.request.ImportJsonInJobRequest
import com.collibra.dgc.importer.api.request.ImportJsonInJobRequest.ImportJsonInJobRequestBuilder
import com.collibra.dgc.workflow.api.exception.WorkflowException


def domain = domainApi.addDomain(AddDomainRequest.builder()
	.communityId(string2Uuid(communityId))
	.description(codelistDescription.toString().trim())
	.typeId(string2Uuid(codelistDomainTypeId))
	.name(codelistName)
	.build()
	)
	
def codeset = assetApi.addAsset(AddAssetRequest.builder()
	.domainId(domain.id)
	.name(codelistName)
	.typeId(string2Uuid(codeSetTypeId))
	.build()
	)
	
def attributes = [[typeId: descriptionTypeId, value: codelistDescription.toString().trim()]]
  .findAll{it.value}.collect { 
	assetApi.setAssetAttributes(SetAssetAttributesRequest.builder()
		.assetId(codeset.id)
		.typeId(string2Uuid(it.typeId))
		.values([it.value])
		.build()
		)
}

if(!fileApi.getFileInfo(string2Uuid(importFile.toString())).getName().endsWith('.xlsx')){
	def dgcError = new WorkflowException("File Type Not Recognized, please use .xlsx")
	dgcError.setTitleMessage("File Type Not Recognized");
	throw dgcError;
}

workbook = new XSSFWorkbook(fileApi.getFileAsStream(string2Uuid(importFile)))
def sheet = workbook.getSheetAt(0)
if(sheet.getPhysicalNumberOfRows() < 2) {
	def dgcError = new WorkflowException("No records could be found in the excel file provided")
	dgcError.setTitleMessage ("No record");
	throw dgcError;
}
def rowIter = sheet.iterator()
rowIter.next() // skip the header
DataFormatter formatter = new DataFormatter()

def payload = rowIter.collect{ row ->
	
	if(row.getCell(0) == null || row.getCell(1) == null){
		def dgcError = new WorkflowException("Incorrect record in the file: value cannot be null")
		dgcError.setTitleMessage ("Incorrect record");
		throw dgcError;
	}
	
	String codeValueName = formatter.formatCellValue(row.getCell(0)).trim()
	String codeValueDescription = formatter.formatCellValue(row.getCell(1)).trim()
	
	[resourceType:"Asset",
	 identifier:[name:codeValueName, domain: [id: uuid2String(domain.id)]],
	 name: codeValueName,
	 type: [name: "Code Value"],
	 attributes: [Description: [[value: codeValueDescription]]],
	 relations: ["00000000-0000-0000-0000-000000007041:TARGET": [[id: uuid2String(codeset.id)]]]
	]
}

def mapper = new ObjectMapper()
String pl = mapper.writeValueAsString(payload)
loggerApi.info("Importing payload: " + pl)

File tempFile = File.createTempFile("NewCodelist", ".json")
tempFile.write(pl);

importerApi.importJsonInJob(ImportJsonInJobRequest.builder()
	.fileName(tempFile.name)
	.file(tempFile)
	.build()
	)

arthurburkhardt

·4 years ago · Edited

Yes, it’s possible to build a workflow to do exactly what you want.
Another possibility is to load the data into a default community with a default status, then start approval workflows for all items + validation rules to check for mandatory attributes.