Integration Library Import Errors

The latest Collibra Integration Library 1.1.9 doesn’t seem to work with the latest version of Collibra DIC. My guess there has been a model change with Collibra and the import. When using its import with the Open API Swagger .
To fix this my guess is the Collibra Integration Library should upgraded to the latest version of the Collibra Rest spec.
Marketplace asset DGC returns the following:

See DGC import error:

2022-10-28 01:57:48.415 [pool-20-thread-1] INFO c.c.c.d.i.s.DatabaseIngestionControllerService - The periodical lifecycle check is completed after PT0.000983S [session_hash=11b42c1875b4fbf96a87526f95732aa3, authenticated_id=00000000-0000-0000-0000-000000900001, trace_id=44fd687688d629e273b00fd40da96f61, trace_flags=01, span_id=64c7a6f6a49c4f60]
2022-10-28 01:57:51.352 [http-nio-0.0.0.0-4400-exec-13] INFO c.c.d.c.s.i.i.OutputViewServiceImpl - TableViewConfig: {

“TableViewConfig”: {*
```
   "displayLength": -1,*
```
```
   "displayStart": 0,*
```
```
   "Resources": {*
```
```
       "Term": {*
```
```
  		"Signifier": {*
```
```
  			"name": "assetName"*
```
```
  		},*
```
```
  		"Vocabulary": {*
```
```
  			"Name": {*
```
```
  				"name": "domainName"*
```
```
  			},*
```
```
  			"Community": {*
```
```
  				"Name": {*
```
```
  					"name": "communityName"*
```
```
  				}*
```
```
  			}*
```
```
  		},*
```
```
           "ConceptType": {*
```
```
               "Id": {*
```

                   "name": "assetTypeId"*

```
               },*
```
```
               "Signifier": {*
```

                   "name": "assetTypeName"*

```
               }*
```
```
           },*
```
```
           "Filter": {*
```
```
               "AND": [*
```
```
                   {*
```
```
                       "Field": {*
```

                           "name": "assetName",*

                           "operator": "IN",*

                           "values": ["Canary REST API","Canary REST API>GET/browseNodes","Canary REST API>GET/browseNodes>200","Canary REST API>POST/getTagData","Canary REST API>POST/getTagData>200","Canary REST API>GET/getUserToken","Canary REST API>GET/getUserToken>200","Canary REST API>GET/getAggregates","Canary REST API>GET/getAggregates>200","Canary REST API>POST/browseTags","Canary REST API>POST/browseTags>200"]*

```
                       }*
```
```
                   },*
```
```
                   {*
```
```
                       "Field": {*
```

                           "name": "domainName",*

                           "operator": "IN",*

                           "values": ["API Assets","Data Assets","Code Values"]*

```
                       }*
```
```
                   },*
```
```
                   {*
```
```
                       "Field": {*
```

                           "name": "communityName",*

                           "operator": "IN",*

                           "values": ["Open Api Spec"]*

```
                       }*
```
```
                   }*
```
```
               ]*
```
```
           }*
```
```
       }*
```
```
   },*
```
```
   "Columns": [*
```
```
       {*
```
```
           "Column": {*
```

               "fieldName": "assetName",*

```
               "index": 0*
```
```
           }*
```
```
       },*
```
```
       {*
```
```
           "Column": {*
```

               "fieldName": "assetTypeId",*

```
               "index": 1*
```
```
           }*
```
```
       },*
```
```
       {*
```
```
           "Column": {*
```

               "fieldName": "assetTypeName",*

```
               "index": 2*
```
```
           }*
```
```
       },*
```
```
       {*
```
```
           "Column": {*
```

               "fieldName": "domainName",*

```
               "index": 3*
```
```
           }*
```
```
       },*
```
```
       {*
```
```
           "Column": {*
```

               "fieldName": "communityName",*

```
               "index": 4*
```
```
           }*
```
```
       }*
```
```
   ]*
```
}*
} [authenticated_id=bd1704ee-625f-4def-a9ae-c0400e22e0ef, session_hash=11d050c92a09cc0b68f05e6c1ec1eca6, ehcache-txid=10791456, trace_id=506b1503a7bffbfe7cee878c88676c45, trace_flags=01, span_id=a0341f8939cdb19a]
2022-10-28 01:57:51.585 [jobExecutor-5] INFO c.c.d.c.s.job.impl.JobServiceImpl - Job ActiveJob{id=‘1e0f73fa-61c9-469f-b629-4019f931e367’, name=‘Export to “exportd206635faef8433f9c500c6c3b6fb208.csv”.’, user=‘bd1704ee-625f-4def-a9ae-c0400e22e0ef’, state=WAITING} started at 2022-10-28T01:57:51.585562Z [session_hash=ed4a004220b3685e16e1fed5a4517c3a, authenticated_id=bd1704ee-625f-4def-a9ae-c0400e22e0ef, trace_id=08b2e1fd778b1ea729ee8f7248a308a1, trace_flags=01, span_id=21128e975839bf6e]
2022-10-28 01:57:51.591 [jobExecutor-5] INFO c.c.d.c.s.job.impl.JobServiceImpl - Preprocessing of job ActiveJob{id=‘1e0f73fa-61c9-469f-b629-4019f931e367’, name=‘Export to “exportd206635faef8433f9c500c6c3b6fb208.csv”.’, user=‘bd1704ee-625f-4def-a9ae-c0400e22e0ef’, state=RUNNING} done. [session_hash=ed4a004220b3685e16e1fed5a4517c3a, authenticated_id=bd1704ee-625f-4def-a9ae-c0400e22e0ef, trace_id=08b2e1fd778b1ea729ee8f7248a308a1, trace_flags=01, span_id=21128e975839bf6e]
2022-10-28 01:57:51.592 [jobExecutor-5] INFO c.c.d.c.s.job.impl.JobServiceImpl - Transaction started for job ActiveJob{id=‘1e0f73fa-61c9-469f-b629-4019f931e367’, name=‘Export to “exportd206635faef8433f9c500c6c3b6fb208.csv”.’, user=‘bd1704ee-625f-4def-a9ae-c0400e22e0ef’, state=RUNNING}. [session_hash=ed4a004220b3685e16e1fed5a4517c3a, authenticated_id=bd1704ee-625f-4def-a9ae-c0400e22e0ef, ehcache-txid=10791462, trace_id=08b2e1fd778b1ea729ee8f7248a308a1, trace_flags=01, span_id=21128e975839bf6e]
2022-10-28 01:57:51.592 [jobExecutor-5] INFO c.c.d.c.io.job.OutputViewJobRunner - Parsing configuration [session_hash=ed4a004220b3685e16e1fed5a4517c3a, authenticated_id=bd1704ee-625f-4def-a9ae-c0400e22e0ef, ehcache-txid=10791462, trace_id=08b2e1fd778b1ea729ee8f7248a308a1, trace_flags=01, span_id=21128e975839bf6e]
2022-10-28 01:57:51.594 [jobExecutor-5] INFO c.c.d.c.io.job.OutputViewJobRunner - Parsing configuration took: 2ms [session_hash=ed4a004220b3685e16e1fed5a4517c3a, authenticated_id=bd1704ee-625f-4def-a9ae-c0400e22e0ef, ehcache-txid=10791462, trace_id=08b2e1fd778b1ea729ee8f7248a308a1, trace_flags=01, span_id=21128e975839bf6e]
2022-10-28 01:57:51.595 [jobExecutor-5] INFO c.c.d.c.io.job.OutputViewJobRunner - Executing viewconfig and querying data [session_hash=ed4a004220b3685e16e1fed5a4517c3a, authenticated_id=bd1704ee-625f-4def-a9ae-c0400e22e0ef, ehcache-txid=10791462, trace_id=08b2e1fd778b1ea729ee8f7248a308a1, trace_flags=01, span_id=21128e975839bf6e]
2022-10-28 01:57:51.605 [jobExecutor-5] INFO c.c.d.c.s.file.impl.FileServiceImpl - Calculated MD5 checksum d41d8cd98f00b204e9800998ecf8427e for the file exportd206635faef8433f9c500c6c3b6fb208.csv /collibra/collibra_data/dgc/temp-files/3136d3eb-f3d8-4d9b-8563-aa89670055dd [authenticated_id=00000000-0000-0000-0000-000000900001, session_hash=4b43e111f47986d8a4e8623892fd35cc, ehcache-txid=10791462, trace_id=08b2e1fd778b1ea729ee8f7248a308a1, trace_flags=01, span_id=21128e975839bf6e]
2022-10-28 01:57:51.608 [jobExecutor-5] ERROR c.c.d.c.io.job.OutputViewJobRunner - An error occurred during the export [authenticated_id=bd1704ee-625f-4def-a9ae-c0400e22e0ef, session_hash=ed4a004220b3685e16e1fed5a4517c3a, ehcache-txid=10791462, trace_id=08b2e1fd778b1ea729ee8f7248a308a1, trace_flags=01, span_id=21128e975839bf6e]
com.collibra.common.exception.CollibraIllegalArgumentException: fileResourceSizeTooSmall
Params: [exportd206635faef8433f9c500c6c3b6fb208.csv]*
at com.collibra.dgc.core.service.file.impl.FileServiceImpl.checkFileSize(FileServiceImpl.java:541)*
at com.collibra.dgc.core.service.file.impl.FileServiceImpl.addFile(FileServiceImpl.java:202)*
at com.collibra.dgc.core.service.file.impl.FileServiceImpl.addFile(FileServiceImpl.java:185)*
at jdk.internal.reflect.GeneratedMethodAccessor3992.invoke(Unknown Source)*
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)*
at java.base/java.lang.reflect.Method.invoke(Unknown Source)*
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)*
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)*
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)*
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)*
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)*
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)*
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)*
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)*
at com.sun.proxy.$Proxy367.addFile(Unknown Source)*
at com.collibra.dgc.core.io.connector.output.impl.file.AbstractFileWrapper.lambda$addFileAndSkipChecksIfNeeded$1(AbstractFileWrapper.java:103)*
at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:82)*
at com.collibra.dgc.core.application.security.SecurityContextHolderUtils.runAs(SecurityContextHolderUtils.java:55)*
at com.collibra.dgc.core.application.security.SecurityContextHolderUtils.runAsSystemUser(SecurityContextHolderUtils.java:67)*
at com.collibra.dgc.core.io.connector.output.impl.file.AbstractFileWrapper.addFileAndSkipChecksIfNeeded(AbstractFileWrapper.java:99)*
at com.collibra.dgc.core.io.connector.output.impl.file.chunked.CsvChunkedFileWrapper.closeFile(CsvChunkedFileWrapper.java:55)*
at com.collibra.dgc.core.io.connector.output.impl.CSVOutputConnector.finishExportToFile(CSVOutputConnector.java:103)*
at java.base/java.lang.Iterable.forEach(Unknown Source)*
at com.collibra.dgc.core.graph.query.engine.GraphQueryEngineToOutputConnectorAdapter.executeDataFetchingInChunks(GraphQueryEngineToOutputConnectorAdapter.java:250)*
at com.collibra.dgc.core.graph.query.engine.GraphQueryEngineToOutputConnectorAdapter.executeQueryAndPopulateOutputConnectors(GraphQueryEngineToOutputConnectorAdapter.java:128)*
at com.collibra.dgc.core.io.job.OutputViewJobRunner.executeDataFetching(OutputViewJobRunner.java:212)*
at com.collibra.dgc.core.io.job.OutputViewJobRunner.run(OutputViewJobRunner.java:177)*
at com.collibra.job.api.IJobRunner.execute(IJobRunner.java:50)*
at com.collibra.dgc.core.service.job.impl.JobServiceImpl$RunJob$1.doInTransactionWithoutResult(JobServiceImpl.java:1255)*
at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:36)*
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)*
at com.collibra.dgc.core.exceptions.ExceptionResolvingWithCachingTransactionTemplate.execute(ExceptionResolvingWithCachingTransactionTemplate.java:28)*
at com.collibra.dgc.core.service.job.impl.JobServiceImpl$RunJob.runJobInTransaction(JobServiceImpl.java:1249)*
at com.collibra.dgc.core.service.job.impl.JobServiceImpl$RunJob.postInitRun(JobServiceImpl.java:1117)*
at com.collibra.dgc.core.service.job.impl.JobServiceImpl$RunJob.run(JobServiceImpl.java:1085)*
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:89)*
at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:152)*
at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:82)*
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)*
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)*
at java.base/java.lang.Thread.run(Unknown Source)*
2022-10-28 01:57:51.609 [jobExecutor-5] ERROR c.c.d.c.s.job.impl.JobServiceImpl - Job ActiveJob{id=‘1e0f73fa-61c9-469f-b629-4019f931e367’, name=‘Export to “exportd206635faef8433f9c500c6c3b6fb208.csv”.’, user=‘bd1704ee-625f-4def-a9ae-c0400e22e0ef’, state=ERROR} error: {} [authenticated_id=bd1704ee-625f-4def-a9ae-c0400e22e0ef, session_hash=ed4a004220b3685e16e1fed5a4517c3a, ehcache-txid=10791462, trace_id=08b2e1fd778b1ea729ee8f7248a308a1, trace_flags=01, span_id=21128e975839bf6e]
com.collibra.common.exception.CollibraIllegalArgumentException: fileResourceSizeTooSmall
Params: [exportd206635faef8433f9c500c6c3b6fb208.csv]*
at com.collibra.dgc.core.service.file.impl.FileServiceImpl.checkFileSize(FileServiceImpl.java:541)*
at com.collibra.dgc.core.service.file.impl.FileServiceImpl.addFile(FileServiceImpl.java:202)*
at com.collibra.dgc.core.service.file.impl.FileServiceImpl.addFile(FileServiceImpl.java:185)*
at jdk.internal.reflect.GeneratedMethodAccessor3992.invoke(Unknown Source)*
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)*
at java.base/java.lang.reflect.Method.invoke(Unknown Source)*
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)*
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)*
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)*
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)*
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)*
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)*
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)*
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)*
at com.sun.proxy.$Proxy367.addFile(Unknown Source)*
at com.collibra.dgc.core.io.connector.output.impl.file.AbstractFileWrapper.lambda$addFileAndSkipChecksIfNeeded$1(AbstractFileWrapper.java:103)*
at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:82)*
at com.collibra.dgc.core.application.security.SecurityContextHolderUtils.runAs(SecurityContextHolderUtils.java:55)*
at com.collibra.dgc.core.application.security.SecurityContextHolderUtils.runAsSystemUser(SecurityContextHolderUtils.java:67)*
at com.collibra.dgc.core.io.connector.output.impl.file.AbstractFileWrapper.addFileAndSkipChecksIfNeeded(AbstractFileWrapper.java:99)*
at com.collibra.dgc.core.io.connector.output.impl.file.chunked.CsvChunkedFileWrapper.closeFile(CsvChunkedFileWrapper.java:55)*
at com.collibra.dgc.core.io.connector.output.impl.CSVOutputConnector.finishExportToFile(CSVOutputConnector.java:103)*
at java.base/java.lang.Iterable.forEach(Unknown Source)*
at com.collibra.dgc.core.graph.query.engine.GraphQueryEngineToOutputConnectorAdapter.executeDataFetchingInChunks(GraphQueryEngineToOutputConnectorAdapter.java:250)*
at com.collibra.dgc.core.graph.query.engine.GraphQueryEngineToOutputConnectorAdapter.executeQueryAndPopulateOutputConnectors(GraphQueryEngineToOutputConnectorAdapter.java:128)*
at com.collibra.dgc.core.io.job.OutputViewJobRunner.executeDataFetching(OutputViewJobRunner.java:212)*
at com.collibra.dgc.core.io.job.OutputViewJobRunner.run(OutputViewJobRunner.java:177)*
at com.collibra.job.api.IJobRunner.execute(IJobRunner.java:50)*
at com.collibra.dgc.core.service.job.impl.JobServiceImpl$RunJob$1.doInTransactionWithoutResult(JobServiceImpl.java:1255)*
at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:36)*
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)*
at com.collibra.dgc.core.exceptions.ExceptionResolvingWithCachingTransactionTemplate.execute(ExceptionResolvingWithCachingTransactionTemplate.java:28)*
at com.collibra.dgc.core.service.job.impl.JobServiceImpl$RunJob.runJobInTransaction(JobServiceImpl.java:1249)*
at com.collibra.dgc.core.service.job.impl.JobServiceImpl$RunJob.postInitRun(JobServiceImpl.java:1117)*
at com.collibra.dgc.core.service.job.impl.JobServiceImpl$RunJob.run(JobServiceImpl.java:1085)*
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:89)*
at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:152)*
at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:82)*
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)*
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)*
at java.base/java.lang.Thread.run(Unknown Source)*

scottnichols

Posted 3 years ago · Edited 1 year ago·Last reply 3 years ago

1 comment

springboot_team

·3 years ago · Edited

Hi @scott.nichols,

It seems that logs provided are of the Collibra Data Intelligence Cloud instance, where the export failed with an exception:

com.collibra.common.exception.CollibraIllegalArgumentException: fileResourceSizeTooSmall
Params: [exportd206635faef8433f9c500c6c3b6fb208.csv]*

The instance appears to throw an error when there is no data that is exported – i.e. the result is empty for the specified TableViewConfig.

However, the Collibra Integration Library should handle this and return an empty list. Just to note that is when the CollibraExportApiHelper class.

Therefore, can you please confirm what is the error with the Collibra Integration Library and maybe the stack trace? Thanks

1	Grant Rollerson SUNCORP CORPORATE SERVICES PTY LTD
2	Tom Friesen Algonquin College
3	SB S Benson Plus3 IT
4	S stefan_busch
5	Brian Lloyd USA TODAY Co.
6	D daverichard