P

Wednesday, March 15th, 2023 10:46 AM

DBT to Collibra Integration(v1.2.2) Error writing to server

Hello @spring-team.collibra.com ,

We’re getting a Error writing to server error.

We installed and tested the dbt to Collibra Integration (v1.2.2) in January; It ran and created assets in Collibra.
We then waited to run it again until we had installed Edge. (They are on separate machines)

Now when we try to run the dbt to Collibra Integration it crashes on an exception. It fails when it tries to create assets.
We cleaned up our environment by removing all previously created assets before rerunning it.

 2023-03-15T09:54:16.379Z  INFO 3392 --- [           main] com.collibra.dbt.Application             : Started Application in 39.15 seconds (process running for 47.239)
2023-03-15T09:54:56.376Z  INFO 3392 --- [0.0-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2023-03-15T09:54:56.381Z  INFO 3392 --- [0.0-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2023-03-15T09:54:56.383Z  INFO 3392 --- [0.0-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 2 ms
2023-03-15T09:54:57.224Z  INFO 3392 --- [0.0-8080-exec-1] c.c.dbt.controller.EntryPointController  : Extract SQL for all accounts
2023-03-15T09:55:00.405Z  INFO 3392 --- [0.0-8080-exec-1] c.c.dbt.component.DBTLineageExtraction   : Processing Account id:18748 Project id:43985
2023-03-15T09:55:02.053Z  INFO 3392 --- [0.0-8080-exec-1] c.c.dbt.component.DBTLineageExtraction   : Job is running! JobId:195960
2023-03-15T10:11:31.461Z  INFO 3392 --- [0.0-8080-exec-1] c.c.dbt.component.DBTLineageExtraction   : Job run has been completed
2023-03-15T10:11:31.493Z  INFO 3392 --- [0.0-8080-exec-1] c.c.dbt.component.DBTLineageExtraction   : Snapshots extraction performing
2023-03-15T10:11:32.340Z  INFO 3392 --- [0.0-8080-exec-1] c.c.dbt.component.DBTLineageExtraction   : Extracted Compiled Sql files are writing to the local lineage directory
2023-03-15T10:11:32.342Z  INFO 3392 --- [0.0-8080-exec-1] c.c.dbt.component.DBTLineageExtraction   : ETL operations successfully has been completed
2023-03-15T10:11:32.343Z  INFO 3392 --- [0.0-8080-exec-1] c.c.dbt.component.DBTLineageExtraction   : Models extraction performing
2023-03-15T10:11:38.239Z  INFO 3392 --- [0.0-8080-exec-1] c.c.dbt.component.DBTLineageExtraction   : Extracted Compiled Sql files are writing to the local lineage directory
2023-03-15T10:11:46.689Z  INFO 3392 --- [0.0-8080-exec-1] c.c.dbt.component.DBTLineageTransformer  : Transforms DBT database entities are performing
2023-03-15T10:13:21.032Z  INFO 3392 --- [0.0-8080-exec-1] c.c.dbt.component.DBTLineageTransformer  : Database assets transformer operation has been completed
2023-03-15T10:13:24.149Z ERROR 3392 --- [0.0-8080-exec-1] c.c.dbt.component.CollibraOperations     : Import has been failed

com.collibra.marketplace.library.integration.exception.CollibraIntegrationLibraryException: {"message":"An unexpected error occurred during execution.","details":"java.io.IOException: Error writing to server","cause":"Error writing to server"}
        at com.collibra.marketplace.library.integration.CollibraImportApiHelper.internalImportAssetsInBatches(CollibraImportApiHelper.java:663) ~[collibra-integration-library-1.2.1.jar!/:1.2.1]
        at com.collibra.marketplace.library.integration.CollibraImportApiHelper.importAssets(CollibraImportApiHelper.java:129) ~[collibra-integration-library-1.2.1.jar!/:1.2.1]
        at com.collibra.dbt.component.CollibraOperations.createAssets(CollibraOperations.java:68) ~[classes!/:1.2.2]
        at com.collibra.dbt.component.DBTLineageTransformer.createDatabaseAssets(DBTLineageTransformer.java:144) ~[classes!/:1.2.2]
        at com.collibra.dbt.component.DBTLineageExtraction.extractSql(DBTLineageExtraction.java:207) ~[classes!/:1.2.2]
        at com.collibra.dbt.component.DBTLineageExtraction.extractSqlByJob(DBTLineageExtraction.java:149) ~[classes!/:1.2.2]
        at com.collibra.dbt.component.DBTLineageExtraction.lambda$processProjectsInAccounts$13(DBTLineageExtraction.java:306) ~[classes!/:1.2.2]
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) ~[na:na]
        at com.collibra.dbt.component.DBTLineageExtraction.lambda$processProjectsInAccounts$14(DBTLineageExtraction.java:280) ~[classes!/:1.2.2]
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) ~[na:na]
        at com.collibra.dbt.component.DBTLineageExtraction.processProjectsInAccounts(DBTLineageExtraction.java:280) ~[classes!/:1.2.2]
        at com.collibra.dbt.component.DBTLineageExtraction.findAllProjectsInAccount(DBTLineageExtraction.java:241) ~[classes!/:1.2.2]
        at com.collibra.dbt.component.DBTLineageExtraction.requestAllAccounts(DBTLineageExtraction.java:125) ~[classes!/:1.2.2]
        at com.collibra.dbt.controller.EntryPointController.requestAllAccounts(EntryPointController.java:38) ~[classes!/:1.2.2]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]

2 years ago

We think we’ve solved it. :smile:

We changed from Collibra Integration Library v1.2.1 to Spring Boot Integration Library (Spring Boot 3.x) (installed it, updated the pom.xml and rebuild the integration).

Our best guess is that some breaking changes must have been introduced somewhere in the back-end.

8 Messages

 • 

710 Points

1 year ago

@spring-team.collibra.com

I’ve encountered similar problem when using CollibraImportApiHelper.class to import assets on our server application.

library: collibra-integration-library-sb3-1.0.1.jar
tag: community-offering

Problem appears when import is called after some time of inactivity (few hours) - the only way to fix it is to reboot the application.
For short periods of inactivity (about 1 hour), I see that library is able to re-login in case of token expiration and everything works fine.

stacktrace logs below:

Caused by: jakarta.ws.rs.ProcessingException: java.io.IOException: Error writing to server
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:275) ~[jersey-client-3.1.3.jar:na]
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) ~[jersey-client-3.1.3.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:662) ~[jersey-client-3.1.3.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[jersey-client-3.1.3.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[jersey-client-3.1.3.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.1.3.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.1.3.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.1.3.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-3.1.3.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[jersey-client-3.1.3.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:661) ~[jersey-client-3.1.3.jar:na]
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:439) ~[jersey-client-3.1.3.jar:na]
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:345) ~[jersey-client-3.1.3.jar:na]
at com.collibra.internal.requests.ImportRequests.lambda$importJson$3(ImportRequests.java:155) ~[collibra-integration-library-sb3-1.0.1.jar:1.0.1]
at com.collibra.internal.requests.CollibraRequest.attemptAndHandleHttpRequest(CollibraRequest.java:110) ~[collibra-integration-library-sb3-1.0.1.jar:1.0.1]
at com.collibra.internal.requests.ImportRequests.importJson(ImportRequests.java:147) ~[collibra-integration-library-sb3-1.0.1.jar:1.0.1]
at com.collibra.internal.logic.ImportOperationsLogic.importJson(ImportOperationsLogic.java:569) ~[collibra-integration-library-sb3-1.0.1.jar:1.0.1]
at com.collibra.internal.logic.ImportOperationsLogic.importJsonFromFile(ImportOperationsLogic.java:425) ~[collibra-integration-library-sb3-1.0.1.jar:1.0.1]
at com.collibra.internal.logic.UpsertAssetsOperationsLogic.upsertAssets(UpsertAssetsOperationsLogic.java:537) ~[collibra-integration-library-sb3-1.0.1.jar:1.0.1]
at com.collibra.internal.logic.UpsertAssetsOperationsLogic.sbUpsertAssetsByName(UpsertAssetsOperationsLogic.java:190) ~[collibra-integration-library-sb3-1.0.1.jar:1.0.1]
at com.collibra.marketplace.library.integration.CollibraImportApiHelper.lambda$internalImportAssets$1(CollibraImportApiHelper.java:705) ~[collibra-integration-library-sb3-1.0.1.jar:1.0.1]
at com.collibra.marketplace.library.integration.InternalSharedLibraryCall.callAndHandle(InternalSharedLibraryCall.java:46) ~[collibra-integration-library-sb3-1.0.1.jar:1.0.1]
at com.collibra.marketplace.library.integration.CollibraImportApiHelper.internalImportAssets(CollibraImportApiHelper.java:703) ~[collibra-integration-library-sb3-1.0.1.jar:1.0.1]
… 124 common frames omitted
Caused by: java.io.IOException: Error writing to server
at java.base/sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source) ~[na:na]
at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source) ~[na:na]
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source) ~[na:na]
at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:415) ~[jersey-client-3.1.3.jar:na]
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:273) ~[jersey-client-3.1.3.jar:na]
… 146 common frames omitted

Loading...