Unity Catalog to Collibra integration - 'commandController' Internal Server Error

@spring-team.collibra.com Might you be able to help @łukasz.banaszewski with this?

Hi @łukasz.banaszewski,

We are reviewing this and will keep you updated. Thanks

Hi @spring-team.collibra.com,
Have you reviewed above issue? Any feedback?

Thanks

@spring-team.collibra.com
Before posting this thread I’ve already checked it without encryption - same error.

@spring-team.collibra.com
I’ve investigated further. Strangely the issue occurs when the springboot projects wants to authenticate to Collibra DGC. I’ve checked the connectivity via the same username / password via CURL or HTTP outside of the Springboot project and it works correctly.

In the application.properties I’ve also changed the URL to another Collibra DGC instance and this time the integration started properly.

Can it be some security setting in Java or on the DGC Cloud side?

Please advise

Hi @łukasz.banaszewski,

Thanks for the update.

To determine what the issue might be, can you please provide the following information:

1. Version of both cloud instances (can be found from the Settings page)
2. Whether a proxy server is used to connect to any of the instances?
3. The logs, when running the integration with the Collibra instance that is not working. Please remove any sensitive information prior to doing so (can be sent as a private message).

Thanks

Hi @spring-team.collibra.com

1. Build Number 20230102165117 Product Version 2023.01.0-66 (both cloud instances)
2. No (both cloud instances)
3. Logs from instance that the integration does not work attached below:

logs2.txt (168.1 KB)

Hi @łukasz.banaszewski,

Thanks for the information provided.

When switching from one instance to another, can you please advise which properties were updated (i.e. the property name, not the value) and any other changes that were made?

Additionally, to help us debug this further, can you please from the Unity Catalog integration source code:

• Open the CommandController class
• Find line 2172 – the constructor method
• Replace the constructor with the content in the attached file
• Compile and test the integration again
• Forward the logs here or as a private message. Please remove any sensitive information.

Thanks

CommandControllerSnippet.txt (3.1 KB)

Hi @spring-team.collibra.com,
I have put the logger.debug lines to CommandController.class and here is the output:

[2023-01-19 10:01:53.166][DEBUG][main][org.springframework.beans.factory.support.DefaultListableBeanFactory] Creating shared instance of singleton bean ‘commandController’
[2023-01-19 10:01:53.176][DEBUG][main][org.springframework.core.env.PropertySourcesPropertyResolver] Found key ‘collibra.url’ in PropertySource ‘environmentProperties’ with value of type String
[2023-01-19 10:01:53.176][DEBUG][main][org.springframework.core.env.PropertySourcesPropertyResolver] Found key ‘collibra.username’ in PropertySource ‘environmentProperties’ with value of type String
[2023-01-19 10:01:53.177][DEBUG][main][org.springframework.core.env.PropertySourcesPropertyResolver] Found key ‘collibra.password’ in PropertySource ‘environmentProperties’ with value of type String
[2023-01-19 10:01:53.177][DEBUG][main][com.collibra.databricks.uc.template.controller.CommandController] Initializing the command controller
[2023-01-19 10:01:53.177][DEBUG][main][com.collibra.databricks.uc.template.controller.CommandController] Setting the Core API client
[2023-01-19 10:01:53.259][DEBUG][main][com.collibra.databricks.uc.template.controller.CommandController] Asset types request
[2023-01-19 10:01:53.812][WARN ][main][org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘commandController’ defined in file [C:\Users\lbanasze\Downloads\sbi-template-unitycatalog-2.0.9\target\classes\com\collibra\databricks\uc\template\controller\CommandController.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.collibra.databricks.uc.template.controller.CommandController]: Constructor threw exception; nested exception is collibra.core.client.ApiException: Internal Server Error
[2023-01-19 10:01:53.812][INFO ][main][org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] Closing JPA EntityManagerFactory for persistence unit ‘default’

Best regards

Hi @łukasz.banaszewski ,

After investigating the logs, we notice that the integration fails when sending the first request to your Collibra instance. This request is the GET findAssetsRequest call. Please use tools such as Postman or CUrl to send the request locally from the same machine running the integration.

curl -X ‘GET’
https://>YOUR COLLIBRA INSTANCE</rest/2.0/assetTypes?offset=0&limit=0&countLimit=-1&nameMatchMode=ANYWHERE&excludeMeta=true&topLevel=false’
-H ‘accept: application/json’ -u “>USERNAME<:>PASSWORD<”

and replace:
>YOUR COLLIBRA INSTANCE<
>USERNAME< and
>PASSWORD<

Then investigate the response and update us with any errors or status codes. If the result is successful, please don’t reply with the response, as it might contain confidential information. If you get an error try adding –insecure before your Collibra URL in the CUrl request.

Thanks

Hi @spring-team.collibra.com
As mentioned in my previous posts the CURL, HTTP, POSTMAN works correctly.

I’ve also used your CURL code with my collibra instance and username/password credentials and it also works.

The connectivity only does not work when I am using Intellij or run the Java Project from command line (without any IDE).

Best regards

Hi @łukasz.banaszewski ,

Since you did not get any errors when sending these requests, the issue is not a connection problem. Therefore, please open the Collibra Instance’s logs, watch for any errors when triggering the integration, and update this post accordingly.

To view these logs, go to your Collibra instance’s console, usually in the format of console->YOUR COLLIBRA INSTANCE< .com. Once again, please hide any sensitive information from your replies.

Thanks

Hi @spring-team.collibra.com
Here is the console error:

2023-01-24 12:03:59.186 [http-nio-0.0.0.0-4400-exec-8] ERROR o.s.b.w.s.support.ErrorPageFilter - Forwarding to error page from request [/rest/2.0/assetTypes] due to exception [The request was rejected because the URL contained a potentially malicious String “//”] [trace_id=cd754925982b908d9d02bad265513eea, trace_flags=01, span_id=8d957f92a32f379a]
org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL contained a potentially malicious String “//”
at org.springframework.security.web.firewall.StrictHttpFirewall.rejectedBlocklistedUrls(StrictHttpFirewall.java:535)
at org.springframework.security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:505)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:206)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:142)
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.servlet.OpenTelemetryHandlerMappingFilter.doFilter(OpenTelemetryHandlerMappingFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126)
at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64)
at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at com.collibra.dgc.web.filter.HibernateStatsFilter.doFilter(HibernateStatsFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.orm.hibernate5.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:156)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)

Hi @łukasz.banaszewski ,

Yes , the collibra.url property doesn’t have a trailing ‘/’ and should be in this format: https://>YOUR INSTANCE<.collibra.com/rest/2.0.

You can you try sending a request to the Collibra API as follows;

Thanks