Azure Purview and Collibra Integration - pom.xml error

Hi @abdutta.geico.com,

From the error details provided it might be that Maven is not installed or configured correctly, therefore, it is not able to download the Maven dependencies required by this integration.

Can you please try following the steps in the Installation section of the Azure Purview integration documentation?

Since the Eclipse IDE is being used, you can also review the Development with the Eclipse IDE [Optional] section.

Thanks

Hi @abdutta.geico.com,

The issue might be occurring because your machine is being blocked from fetching the integration’s dependencies.
To test this, can you try accessing this link in a browser from the same machine you are running the integration on:
https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/2.5.12/spring-boot-starter-parent-2.5.12.pom.

Kindly note that if you don’t want to perform any code modifications and you want to use the integration directly, you can try running this integration from the provided jar file by running the following command:

java -jar sbi-azure-purview-1.2.2.jar -Dspring.config.location=src/main/resources/application.properties
Thanks

Hi @spring-team.collibra.com,

Thanks. I tried that and got this error

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2023-01-06 11:10:16,991 [main] ERROR org.springframework.boot.SpringApplication - Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'azurePurviewSearchComponent': Unsatisfied dependency expressed through field 'uriLimit'; nested exception is org.springframework.beans.TypeMismatchException: Failed to convert value of type 'java.lang.String' to required type 'int'; nested exception is java.lang.NumberFormatException: For input string: ""
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:659) ~[spring-beans-5.3.18.jar!/:5.3.18]

I have set -azure.uri.limit=60

This is what I ran -
java -jar sbi-azure-purview-1.2.2.jar -Dspring.config.location=C:\Code\sbi-template-azure-purview-1.2.2\sbi-template-azure-purview-1.2.2\src\main\resources\application.properties

Hi @spring-team.collibra.com,

On a separate note I was able to resolve the maven dependencies and run, but when I am running this from Postman its failing

2023-01-06 11:48:48,631 [http-nio-8443-exec-6] INFO  org.apache.coyote.http11.Http11Processor - Error parsing HTTP request header
 Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name [0x160x030x010x000xf70x010x000x000xf30x030x030x8b0xa5tV0x000x900xab)0x070xa00xe7s0xddGl0xfc0x0ay0x98DkP0xd90xfc0xc60xf9;0xb90x8eE-0x98 ]. HTTP method names must be tokens
        at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:419) ~[tomcat-embed-core-9.0.60.jar:9.0.60]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:271) [tomcat-embed-core-9.0.60.jar:9.0.60]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.60.jar:9.0.60]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889) [tomcat-embed-core-9.0.60.jar:9.0.60]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) [tomcat-embed-core-9.0.60.jar:9.0.60]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.60.jar:9.0.60]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-9.0.60.jar:9.0.60]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-9.0.60.jar:9.0.60]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.60.jar:9.0.60]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_352] 

Below is the curl command

curl --location --request POST 'https://localhost:8443/sync' \

--header 'Authorization: Basic cHVydmlldzpFTkMoK2JRYUFYZVViWUxZUEgwN0xDZDN1QXRKVll6dS9FNE1UdjN4S2NYOG9GVnk0TktYQVdFaE81eVZjc08wTGFTUyk='

Hi @abdutta.geico.com,

Thanks for the update.

Regarding the error, it seems that the issue might be that from Postman HTTPS is being used, while the Purview Spring Boot integration is configured to allow HTTP requests – e.g. server.ssl.enabled is set to false.

Therefore, can you please update the request protocol from Postman to use HTTP instead of HTTPS?

In case you would like to use HTTPS, you should:

• Set server.ssl.enabled to true
• Create a certificate and set the server.ssl.* properties

For information on generating a self-signed certificate, you can refer to the documentation, section “Generate a Self-Signed Certificate using the Java Keytool”.

Thanks

Yeah thanks I got that resolved. Now I have everything setup as false and enabled only ADLS gen 2

azure.datalake.gen2.upsert.level=FIELD

azure.datalake.gen2.account.whitelist=

azure.datalake.gen2.account.blacklist=

azure.datalake.gen2.service.whitelist=

azure.datalake.gen2.service.blacklist=

azure.datalake.gen2.filesystem.whitelist=

azure.datalake.gen2.filesystem.blacklist=

azure.datalake.gen2.path.whitelist=

azure.datalake.gen2.path.blacklist=

Yet when I am running the application I am getting 404 Not Found

2023-01-06 15:46:11,986 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.FullSyncProcessor - Domains upsert completed
2023-01-06 15:46:11,987 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Initiating Search for :: azure_storage_account
2023-01-06 15:46:12,070 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Preparing and posting to Purview advanced search for azure_storage_account starting at 0 and limit to 100 with:
{filter={and=[{entityType=azure_storage_account, includeSubTypes=false}]}, keywords=*, offset=0, limit=100}
2023-01-06 15:46:13,186 [http-nio-8443-exec-1] ERROR com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - 404 Not Found: [no body]
2023-01-06 15:46:14,291 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Preparing and posting to Purview advanced search for azure_storage_account starting at 0 and limit to 100 with:

What else is missing

I changed the below variable and after that it worked and added some of the assets

purview.rest.api.base.path=/catalog/api/atlas/v2
Added /catalog before /api

It added the storage account, service and container names for the adls gen2 and after that its failing with NullPointer Exception…

What else can cause this…

2023-01-09 23:44:42,135 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.FullSyncProcessor - Domains upsert completed
2023-01-09 23:44:42,135 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Initiating Search for :: azure_storage_account
2023-01-09 23:44:42,230 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Preparing and posting to Purview advanced search for azure_storage_account starting at 0 and limit to 60 with:
{filter={and=[{entityType=azure_storage_account, includeSubTypes=false}]}, keywords=*, offset=0, limit=60}
2023-01-09 23:44:43,765 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Finished Search for :: azure_storage_account

2023-01-09 23:44:43,765 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Importing 2 objects...
2023-01-09 23:44:43,766 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Importing 0 to 2 objects out of 2.
2023-01-09 23:44:43,880 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.util.CollibraImportUtil - Temporary Import Load File located at ::
C:\Users\XXXXXX\AppData\Local\Temp\loadFile7628401101920866375
2023-01-09 23:44:45,024 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Import Job completed for Job ID:: 901c0b61-1355-4a76-9471-598d414a7a6f
2023-01-09 23:44:45,024 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Imported all 2 objects
2023-01-09 23:44:45,118 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Initiating Search for :: azure_datalake_gen2_service
2023-01-09 23:44:45,168 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Preparing and posting to Purview advanced search for azure_datalake_gen2_service starting at 0 and limit to 60 with:
{filter={and=[{entityType=azure_datalake_gen2_service, includeSubTypes=false}]}, keywords=*, offset=0, limit=60}
2023-01-09 23:44:46,462 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Finished Search for :: azure_datalake_gen2_service

2023-01-09 23:44:46,463 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Importing 1 objects...
2023-01-09 23:44:46,464 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Importing 0 to 1 objects out of 1.
2023-01-09 23:44:46,533 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.util.CollibraImportUtil - Temporary Import Load File located at ::
C:\Users\XXXXX\AppData\Local\Temp\loadFile5316200695887683038
2023-01-09 23:44:47,678 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Import Job completed for Job ID:: a4c7c997-8364-4ec0-b2da-b3e5d1ea0502
2023-01-09 23:44:47,679 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Imported all 1 objects
2023-01-09 23:44:47,682 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Initiating Search for :: azure_datalake_gen2_filesystem
2023-01-09 23:44:47,732 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Preparing and posting to Purview advanced search for azure_datalake_gen2_filesystem starting at 0 and limit to 60 with:
{filter={and=[{entityType=azure_datalake_gen2_filesystem, includeSubTypes=false}]}, keywords=*, offset=0, limit=60}
2023-01-09 23:44:48,792 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.AzurePurviewSearchComponent - Finished Search for :: azure_datalake_gen2_filesystem

2023-01-09 23:44:48,795 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Importing 27 objects...
2023-01-09 23:44:48,795 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Importing 0 to 27 objects out of 27.
2023-01-09 23:44:48,874 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.util.CollibraImportUtil - Temporary Import Load File located at ::
C:\Users\XXXXXXXXX\AppData\Local\Temp\loadFile1394590327223069832
2023-01-09 23:44:51,083 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Import Job completed for Job ID:: e1b56a68-206c-4519-95a3-6167b615fed6
2023-01-09 23:44:51,083 [http-nio-8443-exec-1] INFO  com.collibra.marketplace.azure.purview.component.CollibraGenericImportComponent - Imported all 27 objects
java.lang.NullPointerException
        at com.collibra.marketplace.azure.purview.FullSyncProcessor.start(FullSyncProcessor.java:557)
        at com.collibra.marketplace.azure.purview.controller.EntryPointController.fullSyncTriggeredByApiRequest(EntryPointController.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:178)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:750)

Hi @spring-team.collibra.com,

Any insights to move forward from here will be useful.

Regards,
Abhishek

Hi @spring-team.collibra.com,

azure.datalake.gen2.inferred.directories what does this do… I am trying to understand this more. I thought default is true so did not add but now explicitly I made that true and is working. Can you give some examples and help here.

I faced another issue where I was getting error when inserting data, and after debugging I got that schema needs to be mentioned as well. collibra.asset.file.schema=Schema

Now I am getting all the assets, fields, schema, etc…

Thank you for your support.

Regards,
Abhishek