Azure Purview SpringBoot Integration

Can @spring-team.collibra.com you help us on the above query?

Hi Team,

I am getting the below warning in my instance console

I have provided the internal user(collibra user credentails in the application property) the catalog author role . any other permission settings i need to give to the Catalog Author role?

Hi @spring-team.collibra.com

can you help me with the above scenario.?

Thanks

Hi @spring-team.collibra.com team,

Thanks for replying .As suggested by you I already checked the whitespaces its not there.
also in the browser its showing bad credentials.

the logs
2023-01-11 16:23:30,169 [background-preinit] INFO org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 6.2.3.Final
2023-01-11 16:23:30,237 [main] INFO com.collibra.marketplace.azure.purview.Application - Starting
2023-01-11 16:23:30,251 [main] INFO com.collibra.marketplace.azure.purview.Application - No active profile set, falling back to 1 default profile: “default”
2023-01-11 16:23:33,175 [main] INFO com.ulisesbocchio.jasyptspringboot.configuration.EnableEncryptablePropertiesBeanFactoryPostProcessor - Post-processing PropertySource instances
2023-01-11 16:23:33,181 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Skipping PropertySource configurationProperties [class org.springframework.boot.context.properties.source.ConfigurationPropertySourcesPropertySource
2023-01-11 16:23:33,183 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Converting PropertySource commandLineArgs [org.springframework.core.env.SimpleCommandLinePropertySource] to EncryptableEnumerablePropertySourceWrapper
2023-01-11 16:23:33,185 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Skipping PropertySource servletConfigInitParams [class org.springframework.core.env.PropertySource$StubPropertySource
2023-01-11 16:23:33,186 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Skipping PropertySource servletContextInitParams [class org.springframework.core.env.PropertySource$StubPropertySource
2023-01-11 16:23:33,186 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Converting PropertySource systemProperties [org.springframework.core.env.PropertiesPropertySource] to EncryptableMapPropertySourceWrapper
2023-01-11 16:23:33,186 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Converting PropertySource systemEnvironment [org.springframework.boot.env.SystemEnvironmentPropertySourceEnvironmentPostProcessor$OriginAwareSystemEnvironmentPropertySource] to EncryptableSystemEnvironmentPropertySourceWrapper
2023-01-11 16:23:33,187 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Converting PropertySource random [org.springframework.boot.env.RandomValuePropertySource] to EncryptablePropertySourceWrapper
2023-01-11 16:23:33,187 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Converting PropertySource Config resource ‘class path resource [application.properties]’ via location ‘optional:classpath:/’ [org.springframework.boot.env.OriginTrackedMapPropertySource] to EncryptableMapPropertySourceWrapper
2023-01-11 16:23:33,188 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Converting PropertySource defaultProperties [org.springframework.core.env.MapPropertySource] to EncryptableMapPropertySourceWrapper
2023-01-11 16:23:33,478 [main] INFO com.ulisesbocchio.jasyptspringboot.filter.DefaultLazyPropertyFilter - Property Filter custom Bean not found with name ‘encryptablePropertyFilter’. Initializing Default Property Filter
2023-01-11 16:23:33,494 [main] INFO com.ulisesbocchio.jasyptspringboot.resolver.DefaultLazyPropertyResolver - Property Resolver custom Bean not found with name ‘encryptablePropertyResolver’. Initializing Default Property Resolver
2023-01-11 16:23:33,501 [main] INFO com.ulisesbocchio.jasyptspringboot.detector.DefaultLazyPropertyDetector - Property Detector custom Bean not found with name ‘encryptablePropertyDetector’. Initializing Default Property Detector
2023-01-11 16:23:33,800 [main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8000 (http)
2023-01-11 16:23:33,830 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler [“http-nio-8000”]
2023-01-11 16:23:33,831 [main] INFO org.apache.catalina.core.StandardService - Starting service [Tomcat]
2023-01-11 16:23:33,832 [main] INFO org.apache.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.60]
2023-01-11 16:23:34,184 [main] INFO org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2023-01-11 16:23:34,186 [main] INFO org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 3841 ms
2023-01-11 16:23:35,420 [main] INFO org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver - Exposing 14 endpoint(s) beneath base path ‘/manage’
2023-01-11 16:23:36,479 [main] WARN org.springframework.boot.autoconfigure.security.servlet.UserDetailsServiceAutoConfiguration -

Using generated security password: feca301e-6f53-456d-9a6e-6e7d0cc28e67

This generated password is for development use only. Your security configuration must be updated before running your application in production.

2023-01-11 16:23:36,720 [main] INFO org.springframework.security.web.DefaultSecurityFilterChain - Will not secure any request
2023-01-11 16:23:37,024 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler [“http-nio-8000”]
2023-01-11 16:23:37,400 [main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8000 (http) with context path ‘’
2023-01-11 16:23:37,403 [main] INFO com.ulisesbocchio.jasyptspringboot.caching.RefreshScopeRefreshedEventListener - Refreshing cached encryptable property sources on ServletWebServerInitializedEvent
2023-01-11 16:23:37,410 [main] INFO com.ulisesbocchio.jasyptspringboot.caching.CachingDelegateEncryptablePropertySource - Property Source commandLineArgs refreshed
2023-01-11 16:23:37,410 [main] INFO com.ulisesbocchio.jasyptspringboot.caching.CachingDelegateEncryptablePropertySource - Property Source systemProperties refreshed
2023-01-11 16:23:37,410 [main] INFO com.ulisesbocchio.jasyptspringboot.caching.CachingDelegateEncryptablePropertySource - Property Source systemEnvironment refreshed
2023-01-11 16:23:37,411 [main] INFO com.ulisesbocchio.jasyptspringboot.caching.CachingDelegateEncryptablePropertySource - Property Source random refreshed
2023-01-11 16:23:37,412 [main] INFO com.ulisesbocchio.jasyptspringboot.caching.CachingDelegateEncryptablePropertySource - Property Source Config resource ‘class path resource [application.properties]’ via location ‘optional:classpath:/’ refreshed
2023-01-11 16:23:37,412 [main] INFO com.ulisesbocchio.jasyptspringboot.caching.CachingDelegateEncryptablePropertySource - Property Source defaultProperties refreshed
2023-01-11 16:23:37,413 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Converting PropertySource server.ports [org.springframework.core.env.MapPropertySource] to EncryptableMapPropertySourceWrapper
2023-01-11 16:23:37,413 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Skipping PropertySource configurationProperties [class org.springframework.boot.context.properties.source.ConfigurationPropertySourcesPropertySource
2023-01-11 16:23:37,413 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Skipping PropertySource servletConfigInitParams [class org.springframework.core.env.PropertySource$StubPropertySource
2023-01-11 16:23:37,413 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Converting PropertySource servletContextInitParams [org.springframework.web.context.support.ServletContextPropertySource] to EncryptableEnumerablePropertySourceWrapper
2023-01-11 16:23:37,413 [main] INFO com.ulisesbocchio.jasyptspringboot.EncryptablePropertySourceConverter - Converting PropertySource Management Server [org.springframework.boot.actuate.autoconfigure.web.server.ManagementContextAutoConfiguration$SameManagementContextConfiguration$1] to EncryptablePropertySourceWrapper
2023-01-11 16:23:38,208 [main] INFO com.collibra.marketplace.azure.purview.Application - Started Application in 8.875 seconds (JVM running for 21.809)
2023-01-11 16:23:39,505 [RMI TCP Connection(3)-127.0.0.1] INFO org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring DispatcherServlet ‘dispatcherServlet’
2023-01-11 16:23:39,506 [RMI TCP Connection(3)-127.0.0.1] INFO org.springframework.web.servlet.DispatcherServlet - Initializing Servlet ‘dispatcherServlet’
2023-01-11 16:23:39,508 [RMI TCP Connection(3)-127.0.0.1] INFO org.springframework.web.servlet.DispatcherServlet - Completed initialization in 2 ms

Hi @spring-team.collibra.com team,
Can you provide some suggestions I am stuck in the above scenario.

Thanks

Hi @khusbu.kumari.inter.ikea.com,

Thanks for the information provided.

From the logs it seems that Spring Security is generating a password on deployment (instead of using the ones defined in the properties file) which might be the reason why when triggering the integration via Postman, it is returning 401 Unauthorized.

org.springframework.boot.autoconfigure.security.servlet.UserDetailsServiceAutoConfiguration -

Using generated security password: feca301e-6f53-456d-9a6e-6e7d0cc28e67

This generated password is for development use only. Your security configuration must be updated before running your application in production.

Therefore, can you please confirm whether you made any changes to the AuthenticationConfig.java class or any other file? Thanks

Hi @khusbu.kumari.inter.ikea.com ,

Can you run the integration using the jar file directly downloaded from marketplace and let us know whether you face the same issue, please? https://marketplace.collibra.com/listings/azure-purview-to-collibra-integration/

You can use the following command:

java -jar sbi-azure-purview-1.2.2.jar --spring.config.location= >PATH TO YOUR .properties FILE<

Thanks

Hi @spring-team.collibra.com Team,
I downloaded the new template and it was working.
However I am facing a problem in blacklisting and whitelisting the filesystem entities. I am setting the full qualified name in the application property file of the one I want to blacklist and erased the null value from the code which was set in the FullSyncProcessor.java file.

Hi @spring-team.collibra.com team,

Thanks for the reply.
Ok we wont remove the null value from the source code.
But as shown in the screenshot above as of now I have only one filesystem which i want to blacklist.thats why given only its qualified name.

also my question is if I restrict one the filesystem then its Gen2path entities related to it will also get restricted?

Hi @khusbu.kumari.inter.ikea.com,

We looked into this in more detail. Our previous statement that “If you restrict a filesystem, all its child entities (including Gen2 paths and all its children) will also be filtered out.” is incorrect.

In the case of Gen2 accounts, services and filesystems, only the entities specified in the blacklist property will be restricted by the integration. All their child entities will still be imported into Collibra. On the other hand, child entities of a blacklisted Gen2 path (i.e. using azure.datalake.gen2.path.blacklist) will be restricted.

Therefore, in your case, is it correct to say that the one filesystem entity is being restricted but all its child path entities are being imported into Collibra? But you also want to restrict all the blacklisted filesystem’s child path entities, right?

To ignore the Gen2 path entities, you need to set the same qualified name of the filesystem in both azure.datalake.gen2.filesystem.blacklist and azure.datalake.gen2.path.blacklist.

Note that by setting a filesystem qualified name in azure.datalake.gen2.path.blacklist, the integration will restrict all paths whose qualified name starts with the qualified name you set.

Hope this helps.

Thanks

Hi @spring-team.collibra.com team

Thanks for the clarification.
In my case i just want to restrict one filesystem which i have given in the above screenshot.
However the blacklisted filesystem is still getting imported

Hi there @spring-team.collibra.com,
for the above scenario it worked actually long back. Thanks for the inputs.

I have a new query. When we run the application all the domains are created and the ones having assets in them remains there and the empty domains get deleted.

How do i blacklist the gen2account and service. domain i.e.
Azure Storage Account ,
Azure Datalake Gen2 Filesystem and
Azure Datalake Gen2 Service
(I have mentioned the number of entities in each domain because as there are 2 and both of them i m trying to restrict but in the code there are conditions )

For example.
I have 2 accounts and 2 services and 4 filesystem. interlinked to each other.
I dont want the above three domains to get created when i run the application however their child assets which are in the following domain i.e Azure Datalake Gen2 Path and Azure Datalake Gen2 Resource Set should be created.

Hence , for the above scenario in the application.properties I tried the following combination

azure.datalake.gen2.upsert.level=FIELD

azure.datalake.gen2.account.whitelist=
azure.datalake.gen2.account.blacklist= /// blacklisted both accounts with their full qualified name

azure.datalake.gen2.service.whitelist=
azure.datalake.gen2.service.blacklist= // blacklisted both services with their full qualified name

azure.datalake.gen2.filesystem.whitelist=
azure.datalake.gen2.filesystem.blacklist=// blacklisted all 3 filesystem with their full qualified name

azure.datalake.gen2.path.whitelist=
azure.datalake.gen2.path.blacklist=

But the result is neither assets nor a domain is getting ingested… The account blacklisting is blocking all its child asets too.

Hence, my query is how can I restrict the creation of those 3 domains(Azure Storage Account ,
Azure Datalake Gen2 Filesystem and
Azure Datalake Gen2 Service
) and at the same time import the other 2 domains ( Azure Datalake Gen2 Path and Azure Datalake Gen2 Resource Set)and its assets too.

Thanks,

Hi @spring-team.collibra.com

Can you help with the above scenario… I tried again with same properties this time account domain is not getting created but service is getting created and the asset name is having full name…