Skip to main content
N
naveenjayaram

12 Messages

Tuesday, September 13th, 2022 12:33 PM

Google Cloud Dataflow to Collibra Integration

Hi All,

I am trying to set up Google Cloud Dataflow to Collibra Integration application in my laptop.
I am able to start the spring boot application successfully but when i hit the service i get below error

{
“message”: “Internal error during execution.”,
“exceptionMessage”: “302 Found from GET https://console.cloud.google.com/dataflow/jobs; nested exception is org.springframework.web.reactive.function.UnsupportedMediaTypeException: Content type ‘application/binary’ not supported for bodyType=com.collibra.marketplace.gcp.dataflow.pojo.ListJobsResponse”
}

Could you please let me know what properties to be set for below fields
google.cloud.platform.base-url=
google.cloud.platform.scopes[0]=

Thanks & Regards,
Naveen

Questions

 • 

Updated 

2 years ago

110

10

0

0

Community_Alex

701 Messages

 • 

18.5K Points

2 years ago

@spring-team.collibra.com, is this something you could help @naveen.jayaram with or no?

0

0

springboot_team

368 Messages

2 years ago

Hi @naveen.jayaram,

Thanks for trying the GCP Dataflow to Collibra integration.

Can you please advise regarding the below:

  • Whether any modifications were made to the integration code?
  • Which version of the integration is being used?

For the specified properties, these need to be set as follows:

  • google.cloud.platform.base-url -> Set to the project URL. Example: https://dataflow.googleapis.com/v1b3/projects/<myproject>
  • google.cloud.platform.scopes[0]- > Set to the GCP Dataflow scope that should be used. Example: https://www.googleapis.com/auth/cloud-platform

More information is available in the GCP Dataflow to Collibra integration documentation.

Thanks

1

0

naveenjayaram

12 Messages

Hi @spring-team.collibra.com,

Thanks for replying.

There was no change done to the code.
we changed only the application.property file

trigger.api.username=
trigger.api.password=
collibra.url=
collibra.username=
collibra.password=
google.cloud.platform.credentials-path=
google.cloud.platform.base-url=
google.cloud.platform.scopes[0]=

I updated the property as suggested above
google.cloud.platform.base-url=https://dataflow.googleapis.com/v1b3/projects/<gcp_project_id>

but now i am getting a different error

2022-09-15 17:17:42.694 [http-nio-4442-exec-1] INFO  com.collibra.marketplace.gcp.dataflow.repository.JobCalls - Retrieving Job Details By Id.

2022-09-15 17:17:48.908 [http-nio-4442-exec-1] ERROR com.collibra.marketplace.gcp.dataflow.exception.GlobalExceptionHandler - Internal error during execution.
org.springframework.web.reactive.function.client.WebClientRequestException: Failed to resolve 'dataflow.googleapis.com’; nested exception is java.net.UnknownHostException: Failed to resolve ‘dataflow.googleapis.com’
at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) ~[spring-webflux-5.3.18.jar:5.3.18]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint ⇢ Request to GET https://dataflow.googleapis.com/v1b3/projects/burner-navjayar/jobs/2022-09-02_05_30_39-724786561846443801?view=JOB_VIEW_ALL [DefaultWebClient]
Original Stack Trace:
at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) ~[spring-webflux-5.3.18.jar:5.3.18]
at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.Mono.subscribe(Mono.java:4400) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:225) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:274) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:415) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:491) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:299) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:194) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$ClientTransportSubscriber.onError(HttpClientConnect.java:304) ~[reactor-netty-http-1.0.17.jar:1.0.17]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:194) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onError(DefaultPooledConnectionProvider.java:155) ~[reactor-netty-core-1.0.17.jar:1.0.17]
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:477) ~[reactor-netty-core-1.0.17.jar:1.0.17]
at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.lambda$drainLoop$9(SimpleDequePool.java:431) ~[reactor-netty-core-1.0.17.jar:1.0.17]
at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:194) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.onError(DefaultPooledConnectionProvider.java:538) ~[reactor-netty-core-1.0.17.jar:1.0.17]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.Operators.error(Operators.java:198) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.MonoError.subscribe(MonoError.java:53) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.Mono.subscribe(Mono.java:4400) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:534) ~[reactor-netty-core-1.0.17.jar:1.0.17]
at reactor.netty.transport.TransportConnector.lambda$doResolveAndConnect$11(TransportConnector.java:341) ~[reactor-netty-core-1.0.17.jar:1.0.17]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:109) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.InetSocketAddressResolver$2.operationComplete(InetSocketAddressResolver.java:86) ~[netty-resolver-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1055) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1000) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:418) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsResolveContext.access$600(DnsResolveContext.java:66) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:467) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsQueryContext.tryFailure(DnsQueryContext.java:240) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsQueryContext$4.run(DnsQueryContext.java:192) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) ~[netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99) ~[reactor-core-3.4.16.jar:3.4.16]
at reactor.core.publisher.Mono.block(Mono.java:1707) ~[reactor-core-3.4.16.jar:3.4.16]
at com.collibra.marketplace.gcp.dataflow.repository.JobCalls.getJobById(JobCalls.java:60) ~[classes/:?]
at com.collibra.marketplace.gcp.dataflow.JSONRequestProcessor.start(JSONRequestProcessor.java:93) ~[classes/:?]
at com.collibra.marketplace.gcp.dataflow.GCPWebhookController.batchSyncRequest(GCPWebhookController.java:44) ~[classes/:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.18.jar:5.3.18]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.18.jar:5.3.18]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.18.jar:5.3.18]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.18.jar:5.3.18]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.18.jar:5.3.18]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.18.jar:5.3.18]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) [spring-webmvc-5.3.18.jar:5.3.18]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) [spring-webmvc-5.3.18.jar:5.3.18]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.3.18.jar:5.3.18]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) [spring-webmvc-5.3.18.jar:5.3.18]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) [tomcat-embed-core-9.0.60.jar:4.0.FR]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.3.18.jar:5.3.18]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) [tomcat-embed-core-9.0.60.jar:4.0.FR]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.60.jar:9.0.60]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:178) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.18.jar:5.3.18]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.18.jar:5.3.18]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.18.jar:5.3.18]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) [spring-security-web-5.5.5.jar:5.5.5]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) [spring-web-5.3.18.jar:5.3.18]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) [spring-web-5.3.18.jar:5.3.18]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.3.18.jar:5.3.18]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.18.jar:5.3.18]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.3.18.jar:5.3.18]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.18.jar:5.3.18]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) [spring-boot-actuator-2.5.12.jar:2.5.12]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.18.jar:5.3.18]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.18.jar:5.3.18]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.18.jar:5.3.18]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-9.0.60.jar:9.0.60]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.60.jar:9.0.60]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.net.UnknownHostException: Failed to resolve ‘dataflow.googleapis.com’
at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1047) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1000) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:418) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsResolveContext.access$600(DnsResolveContext.java:66) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:467) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsQueryContext.tryFailure(DnsQueryContext.java:240) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.resolver.dns.DnsQueryContext$4.run(DnsQueryContext.java:192) ~[netty-resolver-dns-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) ~[netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.75.Final.jar:4.1.75.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]

Thanks & Regards,
Naveen

Like

Reply

2 years ago

0

springboot_team

368 Messages

2 years ago

Hi @naveen.jayaram,

Thanks for the information provided.

It seems that this error is related to the GCP Dataflow endpoint name not being resolved to its IP address (DNS issue). Can you please try again or check whether there might be any network-related issues? Thanks

 Caused by: java.net.UnknownHostException: Failed to resolve ‘dataflow.googleapis.com’

0

0

naveenjayaram

12 Messages

2 years ago

Thanks,

After adding location it worked fine
https://dataflow.googleapis.com/v1b3/projects// locations/

Is there any example for the dataflow job … we have deployed a sample dataflow Hello world but looks like we are getting a error as it expects Dataset details

It would be handy if you have some dataflow examples for which this microservice works

// Retrieves the table name from Display Data
DisplayData datasetNameDisplayData = jobDetails.getPipelineDescription().getDisplayDataByKey(“datasetName”);
if (datasetNameDisplayData == null) {
LOGGER.info(“No lineage will be added for Job having name ‘{}’”, jobDetails.getName());
return false;
}
final String tableName = datasetNameDisplayData.getStrValue();

	// Retrieves the input file pattern from Display Data
	DisplayData inputFilePatternDisplayData = jobDetails.getPipelineDescription()
			.getDisplayDataByKey("inputFilePattern");
	DisplayData inputFileDisplayData = jobDetails.getPipelineDescription().getDisplayDataByKey("inputFile");

Thanks & Regards,
Naveen

0

0

springboot_team

368 Messages

2 years ago

Hi @naveen.jayaram,

Thanks for the update.

Regarding GCP Dataflow, the job type would be a Streaming one, where under the job Pipeline Options, there should be the dataset Name specified. Unfortunately we do not have a sample Job available. Thanks

Type:
job-type

Dataset Name:
pipeline-options

0

0

naveenjayaram

12 Messages

2 years ago

Thanks @spring-team.collibra.com.

I was able to export the metadata of dataflow to collibra.

But i don’t see any transformation details exported as part of GCP Dataflow Steps .

In the lineage can we show the transformation logic as well ?
Does it support column level lineage.

Thanks & Regards,
Naveen

0

0

springboot_team

368 Messages

2 years ago

Hi @naveen.jayaram,

The GCP Dataflow integration imports both normal assets and relations, and technical linage through the use of the Collibra Lineage Harvester.

For the normal assets and relations part, a metamodel (Diagram view) similar to the one depicted in section “Sample Relationship Diagram” of the documentation should be made available.

Additionally, for technical lineage, in case there is the lineage information in the Job data, technical lineage would be imported. To be able to view this type of lineage, from the Collibra instance, you should open a Column asset and then select the “Technical Lineage” tab from the left-hand side menu.

To be able to confirm whether there was lineage that was processed by this integration, can you please forward the integration logs as a private message?

Also, you can check whether there is a log stating that “No lineage will be added for Job having name” which would indicate that no technical lineage would be imported. Thanks

0

0

springboot_team

368 Messages

2 years ago

Hi @naveen.jayaram,

Thanks for the information provided.

Can you please provide a fairly detailed explanation (and maybe including examples) of what is expected to be able to open an internal request?

Since this does not have an ETA and the Java source code is provided, you can also do the code modifications as required. Thanks

0

0

naveenjayaram

12 Messages

2 years ago

Hi @spring-team.collibra.com

Thanks for the information.

As part of our poc we took a example of dataflow provided here https://github.com/GoogleCloudPlatform/professional-services/tree/main/examples/dataflow-bigquery-transpose

and updated code to the parameters of dataflow example

like below
DisplayData datasetNameDisplayData = jobDetails.getPipelineDescription().getDisplayDataByKey(“inputTableSpec”);

Instead of datasetname

After running the dataflow integration library we got the lineage json, i have uploaded

[lineage.txt|attachment](upload://jon2DeGI9RoM6rbgW3xg9YBM0C8.txt) (2.2 KB)

I was expecting the library could parse each transformation step in the dataflow and in the source code of the lineage it could show up. But looks like it just extracts the dataflow step name and adds it in the source code.

Thanks & Regards,
Naveen

0

0

springboot_team

368 Messages

2 years ago

Hi @naveen.jayaram,

Thanks for the information provided.

0

0

Related Tags

dataflow

community-offering

Related Conversations

Loading...
Loading...

Was this helpful?