Jobserver Polling or Pushing

Hi all

Which option - 1 or 2 - has worked better for you? appreciate the insights, challenges faced.

Option 2 always creates an inbound connection to the client network (assuming job server is on-prem, and hence might not be preferred.

noor

Posted 5 years ago · Edited 2 years ago·Last reply 5 years ago

7 comments

davidwright

·5 years ago · Edited

Whatever you do, make sure that you consider how services will change as the move to the Edge server occurs. My guess is that polling and no inbound will be the way that Edge will work. All of our test of the jobserver have been utilizing the poling option due to security limitations.

christopherthullen1

·5 years ago · Edited

Here are my two cents about what I understand:

Up to a year or so ago, the Collibra server would initiate the connection with the on-premise jobserver. This required configuring the firewalls to let the connection go through.

Since then, the preferred approach is to configure the job server to ping the outside Collibra server every few minutes, or more often if necessary. The outside server then responds to the ping with an ingestion task when such a task is requested. The connection is initiated from inside the company’s firewall, which is much easier to configure. In fact, many networks do not need a configuration in this case. With this configuration, the Collibra server cannot start an ingestion job immediately, but has to wait for the next out-going ping.

A coaching session helped me set things up. It was easy to do and only required one hour.

Former User

·5 years ago · Edited

Agree with Arthur. Coaching services will give you a very good view of your options.

For us, the polling option behind a reverse proxy made a lot of sense. Argument was that we were not ok with a third party public cloud provider initiating any connections that eventually get made to data sources that range in confidentiality rating.

Of course, every use case different. Have a chat and assess your options

noor

OP5 years ago · Edited

Yes Arthur. Got the contact today from my alliances manager. Need to initiate the mail…thanks…

arthurburkhardt

·5 years ago · Edited

@noor.shaik it would probably be very valuable for you to jump onto a coaching session. All those “getting started” topics are exactly their area of expertise.

arthurburkhardt

·5 years ago · Edited

https://productresources.collibra.com/docs/cloud-console/latest/#Console/Infrastructure/JobserverService/DGCCloudToJS/to_dgc-cloud-to-jobserver.htm

In default installations, a Jobserver is installed on-premises and a Collibra Data Intelligence Cloud sends ingestion and profiling jobs to it. However, we highly recommend to reverse this communication, so that the on-premises Jobserver polls for jobs. This is often required for security reasons.

noor

OP5 years ago · Edited

That’s exactly the same I was referring to in the screenshot I attached. To reverse the communication, there are many bunch of steps that need to be carried out. Instead, I want the traditional way of doing things to be in place (simple but elegant too i.e. I will trigger the ingestion jobs via the command line utilities being called from a scheduler like Control-M). The security concerns can be addressed by installing job server also in cloud, and having private links concept etc while reaching the on-prem data sources.