Alternatives to Mulesoft

With Collibra looking to move away from mulesoft, has anyone else found a suitable alternative? i.e. an approach that both allows Collibra to trigger api’s outside of Collibra as well as read incoming triggers from those external systems as well? So far everything we’re seeing only appears to be one-directional…

roryholmes1

Posted 5 years ago · Edited 1 year ago·Last reply 5 years ago

9 comments

constancebarton

·5 years ago · Edited

Rory,

When Collibra says it is “moving away from Mulesoft” I believe they mean they are no longer supporting the Mule-to-Collibra connector and bundled licensing of Mulesoft with Collibra. However, you could still have an independent instance of Mulesoft that you use to connect to Collibra using the REST API instead of the Collibra connector. This is being used by some companies that already use Mulesoft for other types of integrations.

arthurburkhardt

·5 years ago · Edited

@constance.barton while this is true, I respectfully think you are missing the point. You discuss INBOUND integration flows (i.e. an external application initiates and drives the integration) while this whole discussion is about OUTBOUND integration (Collibra as initiator). The only possibilities today are the collibra connect workflow delegate and open groovy language, but without any support for secret management. REST API does NOT provide any outbound integration capabilities.

Example of OUTBOUND integration is a flow to automate data access, which has long been advertised as an important piece of the “Data Intelligence” vision by Collibra: Cloud data governance — Collibra | Google Cloud Blog

When proper outbound integration capabilities are enabled, it will unlock new use cases:

Data access automation
Multi-application process orchestration
Event-driven architecture
Real-time lookup (instead of data synchronization)
…

pmahend3

·5 years ago · Edited

That is a fascinating approach Ahmad.

The answer to your original question Rory, an alternative for the current mule ‘gateway’ project that manages outgoing calls from workflows to mule projects has been in the works for a while. It’s a shame that I myself don’t have a lot of details to share on how this new approach is going to look like but it will certainly help with what you’re trying to do.

As far as what others I spoke to are using, I’ve seen a wide variety of languages and platforms but majorly Python, Java, C# while many others continue to use MuleESB to implement something along the lines Arthur mentioned.

Something else that’s been in the works is a set of reference SpringBoot integration projects just like we have Mule projects on the marketplace that will be posted on the same site that will help get things started if you’re a Java shop. I hope this helps.

Thanks

Former User

·5 years ago · Edited

Hi
we are develop a connector with microstrategy in springboot JAVA. This connector will be deploy in a kubernetes cluster on prem and use Collibra APIs in order to push the data in the Operational model after get this metadata from Microstrategy via API

We use cronjob to planned the task and up the information to Collibra
regards
Noe

Former User

·5 years ago · Edited

For one of my clients implementations we developed essentially a Collibra Middleware layer (serverless cloud function) that handled both events. The middleware had two endpoints: incoming and outgoing

Collibra workflow would trigger the outgoing endpoint and propagate the information to an events hub
Event notifier would trigger the inbound endpoint when there was a new record for Collibra

There were some drawbacks of course but provided an alternative to building more Mule flows.

arthurburkhardt

·5 years ago · Edited

That’s interesting Ahmad, are you able to share some additional insights?
Is the below diagram what you mean? (of course simplifying the “event hub” with consumers, etc.

Former User

·5 years ago · Edited

Almost spot on - here’s a quick mock-up:

All requests are authenticated and we used API keys along with some network access restrictions to make sure Collibra is the only one that can reach that outbound endpoint. (Inbound was much easier as we could integrate the function with a secrets manager). The Events Hub portion is quite simplified but that was the fundamental idea. This actually had a couple of benefits versus using Mule:

It was much more efficient (compute + cost)
It was far more scaleable as all that we need to do to onboard new systems is get them to send data over to events hub in the agreed upon format and it would instantly reach Collibra

davidwright

·5 years ago · Edited

We do something that is very similar. We have replaced a number of our connect integrations utilizing:

Output Module, Core v2 APIs, Import API used in replacing the Connect API
Lambas written in Python that do the actual transformation and integration with the above APIs
Orchestration utilizing SNS or timers, depending on the use case
Self service interaction with the services using a REST API that we created and deployed on a light node.js server

This model has worked well for us and allows for a lot of flexibility.

arthurburkhardt

·5 years ago · Edited

Yes indeed, which is a shame.
Nothing prevents you from using groovy to call external APIs, but with poor code coverage, poor deployment and poor secret management, it seems challenging to use it.

The only possibility at this time beyond collibra connect is polling collibra at regular intervals.