Reporting Data Layer

Question

Please enter here remarks, questions or feedback regarding the Marketplace listing Reporting Data Layer: https://marketplace.collibra.com/listings/reporting-data-layer/

akashpandey · Answer

Hi,

This is super useful and has helped to create report on the comprehensive dashboard. But the snapshot is too huge to be refreshed on a scheduled basis.

I suppose there is a plugin : https://marketplace.collibra.com/listings/output-module-to-parquet/

This would help create the same parquet files from Output Module. But unable to run this package due to integration library version compatibility issues.

I have raised the issue in the below post.
https://community.collibra.com/conversations/insights-data-access/output-module-to-parquet/66ebee39e15cf8648e815956

Kindly help if anyone has successfully done this integration.

tomaskissik · Answer

Hi Tom,
regarding your first point, we have written a simple Glue python shell job that in few lines retrieves the data from API using request package, it extracts the zip into a s3 bucket using zipfile and boto3 and using boto3, it also generates a view which have a query that uses only the latest snapshot/partition . We also use authlib library for jwt authentication.

The Job runs daily and provides convenient backup of the data and good base for analytics.

The new python 3.9 shell jobs are very easy to use including packages installation https://aws.amazon.com/blogs/big-data/aws-glue-python-shell-now-supports-python-3-9-with-a-flexible-pre-loaded-environment-and-support-to-install-additional-libraries/

tomkuppens1 · Answer

Hope this thread is still active.
We are a big user and fan of the RDL! Some questions and thoughts:

We are looking into moving our RDL solution to the cloud.; there are many routes we could take, but just thinking outside the box here: could it be possible to get read only access to the cloud filesystem directly (I presume this is a S3 bucket). This would make the transfer to our cloud storage much more convenient and would open so many possibilities.
the RDL snapshot is ready at about 8:00AM local time which is rather late. We export and import in our reporting solution daily and because the attribute object is getting really big, and the processing takes more and more time: making the new RDL data only available at noon in our RDL solution. Processing also only can start at 8:00 which puts a burden on our processing system. Nightly processing would be more ideal. Any chance we could shift the start of the snapshot creation to an earlier time so that it’s ready at let’s say 3 a 4AM-ish local time.

Thanks!
Tom Kuppens

tomaskissik · Answer

Hi, please where it is possible to download DDL scripts for Reporting Layer version 1.4 that is documented at https://productresources.collibra.com/docs/reporting/1.4/#Reporting/co_reporting-intro.htm ?

This Marketplace listing allows to download version 1.3 only that doesn’t have DDL for complex relations.
Thank you.

tomaskissik · Answer

Hi, please where it is possible to download DDL scripts for Reporting Layer version 1.4 that is documented at https://productresources.collibra.com/docs/reporting/1.4/#Reporting/co_reporting-intro.htm ?

This Marketplace listing allows to download version 1.3 only that doesn’t have DDL for complex relations.
Thank you.

1	Grant Rollerson SUNCORP CORPORATE SERVICES PTY LTD
2	Tom Friesen Algonquin College
3	SB S Benson Plus3 IT
4	S stefan_busch
5	Brian Lloyd USA TODAY Co.
6	D daverichard

Reporting Data Layer

Thanks! Tom Kuppens

Thanks!
Tom Kuppens