DB2 z/os Catalog Ingestion
I would be interested if anyone has been successful ingesting using the DB2 for z/os into Collibra for just one database. We haven’t been able to figure out how to ingest by database. We’ve tried using specialRegisters, but haven’t yet been successful. The database property ($database) in the connection string points to the sub-system, not a particular database. Our databases are using the same schema name, so we can’t just sync by schema without getting metadata from multiple databases.
This is the connection string: jdbc:db2://$host:$port/$database
Former User
Posted 5 years ago · Edited 1 year ago·Last reply 5 years ago
6 comments
laurenzhiller
·5 years ago · EditedHi @cxlarsen.larsen,
did you find a solution for ingesting metadata by database in the meanwhile? We are facing exactly the same problem for z/OS in the moment.
bartvanderlocht
·5 years ago · EditedWe did a proof-of-concept using Catalog for our DB2-catalog, and found the same issue. (We ended with 125000 columns in one domain).
There isn’t an ‘out-of-the-box’ solution at the moment. We had a call with Peter Princen about this (and other issues), so they are aware of it
We decided a to build a custom solution, which isn’t so difficult (ODBC-connection, query on the DB2 catalog-tables, import in Collibra). You loose the profiling/sample data in that case, but that wasn’t an issue for us
Former User
OP5 years ago · EditedUnfortunately that wouldn’t help much, as each database/application uses the same naming convention. We have multiple databases/applications that use the same schema. First, we want one application per Collibra domain. Second, we wouldn’t be able to put together an exclusion list that would identify just one application.
arthurburkhardt
·5 years ago · EditedHave you contacted your customer account manager to set up a meeting with the data catalog product team on the “staging” repository?
I know nothing about it beyond having heard the concept. That would allow to import all the metadata into a staging repository before importing into the data catalog. Maybe that would be a solution to your problem.
Former User
OP5 years ago · EditedI’ve heard it is in the roadmap. Working with my rep to find out more.
arthurburkhardt
·5 years ago · EditedCan’t you use the “Tables excluded from registration” option?
https://productresources.collibra.com/docs/cloud-user/2020.11/#Catalog/RegisterDataSource/ta_register-data-source-own-driver.htm?Highlight=exclude
If you have 5000 tables in the same schema, you probably have some logic/prefix to identify tables per “database”.
Identify the minimum set of prefixes, and add them here, like so:
“TA*”, “TC*”, etc.