Migration feature and assets
Summary: Lance Harlan raised a question about migrating Dev catalogs to Prod using the migration tool in the Collibra platform. The issue was that while they successfully imported users, groups, and workflows, the dictionaries weren't included, despite using compatible environment versions: DEV (2024.01) and PROD (2023.11.4). rosariocarbone responded, clarifying that the migration tool primarily targets the operating model, not data like dictionaries. They advised against using data backup and restore between dev and prod environments, noting that such processes are for disaster recovery, with potential implications on users and SSO. Instead, recommended methods include connecting directly to prod data sources in prod environments, manually exporting/importing domain views, and experimenting with global views for asset migration.
Good morning Collibra friends,
We’re trying to use the migration tool under settings to migrate our Dev catalogs to Prod. We’re working with a clean slate in Prod. We were successful in importing the skeleton of our Dev environment, users, groups, and workflows, but none of the actual dictionaries came with it. We see the warning in the documentation, but this seems to only be an issue with environments up to 2021.07. Our Dev and Prod are not on the same (DEV: 2024.01 and PROD: 2023.11.4) but these are later than the warning. Do these environments have to be exactly the same for the dictionaries to also be imported in the migration, or am I not understanding the purpose of this migration tool?
rosariocarbone
·2 years ago · EditedHi @lharlan.trustmark.com
The migration feature exclusively executes the migration of the operating model, including roles, workflows, assignments, and views. It has not the capability to migrate assets, attributes and relations (the data). Migrating data from one environment to another can be done through data backup and restore processes. However, it is important to highlight that this method is not recommended for data migration, particularly from development / test to production environments.
Such a migration using backup and restore would include all the data in Collibra, including users, with potential implications for Single Sign-On (SSO) and other implications. It is common practice for customers to connect development/test data sources in their development/test Collibra environments rather than production data sources. In summary, using backup and restore processes for migration between non-production and production environments is discouraged, as these processes are intended primarily for disaster recovery. However, some customers use this approach to migrate from prod to dev/test environments when they are planning dev/test work and prefer to work on a non-prod environment that mirrors the prod environment.
The recommended approach is to connect and ingest metadata directly from prod data sources in production environments. As for other assets, such as for example glossaries containing business terms, the domain view export/import features can be utilised from development to production after aligning the operating model.
An alternative technique involves:
This approach also facilitates the creation or update of communities and domains in the production environment. It also allows for the migration of assets using same identifiers, if required. Attention should be paid regarding the characteristics exported in the files. Modifications directly in the files exported can be made to achieve the desired outcome upon import into the production environment.
Domains and global views additionally support the export/import of relations between assets.
I recommend experimenting and testing with the view techniques and determine the most suitable approach for your needs. Document each step, and once satisfied with the process, execute the migration.
I hope this can help.
Regards
Rosario Carbone
Brittany Crews
·11 months agoThe recommended approach is extremely tedious, time consuming, and allows for the potential of user error. Has Collibra developed a more efficient process for this over the last year? This inefficient process honestly makes me rethink using Collibra as my company's tool. I have never heard of a company not having a support team who can do system syncing and instead makes that the responsibility of the customer to handle on their own....