Collibra Community

Edge

Created Nov 18, 2023

9 members

19 discussions

The Edge topic in the Collibra Community is dedicated to supporting users working with Collibra Edge, the platform’s integration layer for harvesting metadata and lineage. It's a space to explore setup guidance, troubleshoot scaling issues, manage secure credential integration, and share best practices for running Edge in environments like Kubernetes and cloud platforms.

andrewsouthcombe
2 months ago
Scaling Edge on AKS
We have Edge on K3s at present and as we have hundreds of databases to sync we've hit capacity limits. We have setup an Azure Kubernetes Cluster in our dev environment and migrated our dev Edge site. The nodepool has autoscaling enabled, but Edge never seems to prompt it to scale. Even if I submit 100 jobs at once it sticks to the initial node capacity. Is there any way to change this behaviour? Collibra's Edge FAQ indicates Edge sites are horizontally scalable on AKS but doesn't give any instructions to make this happen.

0
andrewsouthcombe
3 months ago
Migrating Edge from K3S to AKS with vault integration
We currently have Edge installed in Azure VMs using K3S. We are using Azure Key Vault for secret management with managed identity authentication. We plan to migrate to Azure Kubernetes Service but the documentation doesn't really cover this type of migration, or what happens with vaults. I believe the migration process is: Backup your current edge site and save the file uninstall the current edge site Install on AKS supplying the backup file I have two questions related to this: What happens with the key vault in this scenario? Does installing from the backup recreate it as-is or do I need to recreate the integration and update the edge connections? Is managed identity authentication supported when Edge is running on AKS? The docs state: "In order to use the Managed Identity assigned to Azure Virtual Machine authentication method, you must install your Edge site inside of the Azure VM." but this is ambiguous. Does this mean it only works if you have on K3S on an azure VM and that AKS is not supported? Or does it mean the AKS nodes have to be Azure VMs? https://productresources.collibra.com/docs/collibra/latest/Content/Edge/EdgeSecurity/ta_integrate-edge-with-vault.htm

4
Gian-Luca Matsuda
5 months ago
Edge Site Offline - Read Only
Hello, I saw a post that asked about this but it was over a year old, so I thought I'd ask now. Both my dev and prod environments have gone offline and I can't seem to get them back online. We've restarted the environments for both in the admin consoles and rebooted the servers. Do I really need to delete and reinstall my edge sites altogether, or is there something else that can be done? I have a support ticket to Collibra about this, just thought I'd ask the community in the meantime. The last time this happened it was only a day or so until an upgrade finished.

2
prernaSin
6 months ago
Technical lineage through IICS
Hi All, Has anyone tried data lineage through IICS. I am trying to create a connection, but it is asking me IICS userid and password. People in my organization uses single sign on for logging to IICS. Is there any way we can do this without having password? Looking for suggestions.

5
tomwilliams1
9 months ago
How can I configure BigQuery Technical Lineage "other query" when I am using INFORMATION_SCHEMA.JOBS instead of ROUTINES/PARAMETERS
hello, I am configuring a BigQuery Technical Lineage, but I have realized that in the section "other queries" the tables from information schema used is not the same from the default collibra query : SELECTr.routine_name,CONCAT('CREATE ', r.routine_type, ' `', r.routine_schema, '.', r.routine_name, '`(',ARRAY_TO_STRING(ARRAY_AGG(CONCAT(p.parameter_name, ' ', p.data_type) ORDER BY p.ordinal_position), ', '),') ',CASE WHEN r.routine_type = 'FUNCTION'THEN CONCAT('AS (', r.routine_definition, ')')ELSE r.routine_definitionEND) as sourceCode,r.routine_type as groupName,r.routine_schema as schemaNameFROM `##PROJECT_ID##`.`##DSNAME##`.`INFORMATION_SCHEMA.ROUTINES` rJOIN `##PROJECT_ID##`.`##DSNAME##`.`INFORMATION_SCHEMA.PARAMETERS` pUSING (specific_catalog, specific_schema, specific_name)WHERE r.routine_body = 'SQL'AND p.is_result = 'NO'GROUP BY r.routine_schema, r.routine_name, r.routine_type, r.routine_definitionUNION ALLSELECTr.routine_name,CONCAT('CREATE ', r.routine_type, ' `', r.routine_schema, '.', r.routine_name, '`(',ARRAY_TO_STRING(ARRAY_AGG(CONCAT(p.parameter_name, ' ', p.data_type) ORDER BY p.ordinal_position), ', '),') ',CONCAT('LANGUAGE js AS """', r.routine_definition, '"""')) as sourceCode,r.routine_type as groupName,r.routine_schema as schemaNameFROM `##PROJECT_ID##`.`##DSNAME##`.`INFORMATION_SCHEMA.ROUTINES` rJOIN `##PROJECT_ID##`.`##DSNAME##`.`INFORMATION_SCHEMA.PARAMETERS` pUSING (specific_catalog, specific_schema, specific_name)WHERE r.external_language = 'JAVASCRIPT'AND p.is_result = 'NO'GROUP BY r.routine_schema, r.routine_name, r.routine_type, r.routine_definition As what i want to use is as i have to use INFORMATION_SCHEMA.JOBS or INFORMATION_SCHEMA.JOBS. After checking an example I have seen that the source code extracted in the default query is javascript but i have dbt / sql information in my source code. Has anyone had this situation or has knowledge that could help me? any help is welcome, thank you in advance! Sincerely, Williams Tom

1
davidwright
11 months ago
Please Vote: Ideation to fix a technical lineage issue
Fellow Data Citizens, we are looking for help in moving forward an ideation. We have noticed an issue with technical lineage where relations are being deleted by the lineage harvesting process when the relationship type is the same as used by the lineage harvester. The Issue: When ingesting lineage via the lineage harvester, any relation created via the UI, import file or via API is deleted by the lineage harvester if it is the same relationship type as one harvested by the lineage harvester, By example, your team imports source/target relations to document an ETL process from a lineage harvested database table to some target. The next time that the lineage harvester runs, that imported ETL lineage is deleted. This means that your team needs to replace that lineage whenever the harvester is run which is an onerous and possibly costly process. The product team offered a workaround by which the lineage is loaded via custom technical lineage. This is complex given the lack of an API as well as inflexible for those that may want to use the UI or upload file capability. The Solution: Our team has proposed a solution to an ideation ( https://productresources.collibra.com/ideation-platform/?id=DCC-I-196 ) to solve the issue. Our proposed solution would be an enhancement to limit the modification or deletion of relationships by technical lineage to the relations created by technical lineage. There is a similar ideation out there that was recently posted as well: https://ideas.collibra.com/ideas/CDL-I-8 Given the existence of another ideation and comments on our ideation, we would anticipate that many customers are facing the same issue. Please help us move this issue up the priority list for the Edge team by voting for both ideations. In that way, we can show the impact of this issue and need for its resolution. Thank you! Dave Wright

0
user_a0c573
1 year ago
Can I reduce the Collibra DIC Edge spec below the minimum recommendation
Greetings, I am reaching out regarding the sizing of the Edge Site within a Google Compute Engine. Referring to the URL provided ( https://productresources.collibra.com/docs/collibra/latest/Content/Edge/EdgeSitesInstallation/ref_edge-site-system-requirements.htm?Highlight=edge%20%20system%20requirements ), the minimum hardware requirement for Collibra Data Intelligence Cloud is stated as a 16-core CPU with x86 architecture and 64 GB of memory. However, considering that the current utilization is less than 2%, with expectations of enhancement later on, my client is inquiring about the possibility of downsizing to a 4 CPU and 16 GB RAM configuration, essentially reducing to 1/4th of the current specifications. Additionally, there's a plan to resize the VM as the load increases in the future. My query pertains to the feasibility of this option and whether it might cause any issues with the existing installation.

1
celenemcfall
1 year ago
Reduce security risks by enabling Edge to use your Vault credentials
Edge now supports integrating with your Vault applications and services for easier credential access and management policy enforcement. By removing sensitive information from your edge site, you reduce security risks. Edge supports integration with CyberArk, HashiCorp, Azure Key Vault, AWS Secrets Manager, and Google Secret Manager. To learn more and get started, read our documentation .

0
darnelwhite
1 year ago
Preserve asset history during Edge Migration
I’m looking for information around how to preserve asset history information while migrating to Edge. We plan to begin our migration later this year, but our main concern is losing historical information relating to assets especially around when assets received an updated status of ‘Accepted’. Has anyone had this issue or is there a way to preserve our asset history as we migrate to Edge?

0
glendobson
1 year ago
Salesforce Lineage Issues
I have been stuck for a number of years now in trying to find a solution or workaround to allow me to gather lineage across my core Salesforce platform. We utilise Edge in our environment and know that the integration at present isn’t very supportive of the lineage harvesting from Salesforce but I am hoping given the wide use of Salesforce that someone has possibly come up with a creative solution for this ongoing problem!

0

Scaling Edge on AKS

Migrating Edge from K3S to AKS with vault integration

Edge Site Offline - Read Only

Technical lineage through IICS

How can I configure BigQuery Technical Lineage "other query" when I am using INFORMATION_SCHEMA.JOBS instead of ROUTINES/PARAMETERS

Please Vote: Ideation to fix a technical lineage issue

Can I reduce the Collibra DIC Edge spec below the minimum recommendation

Reduce security risks by enabling Edge to use your Vault credentials

Preserve asset history during Edge Migration

Salesforce Lineage Issues