Technical: Processing Volume / Hardware Recommendations
Processing Volume
Q: Do You Scan Every Row? How To Handle ~2TB?
A: Yes, We Define Agent To Send Compute; Work With Available Hardware; Could Be Databricks, Spark Cluster, Other Endpoint
Q: If Customers Spin Up 1TB RAM, Do You Often Run Into Tables Larger Than That?
A: Typically If They Have Larger Datasets, They Have Something To Point To
Q: Is There Plan To Have Number Of Agents And Load Balancers For Large Jobs?
A: Customers Can Carve Multiple Agents To Send To HDFS-Backed; Load Balancer Is Future Roadmap
Q: Controls In Place For SQL Pass Through In Case It Gets Unwieldy?
A: No, We Don’t Parse Or Analyze; We Try To Keep Flexible As Long As Valid Syntax
Q: Edge Node in your architecture diagram in documentation - is this the agent?
A: Yes, This Is Edge Node On Spark Cluster
Q: Does the tool have a random sampling capability?
A: Filters and limits are available
Q: Can I Run 2 Jobs Simultaneously?
A: Yes, Kubernetes / yarn resource manager controls jobs, Collibra DQ helps size jobs.
Q: Largest data Collibra DQ has processed?
A: 4B rows / 3-4TB, Run Once To Form Baseline, Then Additional Data, Use Filter Conditions; We Have No Limitations If Customer Has Compute
Q: Whether It’s View Or Complex Query, Running On DB Or Spark?
A: Parallel JDBC Will Pull Data Into Memory. Customers Who Have Analytics Will Typically Have Resources Available. We Also Have Pushdown Option.
Q: Can Collibra DQ Scale?
A: Yes, We Use Spark Which Can Scale. We Also Offer Containers And Kubernetes Deployments
Q: Can Collibra DQ Be Fed Through Subscription Mechanisms Using Topic Or Message Queue?
A: Yes We Support Kafka. Our API Output Can Be Sent To Message Queues
Hardware / Server Recommendations
Q: How to determine optimal compute requirements based on concurrency and volume?
A: We provide sizing guidance based on peak volumes and concurrency requirements. We use daily ingestion rate to determine this based on either a) GBs or b) rows x columns. We can refer to your database internals at the storage layer or use the Estimator spreadsheet to determine these specs.
Q. How much compute is required for only hosting?
A: 4-8gb ram and 64 gb storage. You do not need a lot for hosting. The web app does not consume many resources and the only consideration is around logging and how many concurrent users will be accessing the application.
Q. How much compute is required for pipeline (Rules and profiles) development and reporting?
A: The engine (runtime) is separate from the hosting (web) portion of the application. These resources will be supplied in terms of executors, memory, and cores for each job.
Q. What do we recommend compute requirements specific to Profiling given data accumulates and it will not be a snap shot?
A: Profiling can be customized to scan a portion of the data (subset) or the entire population of data. Each profile is tracked with a run id (date) to describe changes over time.
Q. What is suggested infrastructure (compute) independent of Profiling and Rules? The reason for this request is because profiling will be cumulative volume that increases every day (Need auto scaling support), however rules can be batched on fixed cluster size to handle concurrency, volume and demand (Separate EMR can be used just for Rules management). A separate server (native) can be provisioned for software hosting and normal processes.
A: We would suggest two runtimes (two different agent definitions). We would point 1 to EMR and the other to an elastic compute pool utilizing the Spark Kubernetes Operator (autoscaling based on your compute pool definition). This would be the ideal organization for having 2 separate runtimes for the different activities. The K8 deployment would be installed using Helm commands and EKS.
Q. What is suggested when mixing and matching different server configurations based on concurrency and pipeline processing volumes? Some Rules might require latent tables to be stored in cache before they can be further processed. For such types, special servers can be allotted that can support better caching and compute.
A: What are you doing today to cache latent tables? If you’re doing this already, ideally we would fold into an existing process and runtime layer.
Q: What Is Minimum Systems Requirement?
A: Redhat Or CentOS 7 Or 8 With 4GB of Ram
