LinkID for Shapes
Greetings all, is there any linkid or list of corresponding rows detected by shapes?
Tried looking in the metastore, is there a table? – saw link_id for ‘datashape’ table but it does not seem to show more than one.
Tried configuring page from admin, is there a setting? – saw shape counts all > 1 in configuration but still only 1 row in shapes
Tried export, is there an export function? – export was only giving me 1 shape.
My usecase -> Trying to avoid writing a rule where character count > 2.
I showed shapes and saw not only LEN>2 (XXX) but also an erroneous character %X. Pointed out how the rule would have missed %X which was clearly wrong.
But how do we find those to correct the issue, we only see 1 row in shapes…
Concerned that while not having to create a SQL rule would eventually result in having to write the SQL to find those bad rows.
ericgerstner
OP4 years ago · EditedFrom Vadim
Several of the activities generate findings that are many to one with respect to rows where they occur. Shapes is one, Categorical outliers is another, Anti-Patterns is a third. We do not capture the full list of examples because that could produce a result set approaching the size of the dataset itself. We are able to process data in the Terabyte scale because we do not simply look at each record one by one and put aside the breaks. We use relatively sophisticated techniques that find the issues in aggregate. The trade off to this approach is that for some of the activities, we do not actually have the example row “in hand” (or if we do, we can’t risk collecting and storing all of them because of possible resource requirements).
In those cases, the customer should be able to write relatively simple SQL against the actual data source using the finding as the predicate to find the full list of rows. Realistically speaking, if they had a list of 1M rows of Shape examples, would they go and look at each individual row manually? The actual path to remediation would be to write a query that finds those shapes and then preforms a transform on those rows in that same SQL statement.