4 Messages
How to convert epoch column to date column in DQ?
Use Case: Customer is reading data from a JSON file with a column that has a epoch value. Need to convert that column/value to a date column to run DQ checks against it.
Create Dataset
Click Settings Configuration
Turn On
Click Add on bottom
Select Custom, Select Column, Uncheck Override, Add Name for New Column
Add Expression
date_add(‘1970-01-01’, cast((epoch_time/86400) as int)) as epoch_time_convertcolumn
Click Build Model, Save and Run, Run Job
Verify New Column Created
Note: New Column can be used in DQ Checks as Date Column
If you have use cases for Date Partitions (Incremental Loads), as shown in the “Range WHERE” below, you will have to transform the date column in the where clause for the runtime evaluation to define the scope.
Select Epoch_Time Date Column in where
Uncheck “is Date”
Click Transform
Add Expression
date_add(‘1970-01-01’, cast((epoch_time/86400) as int))
Click Green Check Box
Query is updated
Alternatively, Spark 3.1 and above will support the timestamp_seconds function to accomplish this UC:
https://spark.apache.org/docs/latest/api/sql/#timestamp_seconds
No Responses!