MD5 or SHA1 (or similar) encoding in Adrenaline Dataflows

jaeW_at_Onyx Budapest / Portland, OR πŸ”΄

related to

Sometimes users will use MD5 or SHA1 encoding to create a surrogate key for application in dimensional modelling-informed workflows.

in our use case we needed to create a dimensional table by perform a set of operations on a distinct set of values spread across several columns.

the dataset itself was 50M+ rows, the number of distinct values was around 20k. we need MD5 or SHA1 encoding to create a surrogate key between the transactions and the set of distinct values.

Currently this pipeline starts in Magic (generate the surrogate key) -> Adrenaline (perform the ETL -> Reprocess the data in Magic so we can Accumulate history using Partitions

Jae Wilson
Check out my πŸŽ₯ Domo Training YouTube Channel πŸ‘¨β€πŸ’»

**Say "Thanks" by clicking the ❀️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"
2 votes

Active Β· Last Updated