Magic ETL - Using Group by - run time impact question
Wondering what is the best practice to reduce overall run time of a ETL dataflow...
I've got 5 input datasets totalling about 14M rows of data that are ultimately appended together in the final result.
Is it better to use group by in the magic ETL for each dataset (5 steps) or one consolidated step after the appends?
Also, are there certain items in Magic ETL that are run time bandits?
Comments
-
Is anyone able to help out with this request?
0 -
Hello @swagner,
It is hard to determine what might be causing long run times in an ETL without looking at the ETL in specific.
Generally, it is most efficient to use a select columns tile and select only the columns you need.
Next would be to filter your data down to only the rows you need.
After filtering your data to only the columns and rows that you need grouping your data will help reduce the size.
In regards to your group by question. Normally it will not make a difference between doing the group by's before the append vs after. If you can provide a screenshot of your ETL I can look to see if there are any other steps that we might be able to optimize.**Say “Thanks" by clicking the thumbs up in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1
Categories
- 10.8K All Categories
- 3 Private Company Board
- 1 APAC User Group
- 12 Welcome
- 38 Domo News
- 9.7K Using Domo
- 1.9K Dataflows
- 2.5K Card Building
- 2.2K Ideas Exchange
- 1.2K Connectors
- 343 Workbench
- 258 Domo Best Practices
- 11 Domo Certification
- 464 Domo Developer
- 50 Domo Everywhere
- 104 Apps
- 714 New to Domo
- 85 Dojo
- Domopalooza
- 1.1K 日本支部
- 4 道場-日本支部へようこそ
- 26 お知らせ
- 64 Kowaza
- 299 仲間に相談
- 653 ひらめき共有