Queuing of dataflow in case of parallel trigger request (manual and automatic)
Problem statement:
Let us consider a daily once scheduled dataflow task (on datasource refresh) that takes over 2 hr to complete.
There are instances when I had to force run the dataflow, and during this time when there is a scheduled run, it will be ignored. Although we can say that the developer needs to be diligent in handling such scenarios, when there is a bigger team such issues can potentially happen. This can lead to data loss (esp because the dataflow is triggered after a fresh data is loaded by datasource). It would serve better if this aborted automatic process is logged in the history of dataflow, but it is not tracked currently.
Suggestions:
Like we have in many ETL tools, we can queue the dataflow if it is triggered when the run is in progress. And in the "history" tab we can have a feature to track/monitor the queuing. If needed we can restrict the number of ques there can be for the dataflow (to one - the latest trigger)
Thanks!
Categories
- 10.8K All Categories
- 3 Private Company Board
- 1 APAC User Group
- 12 Welcome
- 38 Domo News
- 9.7K Using Domo
- 1.9K Dataflows
- 2.5K Card Building
- 2.2K Ideas Exchange
- 1.2K Connectors
- 343 Workbench
- 258 Domo Best Practices
- 11 Domo Certification
- 464 Domo Developer
- 50 Domo Everywhere
- 104 Apps
- 714 New to Domo
- 85 Dojo
- Domopalooza
- 1.1K 日本支部
- 4 道場-日本支部へようこそ
- 26 お知らせ
- 64 Kowaza
- 299 仲間に相談
- 653 ひらめき共有