Magic ETL trigger timing - input datasets of varied size
I think I just discovered that if a magic ETL is configured to run "only when datasets are updated", and has multiple input datasets, it will start running as soon as the smallest/quicket input dataset has been updated. If a larger/slower input dataset finishes updating before the ETL finishes running, the ETL will not run again.
Timeline of a concrete example that happened this morning for an ETL that has six input datasets. I've numbered them Input 1 through Input 6 in order of size. Input 1 has 39 rows and Input 6 has over a milion rows:
05:00:00 - Inputs 1-6 start updating
05:00:17 - Input 1 finishes updating, ETL starts running
[various times] - Inputs 2-5 finish updating
05:03:04 - Input 6 finishes updating
05:03:58 - ETL finishes running
Unfortunately the new data in Input 6 was not included in the ETL run, and the ETL was not triggered to run again. This makes sense in a way but also seems like a bug.
I'm going to report it as a bug to Domo Support but meanwhile was wondering if anyone else has experienced this or has any tips for simple/reliable workarounds?