Compile all the transforms in the the dataflow before running the dataflow

Issue:

Some of the SQL/PostgreSQL dataflow has many transforms in it and takes around 2 hours to run due to the volume of data. When there is a technical error/syntax error in a transform or output (as simple as missing alias) the dataflow runs for a long time (at times more than an hour) before the error is triggered! This wastes a lot of time for the developer

 

My understanding:

I am not very sure why DOMO takes so much time to identify the technical error. My guess is that DOMO does not compile all the transforms before running the dataflow. While, it compiles and runs the transforms one by one and hence it identifies the syntax error only at the runtime.

 

Suggestion:

If my understanding is correct, DOMO can compile all the transforms in the the dataflow before running the dataflow. I hope it is possible.

 

On related note, at times while debugging dataflows, we need to run all the previous transforms manually before running a transform which is at the end. This is very very cumbusome! DOMO can implicitly compile/run the previous transforms. PS: I came across this idea somewhere before and I am not able to find and tag it. Kudos for that suggestion.

 

Thank you! 

 

3
3 votes

· Last Updated

This discussion has been closed.