Magic ETL Errors and Sample Data

Wouldn't it be great if there was an error on a Dataflow (Magic ETL or MySQL) that the sample dataset, by default included the failing record.  Additionally in the Magic ETL, if the record was included it would be great to see the failure point in the preview (which it should do already).  Those things together would prevent a lot of trouble shooting time.

 

OR

 

Give me the option to run the preview with ALL records.  (With current functionaltiy I could then set up a row filter to give me the test case I'm focusing on so it still wouldn't be a huge performance problem.)

 

 

 

Use Case:

On occasion I have some complicated ETL to do.  There is a sample below.  When I run into issues with the dataflow, it can be frustrating to have to go out and create another dataflow to limit my records down to an appropriate sample dataset, change my original Dataflow to use the new testing dataset, to deal with the issue(s).  Then once corrections have been made I change back to the original dataset I need for production use -Test-  and repeat if necessary.  Bleh!

And by "-Test-" I mean save and run and hope nothing errors.  If it does, read/record the error and open the edit the Dataflow again.  Then in the editor, try to determine which dataset we behind the failure.

 

*Depending on the number of input datasets and level of complication, this process can be length, just in these setup steps.

ComplicatedDataflow.PNG

1
1 votes

· Last Updated

This discussion has been closed.