Magic ETL - User selected number of preview rows

Sometimes 100 rows is just not enough rows for a preview especially when filters and joins are beign used.

 

This can often result in previews showing no rows or no values from a joined table even when the data is there.

 

Please add the ability for the user to select the number of rows to load for the Magic ETL preview.

35
35 votes

· Last Updated

Comments

  • This would be very helpful for ETL transforms.  

     

    What I currently do as a work around is to filter my data set first.  For example, if I'm wanting to create a data flow joining our sales and opportunity data I would start by creating a new data set that was filtered before stepping into the ETL.  

     

    Rather than using our entire Sales data set and joining on our entire Opportunities data set, I would use Sales-TerritoryA and join to Opps-TerritoryA.

     

    By using a much smaller (filtered) data set to start with, I can usually test that the logic with the joins is working correctly.  Then I just swap out the input data sets for the full versions.

     

    Admittedly, this is a fair amount of work that could be avoided if this feature was built in.


    ______________________________________________________________________________________________
    “There is a superhero in all of us, we just need the courage to put on the cape.” -Superman
    ______________________________________________________________________________________________
  • Hey Scott,

     

    Great idea. Thanks for sharing that work-around.

     

  • Great idea! Thanks

  • mkozai
    mkozai ⚪️

    Agreed @ST_-Superman-_, I have resorted to doing that as well. Only problem is every time the dataflow runs, it adds extra load to the system.  Also if  you want different data, you have to edit or create a new dataset.  Really trying to avoid dataset proliferation and trying to be a responsible Domo user.  

     

    I would love to instead see this: https://dojo.domo.com/t5/Ideas-Exchange/Select-what-data-loads-from-data-set-when-editing-a-data-flow/idi-p/30112

    Summary: When selecting your inputs, it would be nice to write a where clause to define what data loads.  If you could control that, it would solve a lot of problems.   

  • @mkozai - Just to clarify:  I filter the data set down so that I can build and test the ETL using the preview function.  Once I am happy that the logic is built correctly I change the input data set back to the full data set.  The ETL tool can handle very large data sets, it is the preview tool that has trouble with it.  When you hit save and run, it should be able to handle your full data set.


    ______________________________________________________________________________________________
    “There is a superhero in all of us, we just need the courage to put on the cape.” -Superman
    ______________________________________________________________________________________________
  • mkozai
    mkozai ⚪️

    @ST_-Superman-_ - So then what do you do when you have millions of rows and the data you are trying to preview does not get loaded as part of the sample 400-500k data?  I think that was a key challenge for @dsutclif.  Only solution I konw of is to create a new DataSet that is < the preview limit that will allow you a reliable set of data to test with.  

  • Dani aka "Mr.Dojo"

    Dojo Admin
    **Say "Thanks" by clicking the "heart" in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
    **You can update your Dojo Community name and avatar by clicking on your avatar then the "My Profile" button.
This discussion has been closed.