Run all the steps in a dataflow (to a point)

I have a few fairly decently sized dataflows that have > 8 steps.

 

Sometimes I will come in and need to make a change to step 8, or step 1, or whatever. But I want to preview the changes in the final output set to make sure everything is good to go.


Before I can do that I have to go into each step and click "Run SQL". That isn't so bad when I'm building the dataflow. I just Run SQL and then go to the next step. But when I'm coming back in to work on something, it is a bet annoying. "Wait, did I run that one yet?"

 

It would be cool if I could maybe have checkmarks next to each step and a "Run SQL" button (and a check all box for that)? Or maybe a "run all previous SQL steps" option when I'm on a step?

 

There just should be a faster way to step through a dataflow. At least, I think there should be

83
83 votes

· Last Updated

«1

Comments

  • "I have a few fairly decently sized dataflows that have > 8 steps."

     

    I have an idea for myself: proofread before I post. Smiley Frustrated

  • Agreed!  This needs more votes...

  • Thank you for submitting this idea.  @mattchandler is our PM for this area and will respond.

  • Thanks @product_John  - every time I come back to this post though I cringe at my typo so don't hold that against me ?

  • Thanks for the feedback @SeanPT! This would be a great improvement. I'll add an item to our roadmap to address this issue when we finish up some of our current large initiatives.

     

    I also edited your post to fix the typo you commented about—no need for you to cringe next time you come to the page!

    Best,
    Matt Chandler
    Domo
  • Geh, this is not getting the traction I would expect. This is something that most people are not going to think of, instead they will just live with it. This sucks that it has to go through the queue. Can't yall ask your own programers and see what they vote on this?? I bet they would feel the same as we do. This needs to be put into action! 

    Eric Peterson
    [email protected]
    www.walkeredison.com
  • I've been getting a few likes on this recently and I totally still want this feature.

    But I can tell you that since I wrote this post back in the old days of 2015 I've gotten better at combining transforms. For example, I may have put some unions in one transform and then some joins in the next. Now I'll do that all in one.  It makes the steps a little more muddled but not too bad really.

     

    So this limitation forced me to write some tighter code and that's not a bad thing. Maybe that mindset can help those who are seeing this and saying "yeah, me too!"

  • I have been advised that maybe if I just consolidated joins I would not need this sort of dataset. However, I have tried that. I am generating a full on financial flow that requires extensive data. I am talking over 100 transforms in one data set. I tried consolidating down only to find that the speed of the dataset is hampered severelly. I have optimized the datasets a lot, 15 minute run times average. Am I just unusual with the level of complexity I am doing here or do others also deal with larger sets on a regular basis?

    Eric Peterson
    [email protected]
    www.walkeredison.com
  • itwalkerdison - Are you using redshift or mysql? Even on my largest datasets with lots of joins and unions I generally don't hurt for processing time in redshift. What hurts is data transfer.

     

    Anyway, I don't think you are alone but you may be up there with 100 transforms. At that stage I'd make changes and just hope it runs in the end. The UI is definately not 100 transforms friendly.

  • Sean:

     

    I am running in MySQL. Yes I have heard of the problem being data transfer with redshift vs Mysql being with running code. To clarify I only mentioned the optimization so someone did not think that I had all these tansforms because I did not know how to optimize ? I do know how to optimize for certain. 

     

    So, I am begging Domo to please consider this request SeanPT put in a while ago, and hope there be enough people to vote for this. 

    Eric Peterson
    [email protected]
    www.walkeredison.com
  • I know the optimization pressure oh so well so I appreciate it when others feel that same crunch.

     

    I know I said a few replies ago that I had changed some of my methodology so this wasn't a big issue. Yeah, that came back to bite me yesterday as I had to re-work one of our monsters.

     

     

  • Robot Sad Make that over 200 transforms at this point . . . Is there any traction on this one at all?

    Eric Peterson
    [email protected]
    www.walkeredison.com
  • @btm can we get a status update on this one?

    Thanks!

    Dani

  • btm
    btm

    domo

    💎

    Adding @StevenC, who is our PM over Domo's Data processing tools and will be able to have the best sense of the status of this idea.  Thanks!

  • Thanks @btm . I was working with our developers today and they were going through their code in visual studio and I was reminded that running to a point has been a standard thing in IDEs for a long time.

     

    I was talking to our DOMO Master and he informed me that he would absolutely love this feature so even as we grow at Freddy's we continue to look for this addition. So when it reaches beta go ahead and add our instance ?

  • I agree this will go a long way.  Domo encourages seperate transforms instead of subselects however from a useability standpoint it can be cumbersome.  This could help improve that experience.

  • When using procedures itd be nice if you could run them all in one transform, however for each one you have to do a drop one, a create one, and then a call one. Atleast for the ones I have been running. And since I have to run about 30 of these in one data set you can see where it adds up fast. 

     

    Here is an example procedure I had hoped i could use @ commands with instead, but without any luck. 

     

     

    CREATE PROCEDURE eric_rollup_AR_values ()
    BEGIN

    DECLARE DONE INT DEFAULT 0;

    DECLARE SUBDRY INTEGER;

    DECLARE MNTH DATE;
    DECLARE PREV_MNTH DATE;
    DECLARE YEAR_AGO_MNTH DATE;

    DECLARE AMT DOUBLE;
    DECLARE PREV_ROLLING_AMT DOUBLE;
    DECLARE YEAR_AGO_AMT DOUBLE;

    DECLARE cur_eric_rolling_inventories CURSOR FOR
    SELECT subsidiary_id, accounting_period, amount
    FROM accts_recievable_4 ORDER BY subsidiary_id, accounting_period ASC;

    DECLARE CONTINUE HANDLER FOR NOT FOUND SET DONE = 1;

    ALTER TABLE accts_recievable_4 ADD INDEX( subsidiary_id, accounting_period );

    OPEN cur_eric_rolling_inventories;
    cur_eric_rolling_inventories : LOOP
    FETCH cur_eric_rolling_inventories INTO SUBDRY, MNTH, AMT;
    IF DONE = 1 THEN LEAVE cur_eric_rolling_inventories; END IF;

    # Assumes `rolling_amount` is initially set to `amount`

    SET PREV_MNTH = LAST_DAY( DATE_SUB( MNTH, INTERVAL 1 MONTH ) );
    SET PREV_ROLLING_AMT = ( SELECT rolling_amount FROM accts_recievable_4
    WHERE subsidiary_id = SUBDRY AND accounting_period = PREV_MNTH );
    IF PREV_ROLLING_AMT IS NOT NULL THEN
    UPDATE accts_recievable_4
    SET rolling_amount = rolling_amount + PREV_ROLLING_AMT
    WHERE subsidiary_id = SUBDRY AND accounting_period = MNTH;
    END IF;

    SET YEAR_AGO_MNTH = LAST_DAY( DATE_SUB( MNTH, INTERVAL 12 MONTH ) );
    SET YEAR_AGO_AMT = ( SELECT amount FROM accts_recievable_4
    WHERE subsidiary_id = SUBDRY AND accounting_period = YEAR_AGO_MNTH );
    IF YEAR_AGO_AMT IS NOT NULL THEN
    UPDATE accts_recievable_4
    SET rolling_amount = rolling_amount - YEAR_AGO_AMT
    WHERE subsidiary_id = SUBDRY AND accounting_period = MNTH;
    END IF;

    END LOOP cur_eric_rolling_inventories;
    CLOSE cur_eric_rolling_inventories;

    END

    Eric Peterson
    [email protected]
    www.walkeredison.com
  • I would like to see the SQL data flows look something more like this:2.png

     

     

     

    Notice the "Run SQL" button to the side of each transform that would allow you to run the code without actually clicking into the transform.  Each transform is also numbered and has a section to the right for you to write any notes that may apply to that transform


    ______________________________________________________________________________________________
    “There is a superhero in all of us, we just need the courage to put on the cape.” -Superman
    ______________________________________________________________________________________________
  • Scott:

     

    That would be a piece of heaven ?

    Eric Peterson
    [email protected]
    www.walkeredison.com
  • @ST_-Superman-_This was the exact suggestion that I had a DP16 and more recently at our Atlanta user group meeting. I would love to have a run all button that would just start running each transform until it either ran them all and they checked out or errored out and mark the transform that had the error. 

    @SeanPT @StevenC  Sounds like there is atleast some movement on this feature? 

  • Jmohlenhoff that sounds like the perfect solution. I had been wondering about the error side of things, but what you suggest makes it ultra simple. 

    Eric Peterson
    [email protected]
    www.walkeredison.com
  • @ST_-Superman-_that would be fantastic.

     

    We Thompsons know whats going on ?

  • Great dialogue here, this will greatly help our product teams.Thank you!

     

    Keep the feedback coming!!

     

    CC @btm @StevenC please review the latest comments which provide tremendous details for this idea. As always please update everyone on progress, thanks!

  • This idea has been posted previously... be sure to upvote it as well ?

     

    https://dojo.domo.com/t5/Ideas-Exchange/Run-all-the-steps-in-a-dataflow-to-a-point/idc-p/25179#M4876

     


    ______________________________________________________________________________________________
    “There is a superhero in all of us, we just need the courage to put on the cape.” -Superman
    ______________________________________________________________________________________________
  • Hi Scott,


    I am glad I am not the only one missing this feature. 

    You have my vote.

     

    Please vote  : https://dojo.domo.com/t5/Ideas-Exchange/Run-ALL-SQL-s-in-dataflow-when-debugging/idi-p/27484

  • I'd be happy if you could at least 1-click-run a transform from the transform list view, without having to open each one and click run

  • Is there any way to tell how much traction this has gotten? This is still so much of a pain when working with multi step data flows. What more is needed to get this off the ground?

    Eric Peterson
    [email protected]
    www.walkeredison.com
  • Squeaky wheels right @itwalkeredison

     

    Let's just keep posting until we hear what's going on here.  Unfortunately, you can only upvote once ?

     

    I think redesigning this page would greatly improve the quality of the product for every user that steps into dataflows on a regular basis.  


    ______________________________________________________________________________________________
    “There is a superhero in all of us, we just need the courage to put on the cape.” -Superman
    ______________________________________________________________________________________________
  • Ya, and asking anyone else in our orgs or other orgs we know that have the same issue to vote it up too! That I hope gets it attention. I see that it shows in review, , I hope that means they are looking at it atleast. 

    Eric Peterson
    [email protected]
    www.walkeredison.com
This discussion has been closed.