Magic ETL Append to Output

I would love to be able to append to a Magic ETL output the way I can append to a data set via the Domo workbench.

 

My current use case is to take a static daily update of segment statistics from our email provider (~2 dozen segments) and flatten it out into a single record with a column for each segment. Then I want to append that record every day so I generate a historical trendline for each segment over time.

 

I got this to work with Magic ETL by bootstrapping the output DataSet to make it available and then going back to edit the DataFlow. Then I fed the output data set back in as an input DataSet. Then I appended it with the daily record generation so that I get a new record every day.

 

This doesn't seem like a particularly efficient way to achieve an append operation in the long run, so I'm hoping that down the line we can get an option to append to the output DataSet instead of replacing.

4
4 votes

· Last Updated

Comments

  • Hi @cr1ckt,

    We agree that this would be a huge help—I've got it added to our future roadmap. Thanks for the feedback!

    Best,
    Matt Chandler
    Domo
  • Best,
    Matt Chandler
    Domo
  • Agreed, this would be really helpful. @cr1ckt, can you explain in a little more detail the bootstrapping method you are currently using with Magic ETL? 

     

    "I got this to work with Magic ETL by bootstrapping the output DataSet to make it available and then going back to edit the DataFlow. Then I fed the output data set back in as an input DataSet. Then Iappended it with the daily record generation so that I get a new record every day."

     

    I need to configure something similar and keep accidentally creating multiple outputs with the same name in the process. TIA!

     

     

  • @JustAGirlyGeek: It's possible that an update has broken my workaround since I tried it, but I was referring to the fact that an input DataSet must exist to be selected as the source of the input in the Magic ETL. Therefore, when I am creating the new ETL (and new output DataSet), I have to run it once to get an empty DataSet to append to.

    Here is an example where Bronto Segments has a bunch of rows with the daily state of things and Bronto Segments - History has the running log with a row for each daily summary. Uncollapse columns flattens the rows into a single row and remove duplicates is applied against a batch identifier so that repeated runs on the same daily report don't duplicate rows in the log DataSet. I am not sure why the ETL output would duplicate the output DataSet for you. 

    ETL_Append.png

  • Perfect, the part I was missing was updating the dataflow to use the new output as my input.

  • Exactly - The thing to remember is that Domo recognizes your datasets by their unique identifier, not their name.  That's how you can have multiple datasets with the same name.  If you don't know yet how to find that unique identifier...Check out the address bar when you have a dataset open.  The identifier will be a long string in the address.  

    DataMaven
    Breaking Down Silos - Building Bridges
    Check out my video!
    **Say "Thanks" by clicking the heart in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
This discussion has been closed.