Append a dataset without using recursive design

Is there any other way to append to a dataset without using a recursive design in Magic ETL?

Tagged:

Best Answer

  • Jessica
    Jessica ⚪️
    Accepted Answer

    Thanks I found the append in DOMO Workbench. The basic csv connector from Workbench does not have a setting on the DOMO application side.


    It's on the Workbench side.


    Thanks again

Answers

  • MarkSnodgrass
    MarkSnodgrass Portland, Oregon 🔵

    @Jessica

    You will probably need to provide more specifics as to what fields are in your dataset to work with, but generally I would say yes you can do this without a recursive.

    In the ETL, you can split off the IDs and dates in your dataset and then group by max date. Then join it back to your main dataset and join on the date and id. This would give you the latest of each your unique rows.

    Hope this helps.

  • GrantSmith
    GrantSmith Indiana 🟤

    @Jessica

    It depends if you're wanting to ignore duplicate records or not and how you're pulling in your data. If you're using Workbench you could utilize and UPSERT to overwrite existing records or an APPEND to just tack on the result set to the existing domo dataset.

    Another option would be to utilize a dataset copy connector to take snapshots and set it to append but it could possibly have duplicate rows.

    Most connectors also support APPEND as the update method.

  • jaeW_at_Onyx
    jaeW_at_Onyx Budapest / Portland, OR 🟤

    Without hacking Domo, you can only APPEND or UPSERT data during ingest (not after ETLs have been constructed). That said, if you get crafty you can alter a dataflow to APPEND instead of REPLACE (but not UPSERT)


    In this video i got the dataflow to APPEND but was unable to get it to UPSERT as desired.

Sign In or Register to comment.

Hey, Stranger!

It looks like you're new here. Those who sign in get access to engage with even MORE fire content. To get involved, click one of these buttons!