Updating a DataSet Triggered by the Finish of a Glue Job

Tow
Tow Tokyo ⚪️
in Dataflows

We are using a Connector to retrieve data from Amazon Redshift and display it in Domo.

We use AWS Glue to retrieve data from Oracle and store it in Redshift, but I don't think it's possible to trigger the end of the Glue job to update the DataSet in Domo.

Currently we schedule the DataSet update around the time the Glue job finishes, but in some cases the timing doesn't work due to delays in the Glue job.

If anyone has successfully handled such cases, could you please share them?

Tagged:

Best Answer

Answers

  • GrantSmith
    GrantSmith Indiana 🥷

    Hi @Tow

    Typically when I have a more complex scheduling pipeline like this with data outside of Domo I’ll use a Python script to kick off each of the jobs serially in a single script so I know it’ll run each step in order.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • GrantSmith
    GrantSmith Indiana 🥷

    @jaeW_at_Onyx I believe the issue is that the Glue process happens on AWS's end so the CLI wouldn't be able to kick it off and know when that job has finished.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • jaeW_at_Onyx
    jaeW_at_Onyx Budapest / Portland, OR 🟤

    @GrantSmith , you'd use the CLI to start the execution of the dataset in Domo. presumably, the trigger would be the completion of the Glue job.

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • Tow
    Tow Tokyo ⚪️

    Thank you, @GrantSmith and @jaeW_at_Onyx, for being so helpful. I've never used the CLI before, so I guess I'll need to fully test it. I will make it a future issue.

Categories