SQL DataFlow vs API access

Hi DOJO Community,

Good morning. Do any of you know why when I am querying the API the runtime between two dates is about 2 minutes (querying 130k rows from a 16MM row dataset). But when I query it from SQL given a query using the SQL ETL DataFlow, it takes a very long time to run ( I terminated it each time)

Best Answer

  • GrantSmith
    GrantSmith Indiana 🥷
    Answer ✓

    The MySQL dataflow runs slower because it must transfer 16MM rows to the MySQL server process it and then transfer the data back. Queuing via the api doesn’t need to transfer the data across systems so it’ll be much faster

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Answers

  • DashboardDude
    DashboardDude Chicago 🟠

    Hi @Shane_Lester99 ,

    Can you send a screenshot of the ETL? I bet you have one step that is the culprit and taking forever. May be a join or something.

  • Shane_Lester99
    edited November 7

    SELECT * FROM `plg_small_parcel_recreated_invoice_prod` WHERE `Invoice Date` BETWEEN '2022-10-30' AND '2022-11-6'


    Above is the query.