Joining two datasets in ETL using SQL

user04906
user04906 ⚪️
in Data Platform & Data Science

Hi,

 

I am using ETL magic transform sql function.

Input: Dataset

Transform: SQL query let's call it table A

Output: Select * from table A

 

It is a really simple sql query which should take a maximum of 10 seconds to compute. However, when I save and run the dataflow it is currently on the 10 minute mark.

 

Any idea why this is happening?

 

 

 

Best Answer

  • cwolman
    cwolman 🟠
    Answer ✓

    The size of the dataset will affect the time it takes to complete.  The data needs to be loaded, transformed, and then uploaded and indexed in the presentation layer.  Large datasets will take time to go through the process even if the transforms only take seconds.


    -----------------
    Chris

Categories