Can you filter data from input dataset in a MySQL Flow

FXM32
FXM32 ⚪️
in Data Platform & Data Science

Hello,

I have a dataset that has data for the past 20 fiscal years. I use it in a flow that only looks at the last two fiscal years and when the flow runs it takes a good 10-20 minutes just to read the data in the dataset and then start transforming it.

I was thinking of expediting the loading process by pre-filtering the last 2 years only. I would have tried with dataset views but these are no accepted yet as input datasets in a flow.

I do not want to create another flow that will populate and additional dataset just with data for the past two years. I am looking for a solution that would leverage the existing dataset and put a filter on it.

Any suggestion is greatly appreciated.

Answers

  • GrantSmith
    GrantSmith Indiana 🥷

    Are you using a MySQL dataflow or a Magic ETL data flow?

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • FXM32
    FXM32 ⚪️

    Hi Grant,

    it is a MySQL dataflow

    Thank you

Categories