Is there a way to limit the number of input records on the ETL dataflow ?

user15978
user15978 ⚪️
in Magic ETL

I have 10 GB of data and I want to run a python script on that. But I am getting Out of memory issue because of the size of data.

I want to limit the number of records from the input dataset. How to do it? Thanks in advance for suggestions and solutions.

Comments

  • Unknown
    Unknown

    Hello,

     

    Are you using the Magic Transformation: ETL? Or how are you running python scripts?

     

    If you are using the ETL, what about using the Filter and just choose a subset of your data based on any condition you'd prefer?

    ETL FilterETL Filter

Categories