Is there a way to limit the number of input records on the ETL dataflow ?

Reply
Visitor

Is there a way to limit the number of input records on the ETL dataflow ?

I have 10 GB of data and I want to run a python script on that. But I am getting Out of memory issue because of the size of data.

I want to limit the number of records from the input dataset. How to do it? Thanks in advance for suggestions and solutions.

azu
Visitor

Re: Is there a way to limit the number of input records on the ETL dataflow ?

Hello,

 

Are you using the Magic Transformation: ETL? Or how are you running python scripts?

 

If you are using the ETL, what about using the Filter and just choose a subset of your data based on any condition you'd prefer?

filterDomo.pngETL Filter

Announcements
Looking for the latest Community solutions? Please visit our accepted solutions board here!