Is there a way to limit the number of input records on the ETL dataflow ?

Reply
White Belt

Is there a way to limit the number of input records on the ETL dataflow ?

I have 10 GB of data and I want to run a python script on that. But I am getting Out of memory issue because of the size of data.

I want to limit the number of records from the input dataset. How to do it? Thanks in advance for suggestions and solutions.

azu White Belt
White Belt

Re: Is there a way to limit the number of input records on the ETL dataflow ?

Hello,

 

Are you using the Magic Transformation: ETL? Or how are you running python scripts?

 

If you are using the ETL, what about using the Filter and just choose a subset of your data based on any condition you'd prefer?

filterDomo.pngETL Filter

Announcements
Check out these great solution videos! The new iSolveditWithDomoboard is live, share yours and earn higher rankings in Dojo click here!