Trying to filter out rows, where a certain column has NULL values

Reply
Visitor

Trying to filter out rows, where a certain column has NULL values

Hi,

I have a dataset of almost 2 billion rows. I need to filter out rows, where a certain column has NULL values.

What is the best and fastest way to do it?

I tried creating a DataFlow using the mySQL option, but the flow has been running for more than 24 hours now.

Could any of you please help in understanding how this process can be made to run faster?

Black Belt

Re: Trying to filter out rows, where a certain column has NULL values

With large datasets, I normally have more luck (where speed is concerned) using the ETL tool.

 

You could accomplish what you're looking for using 3 widgets, Input Dataset into Filter Rows into Output Dataset.

 

On the filter rows, just select the row you want to filter and there is an option for IS NOT NULL.

 

That should hopefully be all you need. If not, just let me know.

 

Sincerely,

Valiant

 

**Please mark "Accept as Solution" if this post solves your problem
**Say "Thanks" by clicking the "heart" in the post that helped you.

Major Brown Belt

Re: Trying to filter out rows, where a certain column has NULL values

Are you needing to filter those rows out completely, or do you want to improve your filtering ability in your visualizations?  Is this a text field or numeric?  

 

If you want to keep the rows, but need the null values filled in, you can use the mapper. This works for both text and numeric fields. 

 

If you are simply needing to remove them efficiently, @Valiant 's solution is perfect.  Cat Wink

 

Value Mapper - Numeric.png

 

 

DataMaven
Breaking Down Silos - Building Bridges
Check out my story!
**Say "Thanks" by clicking the heart in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"
Announcements
Looking for the latest Community solutions? Please visit our accepted solutions board here!