Trying to filter out rows, where a certain column has NULL values

user0010240
user0010240
in Dataflows

Hi,

I have a dataset of almost 2 billion rows. I need to filter out rows, where a certain column has NULL values.

What is the best and fastest way to do it?

I tried creating a DataFlow using the mySQL option, but the flow has been running for more than 24 hours now.

Could any of you please help in understanding how this process can be made to run faster?

Tagged:

Comments

  • Valiant
    Valiant 🔵

    With large datasets, I normally have more luck (where speed is concerned) using the ETL tool.

     

    You could accomplish what you're looking for using 3 widgets, Input Dataset into Filter Rows into Output Dataset.

     

    On the filter rows, just select the row you want to filter and there is an option for IS NOT NULL.

     

    That should hopefully be all you need. If not, just let me know.

     

    Sincerely,

    Valiant

     

    **Please mark "Accept as Solution" if this post solves your problem
    **Say "Thanks" by clicking the "heart" in the post that helped you.

  • DataMaven
    DataMaven 🟢

    Are you needing to filter those rows out completely, or do you want to improve your filtering ability in your visualizations?  Is this a text field or numeric?  

     

    If you want to keep the rows, but need the null values filled in, you can use the mapper. This works for both text and numeric fields. 

     

    If you are simply needing to remove them efficiently, @Valiant 's solution is perfect.  Cat Wink

     

    Value Mapper - Numeric.png

     

     

Categories