ETL using Append object not showing all records

Reply
Highlighted
Yellow Belt

ETL using Append object not showing all records

Has anyone noticed that all the other object/functions in ETL in the preview tab show the full number of records except Append Rows.

 

In a current ETL I am building and troubleshooting, I was trying to find out where data is dropping off in error and I got to the Append Rows object I was using and it is by default showing only 100 records.  I tested to make sure this is not where my error is occurring by creating a Test Output object directly after the Append object and the test dataset accurately includes the 2400 records I expected.

 

It makes it difficult to test when the preview function is not working accurately.

Highlighted
Black Belt

I have found that I can't trust the Preview window, regardless of how many rows I tell it to be in the preview. The row amount you choose (10k, 100k, etc.) only affects how many rows it will pull from your input datasets. It doesn't affect how many rows will show in the tile preview windows. Unfortunately, this means that you will need to run the actual ETL and look at your output dataset to know for sure if your tiles are configured correctly. 




**Make sure to like any users posts that helped you and accept the ones who solved your issue.**
Highlighted
Black Belt

.. it's not that Preview isn't showing your preview accurately, it shows a max of 100 rows by design.  That's just ... how it works.

 

What you can do is FILTER some of your data before you get to the APPEND tile.  If you know you're interested in a specific date, customer, or amount, apply the filter and then look.

 

Also, an APPEND will pull all the data from Dataset A and then UNION Input B.  If you're worried about a specifici input .... why not preview the data further upstream?


Jae Wilson
Check out my Domo Training YouTube Channel

**Say "Thanks" by clicking the heart in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"
Announcements
Domopalooza 2021 Call for Presenters: We want to hear how Domo is revolutionizing the way you do business!

Click here to submit your story.