Data Flow help
I am humbly reaching out for some help.
I have a vendor sending a snapshot of a database table every day and not sending me diffs. I am trying to determine the diffs. The connector type is SFTP CSV. The file they send daily is overwritten daily. I currently have the data connector appending the data from the previous day to the new file they sent over that day.
Is this the correct approach? Should I have the connector setup as Append vs Replace or vice versa?
What success should look like as an output dataset that appends updated records? Below, you will find an image of 6 records that are the same as an individual in the file.
As you can see, it's the same row the only difference is the row_id which is really when the file was run and appended per the connector. So for example, imagine that these 6 rows are John Smith, what we care to see is the original John Smith record and only any new/updated changes. So if there were three changes I would only see three new records.
Thank you so much in advance. We are happy to answer any questions.