Connector & DataSet Issue
I am using an out of the box Connector for my ticket data but the only two options are Append and Replace, I firstly pulled down 365 days and now each day I either have the option to do an append or replace, if I replace I lose 365 days data and left with 24 hours so I use Append (API limits mean I couldn't do a total update every 24hrs for 365 days of data). CountDistinct works in most cases but unfortunately there are some fields which are changed in the life an incident so for example, if an incident was changed to a Service Request mid-life (a type field) then it would have one entry for Incident and 1 for SR which gives me incorrect data.
Would there be a way of sorting my dataset using ETL so that each time an update is done my duplicates are removed please? Issue is with this case is that it would need to remove the oldest duplicate because the update has occurred and the type changed.