MySQL Transform Only New Data
I'm trying to use a MySQL dataflow to transform data as it arrives (every minute via Stream API) but obviously I only want to include the new data not repeat the entire dataset transform each time (2.9m rowss and counting). Setting the append flag and the frequency on update doesn't do this, I just get the same 2.9m rows over and over again appended on each run. So how do i modify the job to only include new data?
Sorry if this is basic stuff, I'm pretty new to Domo but know standard data warehousing, ETL, MySQL, MSSQL well from a previous role.
Edit: I've seen reference in other posts to system fields _BATCH_ID_ and _BATCH_LAST_RUN_ but they do not appear in the field list and I tried to use them in a Beast Mode calculated field and couldn't get it to work. If it matters, my dataset is being created by the Stream API but I can see batches in Data Center so it is being tracked somewhere.