Custom connector - append all rows with an ID greater than last appended
I am making a custom connector. The data vendor’s api needs me to send the IDs of the rows I want to ingest, and they can then send up to 10-20k rows without a time-out. So, I need to break up my requests in blocks of 10k rows, and give the ID range each block. I can run this every hour, as each hour will be less than 10k rows, and append.
Question is, what’s the best way to query the existing table, grab the highest ID present, add one, then use this as a starting ID for the next pull?
- the vendor has a version of the Api that works with dates, but even a single day can cause time outs.
- the feed is json
- I am using append, rather than replace
- 10.7K All Categories
- 13 Getting Started in the Community
- 31 Beastmode & Analytics
- 2.1K Data Platform & Data Science
- 59 Domo Everywhere
- 2.7K Charting
- 2.4K Ideas Exchange
- 1.3K Connectors
- 362 Workbench
- 302 Use Cases & Best Practices
- 500 APIs
- 118 Apps
- 48 News
- 753 Onboarding
- 1.1K 日本支部
- Private Company Board