What is the best way to have updating data from mongo collection?
Currently have some time series data that I would need the full historical data, but only needed it to update every hour or so. The update needs to be a merge, so any records that are changed are updated correctly. Right now I don't see a good way to do this within domo ui, and only see custom connectors or api feeds to update within domo.
Is there a good way to accomplish this with the connectors?
Answers
-
Hi @banderson
it sounds like you’re talking about an upsert type of connection. There are some connectors which support upsert however it appears that the MongoDB connector isn’t one of them. You could try and do something like a recursive dataflow to handle the upsert logic yourself however note that it doesn’t scale as well as the dataset gets larger.
There are some good KB articles about this
Magic ETL v1
Magic ETL v2
1 -
Thanks Grant. This will be for a data set that is large that we expect to get exponentially larger. I'll take a look at those and see if they can still be used.
0 -
you may also want to look into data assembler if you have a large dataset and could possibly set it up using some partitions but it’s premium feature. There are a few guidelines for using it which are documented in the knowledge base
https://knowledge.domo.com/The_Appstore/Available_Apps/Data_Assembler_Usage_Guidelines
1 -
@banderson use UPSERT + Domo CLI
https://www.youtube.com/watch?v=kHSeSbSFS1Q (sorry less polished video, but gets the job done).
1
