Connectors (box, dropbox etc) that enables recursive upload of multiple files from a directory

Kaushik

Problem statement:

There were many occations where a user has to upload multiple files (of same format) from a location into DOMO. More the files, tedious the job becomes. For example, when we are introducing a new upstream data source, there are instances where we had to load 100s of historical files into DOMO using dataflow ETL before the automation is set up for current/delta data import. The need for multiple data import can also be true for current data import (not just historical) because many of the application sources does not have feasibility to consolidate data into a singe file in their GUI (they let you download only "date" granular data while we need to load data for "months"! )

 

Workaround:

In my team I created a VBA tool to consolidate the historical files into one file and then loaded into DOMO dataflow (manually). 

 

Suggestions:

Many of the ETL tools in the market (SSIS, Infomatica etc) lets recusrive file integration ie, the ETL process runs recursively for every file in the directory. Domo connecter for the applicable data sources can be upgraded to have this feasibility.

Even better: It is better if the connecter can consolidate all the files first and then run the dataflow task only once. This reduced the turn around time for the entire process. 

 

Thanks!

 

jlazerus

I also think this would be very helpful.  I have a situation where I need to upload numerous files weekly into their own datasets and the filenames change weekly and have an updated date at the beginning of the filename.