Mutiple Zip files files in S3 Bucket

Reply
Highlighted
Yellow Belt

Mutiple Zip files files in S3 Bucket

I am having 2 problems with the S3 Connectors (Advanced and Standard).  I am struggling to find which one is better suited to handle multiple files.

 

Here are the details:

Our vendor is posting a new Zip file each day to an S3 Bucket.  The subfiles within the new daily Zip all have the prefix change based on a physical location identifier.  For example,

 

1345_08_14_20.json

1298_08_14_20.json

 

where 1345 and 1298 are the location identifers

 

Using the advanced connector, I am only able to pull one json file at a time I include the prefix (i.e. 1345) in the SUBFILE field, however, it will only extract that first json file.  I need to pull all the json files within that Zip.  Would re-naming the files to something with a generic name file the issue (i.e. Daily File_1345_08_14_20.json) where i use the Daily_File in the subfile field?

 

The second question is how to append the new file(s) each day as we are not able to enter a variable.  I see the regular S3 Connector will grab most recently modified, however, the Zip option requires that you discover the zipped files using a drop down.

 

Any hints or suggestions would be amazing!!! Thanks

 

 

Highlighted
Black Belt

1) it would probably be easiest to perform server-side scripting that combines the data into one file BEFORE sending it to Domo.

2) most connectors have an option to Replace or Append data during ingestion to Domo (check if there's an option under "Scheduling" during connector configuration).


Jae Wilson
Check out my Domo Training YouTube Channel

**Say "Thanks" by clicking the heart in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"
Announcements
Domopalooza 2021 Call for Presenters: We want to hear how Domo is revolutionizing the way you do business!

Click here to submit your story.