I am having 2 problems with the S3 Connectors (Advanced and Standard). I am struggling to find which one is better suited to handle multiple files.
Here are the details:
Our vendor is posting a new Zip file each day to an S3 Bucket. The subfiles within the new daily Zip all have the prefix change based on a physical location identifier. For example,
where 1345 and 1298 are the location identifers
Using the advanced connector, I am only able to pull one json file at a time I include the prefix (i.e. 1345) in the SUBFILE field, however, it will only extract that first json file. I need to pull all the json files within that Zip. Would re-naming the files to something with a generic name file the issue (i.e. Daily File_1345_08_14_20.json) where i use the Daily_File in the subfile field?
The second question is how to append the new file(s) each day as we are not able to enter a variable. I see the regular S3 Connector will grab most recently modified, however, the Zip option requires that you discover the zipped files using a drop down.
Any hints or suggestions would be amazing!!! Thanks