Amazon S3 connector and Redshift

greg_bradsdeals

I would love to have Amazon S3 connector that is capable of using Redshift COPY command and pull in data from my S3 bucket.

 

The basic idea is to use Redshift UNLOAD command on client (my) side and use COPY command on Domo side to load data into some dataset.

 

Redshift UNLOAD command: http://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html

 

 

For example:

 

I would unload my data into my S3 bucket as follows:

unload ('select * from public.sales;')
to 's3://rs-unloads/sales_'
credentials 'aws_access_key_id=XXX;aws_secret_access_key=XXX'
allowoverwrite
manifest;

 

My S3 bucket:

sales_0000_part_00
sales_0001_part_00
sales_0002_part_00
sales_0003_part_00
sales_manifest

 

Then, within Domo, I would load data from above S3 bucket into 'sales' dataset:

copy sales from 's3://rs-unloads/sales_'
credentials 'aws_access_key_id=XXX;aws_secret_access_key=XXX'
manifest;

 

Note that UNLOAD command is splitting data into multiple files, therefore there is a manifest file. Also, by using Redshift UNLOAD command, we gain better performance while unloading large datasets, and again using COPY command is recommended way of loading data into Redshift by amazon. 

product_John

Thank you for submitting this idea @greg_bradsdeals.  I am assigning to our product manager @kamuela for review.

kamuela

kamuela

@greg_bradsdeals  Thanks for the suggestion.  I'll take it up with our engineering team and see if would make sense to add the enhancement.

 

Thanks,

Sam

 

 

greg_bradsdeals

Hi, is there any feedback on this feature?

greg_bradsdeals

@kamuela, @product_John Just wanted to followup on this post.

kamuela

@greg_bradsdeals We actually have a Redshift Unload connector that we have built that should provide this for you.  It isn't generally available so you will need to submit a ticket to Support and they can help you get it powered up in your account.

 

Thanks,

Sam

 

greg_bradsdeals

So, I'm assuming "Redshift Unload" connector and "Amazon Redshift" database connector are two different things? I will ping support to enable Redshift Unload connector. Thanks!

kamuela

Definitely different. We have Amazon Redshift and Amazon Redshift Unload.

greg_bradsdeals

Great. What are the options after unloading data into S3 bucket using Redshift Unloader? Is there Redhift Copy connector/option to load data into domo? Are there other Redshift connectors available? Sorry for all the questions ?

kamuela

The unload connector will wait for the unload command to complete and then import the data from the S3 bucket into Domo automatically. 

greg_bradsdeals

Perfect! Thank you!