Redshift Performance Problem

Reply
Highlighted
Visitor

Redshift Performance Problem

I have call center data that I pull in through inContact and I need to add a date field from a date/time field in order to not have problems with time. To do this, I've made a redshift flow that selects all the fields and truncates the date/time field to a date field:

 

Select field1, field2, ...,

TRUNC(CONVERT_TIMEZONE('UTC','America/Chicago',"contactStart")) AS contactStartdate

from table

 

Since the data comes in as UTC, I first convert it to central and the truncate it. That way I get the correct date out of the stamp. The table has 52 columns and usually about 3000-4000 rows being replaced every 15min.

 

My problem is I need to update this dataset every 15min, but the redshift query sometime takes 30 seconds or over 10-15min. Any ideas on what is going on/how I can get it to consistently run for only 30 seconds?

 


Accepted Solutions
Yellow Belt

Re: Redshift Performance Problem

I was curious about Redshift run times being inconsistent and posed a question on DOJO day. Here is the question and response:

 

https://dojo.domo.com/t5/Beast-Mode-ETL-Dataflow/Redshift-vs-MySQL-vs-ETL/m-p/38009

 

I tested one of my data flows that had the same requirements you have (select all columns, but convert the timestamps) using both Redshift and MySQL and I did see both more consistent run times and a lower 30 day average duration when using MySQL.

 

So you might want to make a copy of your data flow & convert it to MySQL to see if you can achieve the results you are looking for.


All Replies
Yellow Belt

Re: Redshift Performance Problem

I was curious about Redshift run times being inconsistent and posed a question on DOJO day. Here is the question and response:

 

https://dojo.domo.com/t5/Beast-Mode-ETL-Dataflow/Redshift-vs-MySQL-vs-ETL/m-p/38009

 

I tested one of my data flows that had the same requirements you have (select all columns, but convert the timestamps) using both Redshift and MySQL and I did see both more consistent run times and a lower 30 day average duration when using MySQL.

 

So you might want to make a copy of your data flow & convert it to MySQL to see if you can achieve the results you are looking for.

Announcements
Win free lodging at Domopalooza! We want to hear your Domo story, enter our Dojo contest and win up to four nights lodging at Domopalooza. Click here!