Dataflow: Ability to write R or Python code within a ETL-Dataflow

doc-domo

ETL dataflows are a simple to use pipleline model for transformation, however there are times when I'd like to include R or Python code as transformation so I can leverage an existing library, e.g., machine learning, text mining, etc.

 

My code would read a table from input, e.g., like this

 

df <- DomoR::fetch()

transform the data using R or Python code/libraries,  then output the table to the pipleline, eg.

 

DomoR::put(df)

 

The ELT block would be a transient black box that reads from the input and write to the output.  I imagine this could be done using a container that is quickly spun up to run the R or Python code.  

 

In the longer term, I'd like to be able to create standard ETL blocks that I can drag and drop from the ETL palette. Adding this would make Domo competative with RapidMinder, https://en.wikipedia.org/wiki/RapidMiner

btm

Thank you for submitting this @Doc. I am assigning to our product manager @StevenC to review and comment.

Kurbz

Hi folks,

I was thinking about this very item last night, but would like to extend it to include an SQL block. the SQL block in ELT could accept many inputs and produce an output. As per the idea above, this would really extend the functionality.

thats it, cheers.

Rgds,

K.