Is there a limit to the number of inputs on a Magic ETL dataflow?

MattGo
MattGo βšͺ️

Beginning to create a dataflow in ETL that will require 100+ inputs and don't want to continue if I am going to hit a limit at some point. Anyone know if there is an input limit? Thanks.

Comments

  • To my knowledge, there isn't a cap, however I've never gone beyond 5 inputs.

    Β 

    I'm not familiar with your data landscape but taking a step back, it might be prudent to consider other ETL options given your estimation of inputs. That dataflow would be cumbersome at best to manage and make changes to, not to mention debug for errors and even fully execute.

    Β 

    If possible, I'dΒ  recommend trying to do some of the heavy lifting on-premise maybe in SQL environment.

  • Can you provide some more info about your use case? I agree with the other post, it seems like you may be better off doing things somewhere else to bring into Domo

  • AS
    AS πŸ”΅

    Agreed with responses above.Β  Even if you can I'd have to question if that's something you want to maintain. Maybe if you let us know about why you have 100 sources to begin with.Β  There may be solutions to pre-combine that data before loading into Domo.