2 Issues: MySQL SSH - Query Parameter and STREAM API
Sorry for combo post but I got problems.
1. I'm using the MySQL SSH Connector - I'm using the QUERY option to pull in data from a table. Currently, most are just daily imports but some tables are 1B rows and so doing a 'Select * from table;' is VERY inefficient. What I'd like to do is using a parameter to only pull in the most recent data (if the source table has a date field or if the source field has a PK ID that I could use). It looks like the query parameter option was intended to do that, but I don't see how it would store a variable so it could reference it when it imported. I know the other option here is to just do a recurse DF but it will slow me down so I'm trying to reduce waste where I can.
2. Am I right in thinking that this may be a good case for Stream API? It looks in the documentation that it only works for very large flat files, but that doesn't make sense to me...so I'm hoping I'm wrong with that. I'm not a programmer either so any helpful advice here is VERY much appreciated.