Magic ETL - Change column types and merge files
Hi, I am merging 7 data sets into one. I would like to change the column types on all 7 of the source files before merging into the output data set.
2 columns in each file are "text" columns, but I wish to change them to date type.
Some of the data contains "n/a" as a string. When I ran a simple Set Column Type ETL on this field, the "n/a" prevented the conversion.
I am applying a value mapper to both text columns containing the date (but with column type "text") to convert the "n/a" string to an arbitrary date (e.g. 1970-01-01).
Then I am applying the Set Column Type transformation to change those columns to column type date. After that I run an append rows transformation to complete the ETL and populate the Output file. See below a snippet of the Magic ETL.
When I run this, the ETL creates 2x the number of rows vs. the sum of the original 7 datasets.
Any ideas why this is and how I can achieve my goals (replace column types for 2 columns and combine all 7 sets)?