DataFusion - Flag to mark DataFusion to include all source columns by default

Use Case:

Source DataSet jobs that allow schema changes and are subsequently joined together via DataFusions. Columns get added to primary source and I want them to flow through to the DataFusion.

Current Behavior:

DataFusions pick up new columns in the preview, but they are excluded by default. Each column must be enabled on each DataFusion as it is added to the source DataSet.

Desired Behavior:

Allow a DataFusion to be configured to flip this behavior so that each new column is included by default. This reduces the amount of time spent enabling all the columns as the available DataSet expands. It also allows existing behavior to continue but lets me mark each DataFusion as adopting this paradigm once.

1 votes

Β· Last Updated

This discussion has been closed.