The Union processor converts my column type to String, why?
Image status:
Left result table has no rows.
Right result table has column_a with type double.
Result table after the union has column_a with type String.
Additional detail: column_a with type double is also the input to the yellow processor. In this particular case however all rows are filtered.
Are you using the Union Processor or Spark SQL Union? In case of the latter, are you explicitly listing the columns in the Union statement?
The SPARK SQL Union can mix up columns (see here) and might be the reason for the type conversion.
3 Likes
Using the Union processor does yield a better result
.
Explicitly naming the involved columns in the SQL- query does work as well.
Thank You