Problem(s in general): Weekday / Work week /Week of year troubles of the date-time spark functions.
Wish: An instance wide solution.
Derived question:
Is there a way to gain access/manipulate the “locale” Property of the OD Core?
Is the OD Core written in Java? (I know from experience that the Java standard API for Time-/zones does work well even for odd edge cases.)
Would/Do the spark functions inherit the “locale” attribute?
Hey Jonas,
yes, the locale can be set for the JVM running ONE DATA.
However, it is discouraged to do so, since some third-party (including Spark) logic “assumes” UTC time when there is no further timestamp information. Therefore, we soft-enforce UTC to be the JVM timezone.
On the last question: It depends. When Spark is running in the same JVM as OD Core, it inherits the locale. If it runs its master in a separate process (SEC deployment), locales must be set individually (and should be set to the same to avoid shifts due to “same timezone” assumptions).
You can use different timezone settings in your Spark SQL queries when e.g. parsing timestamps. Maybe that helps. Setting instance-wide locales will have side effects on existing WFs and other logic that and is therefore discouraged.