Successfully reported this slideshow.

Comprehensive View on Date-time APIs of Apache Spark 3.0

1

Share

1 of 27
1 of 27

Comprehensive View on Date-time APIs of Apache Spark 3.0

1

Share

Download to read offline

The talk is about date-time processing in Spark 3.0, its API and implementations made since Spark 2.4. In particular, I am going to cover the following topics: 1. Definition and internal representation of dates/timestamps in Spark SQL. Comparisons of Spark 3.0 date-time API with previous versions and other DBMS. 2. Date/timestamp functions of Spark SQL. Nuances of behavior and details of implementation. Use cases and corner cases of date-time API. 3. Migration from the hybrid calendar (Julian and Gregorian calendars) to Proleptic Gregorian calendar in Spark 3.0. 4. Parsing of date/timestamp strings, saving and loading date/time data via Spark’s datasources. 5. Support of Java 8 time API in Spark 3.0.

The talk is about date-time processing in Spark 3.0, its API and implementations made since Spark 2.4. In particular, I am going to cover the following topics: 1. Definition and internal representation of dates/timestamps in Spark SQL. Comparisons of Spark 3.0 date-time API with previous versions and other DBMS. 2. Date/timestamp functions of Spark SQL. Nuances of behavior and details of implementation. Use cases and corner cases of date-time API. 3. Migration from the hybrid calendar (Julian and Gregorian calendars) to Proleptic Gregorian calendar in Spark 3.0. 4. Parsing of date/timestamp strings, saving and loading date/time data via Spark’s datasources. 5. Support of Java 8 time API in Spark 3.0.

More Related Content

×