The document provides an overview of using PySpark for time series analysis. It discusses that time series data can come from sources like IOT feeds, sensor data, and economic indicators. Time series analysis in PySpark allows for windowed aggregations and temporal joins on massive time series datasets that can be both wide and narrow. While basic analytics are possible in PySpark, libraries like Flint provide additional functions specialized for time series analysis on large datasets in a distributed environment. The document encourages attendees to speak with the author after the talk to see a time series analysis library in PySpark demonstrated.

1. PySpark for Time Series Analysis
David Palaitis
Two Sigma Investments

2. About Me

3. Important Legal Information
The information presented here is offered for recruiting purposes only and
should not be used for any other purpose (including, without limitation, the
making of investment decisions). Examples provided herein are for illustrative
purposes only and are not necessarily based on actual data. Nothing herein
constitutes an offer to sell or the solicitation of any offer to buy any security or
other interest. We consider this information to be confidential and not for
redistribution or dissemination. Some of the images, logos or other material
used herein may be protected by copyright and/or trademark. If so, such
copyrights and/or trademarks are most likely owned by the entity that created
the material and are used purely for identification and comment as fair use
under international copyright and/or trademark laws. Use of such image,
copyright or trademark does not imply any association with such organization (or
endorsement of such organization) by Two Sigma, nor vice versa.

5. Time Series
IOT feeds
sensor data
economic data
An ordered sequence of values of a variable

6. Time Series Analysis

7. Time Series Analysis

8. Time Series Analysis

9. Time Series at Two Sigma
Millions of
Time Series
Big and
Small
(1GB – 1PB)
Narrow (10
columns) and
Wide (1MM
Columns)
Evenly and
Unevenly
Spaced
Observations

11. Let’s start from the beginning …

21. Examples!

22. What’s Missing?

23. You can’t even do “Word Count”

32. “Word Count” !

33. What’s missing? Time.

34. Windowed Aggregations

35. Temporal Joins
} window

40. w is a window specification e.g. 500ms, 5s, 3 business days
RDD[(K,V)] -> RDD[(K,Seq[V])]

41. reduceByWindow(f: (V, V) => V, w): RDD[(K, W)] => RDD[(K, V)]

42. reduceByWindow(f: (V, V) => V, w): RDD[(K, V)] => RDD[(K, V)]

43. https://github.com/twosigma/flint

44. Getting
Started …

52. Looking ahead.

53. Thank You.
Find me after the talk to see Flint in action.