This document discusses weather data and analytics. It notes that over 26 billion weather forecasts are made daily, using data from 2.2 billion unique locations and 200,000 personal weather stations. The document outlines different roles in data science, such as data scientists who build machine learning models versus data engineers who ensure pipelines scale. It also provides steps for setting up a data analytics project, including collecting weather and other data in a data lake, using Spark for analytics, and exploring the data using notebooks.

4. Everywhere Defined
• 26B forecasts /day or 250,000/second
– vs 3.5B Google queries daily
• 2.2 billion unique locations
• 200k personal weather stations
• 200M active mobile users
• Petabytes of data generated daily

5. Our Brands

6. Over
30 Billion
Served

7. Flight Routing

8. Energy Trading

9. Insurance

10. Weather Alerting

11. Decisions at Scale
101001110100101
101001110100101
010100101011001
101010101011100
000011010110010

12. Who Are You?
RDBMS
?

13. Who Are You?
?

14. Social Weather

15. Social Weather
RDBMS

16. Social Weather
RDBMS
SELECT count(*) FROM wx_reports
GROUP BY time / 300000 * 300000

17. Social Weather
Live Reporting
ETL

18. Social Weather
Live
Reporting
Sqoop
M/R

19. Scaling with Spark
Live
Reporting

20. Easing the Transition
101001110100101
101001110100101
010100101011001
101010101011100
000011010110010

21. Easing the Transition
101001110100101
101001110100101
010100101011001
101010101011100
000011010110010

22. Easing the Transition
101001110100101
101001110100101
010100101011001
101010101011100
000011010110010
101001110100101
010100101011001
101010101011100
000011010110010
10100,11101,00101
01010,01010,11001
10101,01010,11100
00001,10101,...

23. Easing the Transition
101001110100101
101001110100101
010100101011001
101010101011100
000011010110010
101001110100101
010100101011001
101010101011100
000011010110010
10100,11101,00101
01010,01010,11001
10101,01010,11100
00001,10101,...

24. Scaling with Spark
Live

25. Scaling with Spark
Live
Reporting

26. Batch Aggregation
val wx_reports = // load data from database
val sql = new org.apache.spark.sql.SQLContext(sc)
import sql.implicits._
wx_reports.toDF.registerTempTable("wx_reports")
val counts = sql("select count(*) from wx_reports group by
timestamp / 300000 * 300000")

27. Streaming Aggregation
val wx_reports = // load from streaming source
wx_reports.foreachRDD { rdd =>
val sql = SQLContext.getOrCreate(rdd.sparkContext)
import sql.implicits._
rdd.toDF.registerTempTable("wx_reports")
val count = sql("select count(*) from wx_reports")
}

28. Data Science Roles
Data Scientist Data Engineer

29. Data Science Roles
Data Scientist Data Engineer
Machine learning expert

30. Data Science Roles
Data Scientist Data Engineer
Machine learning expert Scalable algorithms expert

31. Data Science Roles
Data Scientist Data Engineer
Builds pipelines that
work on her laptop

32. Data Science Roles
Data Scientist Data Engineer
Rewrites her pipelines
to scale better

33. Collaborative Data Science

34. The Analytics OS
Notebooks
Stream
Analytics
Batch
Analytics

35. But …

36. The Real World (Enterprise Version)

37. The Real World (Startup Version)
Application MySQL

38. Step 1: Pick a Problem to Solve

39. Step 2: Build a Data Lake

40. Step 3: Set up Spark
• Direct download
• Hadoop distribution
(Hortonworks, Cloudera, etc)
• Managed service (Elastic
MapReduce, Databricks,
BlueMix, etc)

41. Step 4: Start Collecting Data
• Options:
– Sqoop to move RDBMS tables
– Flume/FluentD to move logs
– Import from Spark-supported data sources
– Using Spark Streaming attached to a queue
– …

42. Step 5: Use a Notebook

43. Final Thoughts

44. Thank You!
Robbie Strickland
@rs_atl
(we’re hiring!)