Hadoop World 2011: Changing Company Culture with Hadoop - Amy O'Connor, Nokia


Published on

We are living in a time of tremendous convergence, convergence of mobile, cloud and social…This convergence is forcing companies to change. At Nokia, we are changing the way we make decisions, from a manufacturing model to a data driven one. Yet making cultural changes is one of the hardest things to accomplish. In this talk, Amy O’Connor will highlight the journey Nokia is taking to evolve its culture - from building a platform for cultural evolution on top of Hadoop, to the administration of Nokia’s data, to how the company conducts the analysis that is enabling Nokia to compete with data.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Hadoop World 2011: Changing Company Culture with Hadoop - Amy O'Connor, Nokia

  1. 1. Using Hadoop to Change Company Culture Amy O’Connor Senior Director, Analytics1
  2. 2. The Amazing Everyday2
  3. 3. The Amazing Everyday3
  4. 4. NOKIA’S HISTORY: 1865 TO Now4
  5. 5. Data is our newest raw material5
  6. 6. TRAFFIC US Government data suggests worsening conditions in urban areas…this yearaverage commute timehas risen by 9 minutes. Federal Highway Commission, Urban Congestion Report January 2011 to March 2011 6
  7. 7. TRAFFICGlobal car sales are growing: there will be 50% more carssold in 10 years as there are today. IHS Automotive, October 20117
  8. 8. More data usually beats better algorithms Anand Rajaramann8
  9. 9. Devices in • Image Sensors use around • Accelerometers the world • Gyroscopes • Compasses • Pressure Sensors Probe points • Microphones collected monthly from • Light Sensors Nokia alone • Assisted GPS9
  10. 10. 18M 24 80M10
  11. 11. Data Storage & Analysis Landscape11
  12. 12. Data Silos Traffic Search Consumer Probes Logs Profile Ad Places Device Data Registry Data12
  13. 13. Smart Data Combining sets of behavioral & contextual data13
  14. 14. A Good Way to Change Corporate Culture14
  15. 15. Getting Children to Eat Peas… Tell them you expect them to eat their peas. Reward them with ice cream if they did. Explain why it’s good for them to eat their peas. Eat your own peas as a good role model. Leann Lipps Birch, Head of Human Development & Family Studies Pennsylvania State University15
  16. 16. Getting Children to Eat Peas… Put them with children who love peas. Change the stories they tell. Leann Lipps Birch, Head of Human Development & Family Studies Pennsylvania State University16
  17. 17. Identity Media Care & Marketing Consumer Profile Products, Transactions Device/User CRM SSO Songs, Delivery Activation Info Device Activation Net Promotor Advertising Campaigns PC Suite Ad Inventory Contacts Campaign Promotions Location Navteq Ad Canvas Premium Content All Probes Data Favorite Routes 3D Imagery Nokia Log Files Street View Data Map Tiles Feature Recognition Asset Imagery Device Programs Social Search NAC, IIA Social UGC Log Files Windows Phone Universal Share Points of Interest Panel data Journeys Data Nokia IT Equipment Master Event Info Registrations Factory Data Device Updates17
  18. 18. Users Analytics Decision- Domain Offline Predictive Makers Expertise Dashboards Analysis Analytics Data Domain Expertise, Key Value Analysts Statistical Skills TeraData Store Domain Expertise, Data Oozie Map Pig Hive Statistical Skills, Reduce Scientists Computer Science Flume HBas e HDFS Developers/ Domain Expertise, Scrib HBas Applications Computer Science e e FTP HBas e18
  19. 19. Collaborative Working Model Present Analytics Offline Predictive Analyze Dashboards Analysis Analytics and Aggregate Key Value TeraData Store Load Oozie Map Pig Hive Reduce Transform Flume HBas e Extract HDFS Scrib HBas e e FTP HBas Platform e19
  20. 20. Collaborative Working Model To BI Tools: Present DataOS, Customer Analytics SPSS, AD Hoc Structured Dashboard Tableau, Reporting Data Cognos Offline Predictive Analyze Hive QL Dashboards Mahout MR Analysis Analytics Rec, and and Pig Analysis Machine Engines Aggregate Queries Job Learning Key Value TeraData MR Agg Store Agg MR User Create Load Metadata Job and Job and Metadata Hive Catalog Data to Data to Interfaces Schema Oracle Teradata Oozie Map Pig Hive Monitor Reduce Create Data Develop Develop Develop Transform Library of and Model Cleanse Validation Partition MR Jobs Manage Definition Job Job Job Transform Flume HBas Catalog Define Monitor Integrate Integrate e Define Extract Data Standard and Manage Historical HDFS Streaming Custom Sources ETL Scrib Data Data ETL HBas Data Feed e e FTP HBas Platform Co-located developer clusters, pre-production cluster, product cluster e Data OS Product Teams and/or DataOS20
  21. 21. Smart Data: Behavioral & Contextual Analytics Aggregation • Update top searches table and Aggregation table in Oracle geo activity Dashboards Offline Predictive Analysis Analytics • Merge multiple data sources • Implement app logic (e.g.; round up Key Value Standard ETL latitude/longitude to TeraData Store • Clean data, remove bad record 3 decimals) and health checks • Partition data by date, hour, type Map Oozie Pig Hive • Archive raw data Reduce Flume HBas Ad Router e HDFS Scrib HBas NAC e e FTP HBas Local Search e21
  22. 22. Merchant Portal Heatmap22
  23. 23. Mapping the World 23 © 2011 Nokia Company Confidential23
  24. 24. Mapping the World Probe data indicates the location, speed, heading, time etc. about a mobile device. Billions of probe records per week. Covers almost the entire world. 24 © 2011 Nokia Company Confidential24
  25. 25. Probe Density: Urban and Arterial 100% AGR25
  26. 26. 792527719 (Jackson Blvd/Financial Pl) From Ref, One lane26
  27. 27. 27
  28. 28. 24 Hours in Our Analytics Ecosystem ~2TB ingested 350M messages via scribe >3000 MR jobs 10TB processed28
  29. 29. Spreading the Word Data Asset Catalog Smart Data Newsletter Stories Realtime Dashboards29
  30. 30. The Amazing Everyday Thanks!30