Mike keating - News Int - 18th BDL meetup


Published on

Slides from Mike Keating's, Big Data Project lead at News International, talk on Big Data at News. 18th Big Data London meetup.

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Mike keating - News Int - 18th BDL meetup

  1. 1. Big Data at News International!Welcome !Big Data at News International!!Big Data Event - 29/05/2013!Mike Keating: Product Owner!@mikerkeating!  Slide 01/14!
  2. 2. Big Data at News International!Introduction!•  Where to start!•  Data and decisions!•  Our technology choices!•  Lessons!!Big Data Event - 29/05/2013!Slide 02/14!
  3. 3. Big Data at News International!Where to start…!Big Data Event - 29/05/2013!Slide 03/14!Lots of data, suppliers, technology and teams!!Take control, bring digital data into one place!Link data sets and look for new approaches!!Make data/ information/ knowledge/ insight available!Build awareness and decision making capability!  
  4. 4. Big Data at News International!Making the basics available via dashboards!Big Data Event - 29/05/2013!Slide 04/14!
  5. 5. Big Data at News International!Understanding our content and consumption!Big Data Event - 29/05/2013!Slide 07/14!Visits per VisitorViewsperVisitBehaviours Across Website Sections
  6. 6. Big Data at News International!Understanding our content in a social world!Big Data Event - 29/05/2013!Slide 08/14!
  7. 7. Big Data at News International!Tracking  subscrip.on  growth  across  products!Big Data Event - 29/05/2013!Slide 05/14!Days Following First SubscriptionSubscriptionsProduct Growth Following Launch
  8. 8. Big Data at News International!Analysing attributes that indicate churn!Big Data Event - 29/05/2013!Slide 06/14!
  9. 9. Big Data at News International!Designing products based on patterns in navigation !The  iPhone  –  Octopus  Naviga4on  The  Website  –  Flower  Petal  Naviga4on  The  iPad  Edi4on  -­‐  Linear  Naviga4on  Big Data Event - 29/05/2013!Slide 10/14!
  10. 10. Big Data at News International!Our technology choices – what?!Big Data Event - 29/05/2013!Slide 11/14!Infrastructure: AWS EC2, S3, RDS, EMR, Cloudformation, Vagrant!Ops: Jenkins, Anthill Pro, Maven, Nexus, Zabbix, CloudWatch!Code & Config: Puppet, Github!Data Retrieval: Java & Python!Data Pipeline: Java Map Reduce, Apache Crunch, !Spark, MRUnit, Celery, RabbitMQ, Python, Flume!Data Storage: HDFS, HBase, AWS S3, MySQL, Redis!Data Schema: Avro!Data Access: Python APIs, Tornado, S3, Hive & AWS EMR!Analysis: R, Pandas, Excel -> analyst’s choice!Visualisation: R, D3, Highcharts, Google Charts!Products: APIs, JS+HTML+CSS!
  11. 11. Big Data at News International!Our technology choices – why?!Big Data Event - 29/05/2013!Slide 11/14!•  Team: Build on team’s skillsets and knowledge. !•  Recruitment: Be conscious of “hire-ability”!•  Open Source: Big wins from usage; greatcommunities; contribute back!•  Versions: use what works; work with alpha releasesbut not as production code!•  Consistency: Try and use what you do today, e.g.AWS!•  Flexibility: Use a “better” product where it makessense!
  12. 12. Big Data at News International!Our technology choices – who?!Big Data Event - 29/05/2013!Slide 11/14!•  Tech Lead: Architecture; Design; Hands-On!•  Delivery Manager: Project Management w Agile!•  Hadoop: build from Java Map Reduce !•  Python: Tornado, Native Python, real-time processing!•  Data Science: Hive, R, Modeling!•  DevOps: AWS, Vagrant, Puppet!•  Experience: Practical experience of Hadoop in production!•  Capability: Ability to learn new tech, design and build!•  Demo: contributions to projects, working examples!
  13. 13. Big Data at News International!Lessons!Big Data Event - 29/05/2013!Slide 12/14!•  Building awareness & common knowledge!•  Building on existing teams, systems and their work!•  Looking for extra capability and output!•  Focus on visuals – it needs to be sharable/ visible!•  Working with range of teams to share outputs!•  Making good tech choices!
  14. 14. Big Data at News International!Thanks!Big Data Event - 29/05/2013!Slide 13/14!Big Data Team – DevOps, Hadoop, Python, UI,Analysts, Test!!Technology Teams – Design, Production Ops, PerfTesting, Security, Products, Platforms, Service Desk!!Editorial, Marketing, Commercial, Finance Teams!  
  15. 15. Big Data at News International!!!Thanks!Big Data at News International              Contact  Us  News  Interna.onal  Technology  @techatni  Mike  Kea.ng;  Product  Owner  @mikerkea.ng  Jobs  via  hBp://joinnitech.co.uk/  Big Data Event - 29/05/2013!Slide 14/14!