Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big Data and Business Insight


Published on

Ian Meyers, Senior Manager, Solutions Architecture, Amazon Web Services & John Kundert, CTO at the Financial Times

Published in: Technology

Big Data and Business Insight

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. November 8th, 2016 Big Data & Business Insight Ian Meyers - Senior Manager, Solutions Architecture, AWS John Kundert - CTO, The Financial Times
  2. 2. Why Cloud for Big Data & Business Insight?
  3. 3. Modern Analytics Applications Variety, volume, and velocity requiring new tools New analysis requirements Cloud Computing Variety of compute, storage, and networking options
  4. 4. Modern Analytics Applications Potentially massive datasets Massive datasets Cloud Computing Massive, virtually unlimited capacity
  5. 5. Modern Analytics Applications Iterative, experimental style of data manipulation and analysis Need for greater agility Cloud Computing Iterative, experimental style of infrastructure deployment/usage
  6. 6. Modern Analytics Applications Frequently not steady-state workload; peaks and valleys Variable workloads and volume Cloud Computing At its most efficient with highly variable workloads
  7. 7. Modern Analytics Applications Absolute performance not as critical as “time to results”. Shared resources are a bottleneck Projects require fast time to results Cloud Computing Parallel compute projects allow each workgroup to have more autonomy, get faster results
  8. 8. One tool to rule them all
  9. 9. Use the right tools for you Virtually unlimited Flexible, strong security Your data format & structure Powerful and fast Selection of tools Pay for what you use Rapid data discovery Pixel perfect reporting Third party & AWS Tools Storage Analysis Presentation
  10. 10. Use the right tools for you Block & Filesystem Storage Managed RDBMS Managed NOSQL Managed data warehouse Data streams SQL for analysis & transaction processing Managed Hadoop/Spark Predictive analytics & Deep Learning Data Workflow Rapid development & data discovery Kibana dashboards Industry leading third party solutions Storage Analysis Presentation Amazon
 S3 Amazon Glacier Amazon Elastic Filesystem Amazon
 RDS Amazon 
 Redshift Amazon EMR Amazon Kinesis Amazon QuickSightAmazon Elasticsearch Service Amazon
 DynamoDB Amazon DynamoDB Amazon Machine Learning
  11. 11. Online Software Store
  12. 12. Media streaming Marketing campaigns Disaster recovery Web site & media sharing Facebook app Ground campaign SAP & SharePoint Marketing web site Social Media Monitoring Consumer social app IT operations Mars exploration ops Interactive TV apps Consumer social app Facebook presence Securities Trading Data Archiving Financial markets analytics Web and mobile apps Big data analytics Digital media Ticket pricing optimization Streaming webcasts Mobile analytics Consumer social app Core IT and media
  13. 13. Certifications and accreditations for workloads that matter AWS CloudTrail - AWS API call logging for governance & compliance Log and review user activity Store data in S3, archive to Glacier, or stream process with AWS Lambda AWS Security AWS CloudTrail
  14. 14. Highlighted Customer Stories – Regulatory Agencies FINRA, the primary regulatory agency for broker-dealers in the US, uses AWS extensively in their IT operations and has migrated key portions of its technology stack to AWS including Market Surveillance and Member Regulation. For market surveillance, each night FINRA loads approximately 35 billion rows of data into Amazon S3 and Amazon EMR to monitor trading activity on exchanges and market centers in the US. In response to the May 6, 2010 Flash Crash in U.S. markets, the SEC used Tradeworx  and AWS to create its Market Information Data Analytics System (MIDAS), which enables the agency to collect and analyze billions of rows of data and to reconstruct any market event down to the individual record, analyzing more than 3 billion data points in seconds rather than weeks or months. For our market surveillance systems, we are looking at about 40% [savings with AWS], but the real benefits are the business benefits: We can do things that we physically weren’t able to do before, and that is priceless.” – Steve Randich, CIO
  15. 15. Japan’s largest mobile service provider 125 node Amazon Redshift DS2.8XL cluster 4,500 vCPUs, 30 TB RAM 2 PB compressed 10x faster analytic queries 50% reduction in time for new BI application deployment Significantly lower operations overhead 68 million customers Tens of TBs per day of data across a mobile network 6 PB of total data (uncompressed) Data science for marketing operations, logistics, and so on Scaling challenges Performance issues The Challenge The Solution
  16. 16. A case study of a transformative digital business model @ 
 John Kundert, CTO
  17. 17. Financial Times Profile ❏ Age: 128 years ❏ Size: ~300 million GBP / ~2000 employees ❏ Location: Global operations with UK at the centre ❏ Challenge: Transformation from print to digital ❏ Strategy: Grow paid for content business models
  18. 18. Challenges transforming into a digital business ❏ Putting the customer at the heart of the organization ❏ Building a world class Product team ❏ Building a world class Engineering team ❏ Getting Data into the heart of the organization ❏ Embedding change into the culture - move fast, be less risk adverse, embrace failure
  19. 19. Open for editorial
  20. 20. Open for customers
  21. 21. Open for knowledge managers
  22. 22. Adding a little colour - factoids Production system 30 TB Speedy Analytics 3 TB 91k queries day 700M records per day 520+ users best case 1 second latency
  23. 23. Our Data Story Certainty
  24. 24. The beautifully reassuring illusion of control • We consulted with the whole business • We defined a business case • We executed an RFP • We asked for a bit more money • We secured our preferred partner(s) • We started legal contracts and formed a team • We performed SSA (source system analysis) • We … Senior management striving for certainty …
  25. 25. No escape velocity The world around us changed faster than our definition of the ‘required change’
  26. 26. Our Data Story
 or Trust - letting go (a little)
  27. 27. 1. We re-organised ourselves
  28. 28. 2. We formed a vision statement ‘enable all of FT and its products to easily discover and trust our data and business intelligence in near real time’
  29. 29. 3. We defined measureable outcomes a. ownership of IP b. reduction in cost (TCO) c. reach d. return
  30. 30. 4. We built a small diverse team ❏ familiarity with the business including our heritage and culture ❏ technical diversity; ❏ traditional data warehousing ❏ software/front end developers ❏ analytical expertise ❏ tech lead with light touch support functions
  31. 31. 5. We empowered the team - ownership ❏ owned the long term vision ❏ owned the technology and the platform ❏ owned the business relationships ❏ managed cost constraints ❏ shared success and kudos
  32. 32. 6. Executive accepted short term (low cost) risk Team empowered to achieve outcomes through their choices start fast fail/succeed learn from results share
  33. 33. 7. Communication based on quality not quantity Multi-channel Full transparency at all times Pull and Push Ask for help when needed Short and clean Open Failure treated as success
  34. 34. The Unguided Missiles
  35. 35. Data Governance and Quality No agreed definition for; ❏ customer ❏ engaged customer ❏ active subscriber ❏ ... Resolution does not lend itself to agile methodologies. DQ image: Sourced from information management group
  36. 36. Cross-programme interdependence Dependencies outside the data programme of work to build; ❏ service levels ❏ new integration points ❏ migration from batch to API’s ❏ change of source
  37. 37. Outcomes
  38. 38. Moving costs from engineering to business value
  39. 39. Reach / Discovery o FT boardroom o FT newsroom o B2B and B2C business o Advertising o Analytics o Product development o Technology
  40. 40. Return / Value • Informed changes in the editorial workflows - moving towards a digital first production process • Calibrated the B2B paid for content business model - free vs paid • Predictive analytics - leading indicators driving B2C subscriptions • Optimizing the subscription models - metered model to trial model • Product Development - moving from hunches to following the data • Cultural change - evidence based accelerating our ability to change - test more / talk less
  41. 41. Governance / Trust o universal definition with data governance for all core metrics o enterprise wide adoption of new metrics for Engagement o subscription wide adoption of new lifetime value metrics o product and investment decisions based on an agreed version of the truth
  42. 42. Learn from other AWS customers Big Data Case Studies
  43. 43. APN Partner-provided labs AWS Big Data Test Drives
  44. 44. Webinars, Bootcamps, and Self-Paced Labs New course on Big Data AWS Training & Events
  45. 45. Thank You! Ian Meyers - Senior Manager, Solutions Architecture, AWS John Kundert - CTO, The Financial Times