eSynergy - Windows Azure: Introduction to big data and hadoop


Published on

eSynergy Solutions was the Gold sponsor for the UK Microsoft Open Source Cloud event held in June 2012.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • A zettabyte is equal to 1 billion terabytes.1,000,000,000,000,000,000,000 bytes = 1021 bytesAs of 2009, the entire Internet was estimated to contain close to 500 exabytes.[7] This is a half zettabyte.
  • Coined by Gartner in 1990 as a challenge - Volume, Velocity, VarietyNow represents a huge opportunity
  • There has never been a more exciting time with respect to the world of data. We are seeing the convergence of significant trends that are fundamentally transforming the industry and a new era of tech innovation in areas like social, mobile, advanced analytics and machine learning. We’re seeing an explosion of data, there is an entirely new scale and scope to the kinds of data we are trying to gain insights from.  There’s a lot of talk about this – estimates are that the total amount of digital information in the world is increasing 10X every 5 years, with 85% of this data coming from new data types eg. Sensors, RFIDs, WebLogs etc. This presents a huge opportunity for Businesses that tap into this new data to identify new opportunity and areas for innovation. However, having a platform that supports the data trend is only part of today’s challenge; you need to also make it easier for people to access so that they can infer insight and make better decisions. If you think about the user experience, with everything we are able to do on the Web, our experiences through social media sites, how we’re discovering, sharing, and collaborating in new ways – User Expectations of their business and productivity applications are changing as well. 
  • Today new types of questions are being asked to drive the business. These questions include: Questions on what is being said on Social sites or about consumer behavior on the Web:How effective is my online campaign? Who am I reaching? How can I optimize or target the correct audience? What is my brand and product sentiment? There are also new set of questions that that require connecting to live data feeds. As examples: Shipping companies - analyze data on truck delivery times and traffic patterns to fine-tune routing.Retailers - analyze sales, pricing and economic, demographic and live weather data to tailor product selections at particular stores and determine the timing of price markdowns. There are also new Advanced Analytics in use today:Fraud detection is one area where credit card companies are utilizing new advanced analytical techniques to better detect fraudulent credit card transactions. Some of you may have experiences this first hand  New opportunity and competitive differentiation will come to those that can harness new data types to answers these new world questions.
  • Yahoo! (Gartner BI Excellence Award Winner) is driving growth for existing revenue streams:Yahoo! manages a powerful, scalable advertising exchange that includes publishers and advertisers.Advertisers want to get the most out of their investment by reaching their targeted audiences effectively and efficiently.Yahoo! needs visibility into how consumers are responding to ads alongmany dimensions (websites, creative, time of day, gender, age, location) to make the exchangework as efficiently and effectivelyas possible.Yahoo! doubled its revenue by allowing campaign managers to “tune” campaign targeting and creative.Yahoo! drove an increase in spending from advertisers since they got better performance by advertising through Yahoo!.Yahoo! TAO exposed customer segment performance to campaign managers and advertisers for the first time.GE is driving operational efficiencies:GE is running several use cases on its Hadoop cluster while incorporating several different disparate sources to produce results. Along with sentiment analysis, GE is running web analytics on its internal cloud structure and looking at load usage, user analytics, and failure mode analytics. GE built a recommendation engine for its intranet involving various press releases users might be interested in based on their function, user profiles, and prior visits to its site. GE is working with several types of remote monitoring and diagnostic data from energy and wind businesses.GE is developing technologies to help minimize bottlenecks in train traffic and reduce air travel disruptions from unscheduled engine maintenance, Ruh said. That will enhance the service offerings that accounted for 45 percent of last year’s $94 billion of industrial revenue, according to GE. Sensors on 6,400 GE wind turbines are already reporting data monitored in control centers in Salzbergen, Germany, and at the headquarters of its renewable energy business in Schenectady, New York. For example, the software can look at the electrical signature generated when a gas turbine starts up, said Courtney. “The question is what is the digital signature of that load in normal startup mode and then what happens if there’s an anomaly? Have you ever seen that anomaly before? Given this waveform, you can go back five years and look at other anomalies and whether they were part of a subsequent system failure.” If similar anomalies caused a system failure, you can examine how much time it took after the anomaly for the failure to happen. That kind of data lets the manufacturer prioritize fixes. Klout is creating new businesses and revenue streams:Klout’s mission is to help everyone understand and leverage their influence. Klout uses Big Data to unify the social web (consumers, brands, and partners) with social networking and activity, along with data to generate a Klout score and enable analysis, targeting, and social graphs.Helps consumers manage their “social brand.”Helps brands reach influencers at scale.Helps data partners enhance their services (customer loyalty, CRM, media and identity, and marketing). For example, the Palms uses Klout scores in addition to their normal customer rewards program to determine whether or not to upgrade their customers to a better room during their stay. The Huffington Post uses Klout to help serve the best curated Twitter content.Klout Case Study:
  • ,
  • eSynergy - Windows Azure: Introduction to big data and hadoop

    1. 1. What’s the social sentiment How do I better predictfor my brand or products future outcomes? How do I optimize my fleet based on weather and traffic patterns?
    2. 2. Measures and ranks online user Identify faults in gas turbines Increases ad revenue by processing influence by processing more than before they happen 3.5 billion events per day 1 billion signals per day Massive Cloud Near Real-Time Volumes Connectivity Insight Processes 464 billion rows per Connects across 15 social networks Receive signals from turbines and quarter, with average query time via the cloud for data and API compare to normal signals and to under 10 secs. access ones when fault subsequently occured1. Klout Case Study:
    3. 3. 0207 444