Hadoop in the Cloud: Real World Lessons from Enterprise Customers
•Download as PPTX, PDF•
2 likes•859 views
Hadoop in the Cloud: Real World Lessons from Enterprise Customers
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Hadoop in the Cloud: Real World Lessons from Enterprise Customers
Similar to Hadoop in the Cloud: Real World Lessons from Enterprise Customers (20)
More from DataWorks Summit/Hadoop Summit
More from DataWorks Summit/Hadoop Summit (20)
Recently uploaded
Recently uploaded (20)
Hadoop in the Cloud: Real World Lessons from Enterprise Customers
- 6. Storage Azure HDInsight R Server Local (HDFS) or Cloud (Azure Blob/Azure Data Lake Store) Analytics
- 7. Azure HDInsight Hadoop Meets the Cloud Microsoft’s managed Hadoop as a Service 100% open source Apache Hadoop Built on the latest releases across Hadoop (2.7) Up and running in minutes with no hardware to deploy Run on Windows or Linux Supported by Microsoft
- 9. Rockwell Automation is partnered with one of the six oil and gas super majors to build unmanned internet-connected gas dispensers. Each dispenser emits real-time management metrics allowing them to detect anomalies and predict when proactive maintenance needs to occur. Store sensor data every 5 minutes Temperature, pressure, vibration, etc. Tens of thousands of data points / second Azure Blobs Azure HDInsight Hive, Pig, Azure SQL DB Power BI for O365 Mobile Notification Hub Mobile Device Real-time notification
- 10. JustGiving wanted to harness the power of their data by using network science to map people’s connections and relationships so that they could connect people with the causes they care about. Based on 15 years of data, the JustGiving GiveGraph is the world’s largest ecosystem of giving behavior. It contains more than 81 million person nodes, thousands of causes and 285 million connections and is the engine that drives JustGiving’s social platform, enabling levels of personalization and engagement that a traditional infrastructure would be unable to deliver. SQL Server On-premises Agent Azure Blobs Azure HDInsight Give Graph Azure Tables Web APIWebsite + Event store Service Bus Serves results Azure Cache Activity Feeds
- 11.
- 14. *Pending IDC study found on a per TB basis, Microsoft customers using cloud-based Hadoop in Data Lake have a 63% lower TCO than on- premises
- 18. Always on cluster Cluster as a service Storage choice Local HDFS, Azure Blob, Azure Data Lake Store Azure Blob, Azure Data Lake Store Job Scheduling Oozie Azure Data Factory Data persistence after cluster deletion N/A Azure Blob, Azure Data Lake Store Metadata persistence after cluster deletion N/A Azure SQL
- 21. Partition 1 Partition 2 Partition 3 2014-10.part0 2014-11.part0 2014-12.part0 2014-10.part1 2014-11.part1 2014-12.part1 2014-10.part2 2014-11.part2 2014-12.part2 2014-10.part3 2014-11.part3 2014-12.part3
- 22. Partition 1 Partition 2 Partition 3 2014-10.part0 2014-11.part0 2014-12.part0 2014-10.part1 2014-11.part1 2014-12.part1 2014-10.part2 2014-11.part2 2014-12.part2 2014-10.part3 2014-11.part3 2014-12.part3
- 23. Partition 1 Partition 2 Partition 3 2014-10.part0 2014-10.part1 2014-10.part2 2014-11.part0 2014-11.part1 2014-11.part2 2014-12.part0 2014-12.part1 2014- 12.part2 2014-10.part3 2014-11.part3 2014-12.part3 Partition 4
- 24. No limits on file sizes Analytics scale on demand No code rewrites as you increase size of data stored Optimized for massive throughput Optimized for IOT with high volume of small writes PB TB GB PB TB
- 25. Goals
- 31. • • • •
- 33. Azure Security: Encryption At Rest Azure Blob Storage (In Preview) • Encryption @ rest using Microsoft managed keys • Customers can use Azure Storage configuration to manage encryption. No HDInsight changes required.
- 34. RBAC : Securing HDInsight with Blue Talon (ISV)
- 36. Deep integration to Visual Studio Easy for novices to write simple queries Robust environment for experts to also be productive Integrated with Pig, Hive, and Storm Playback that visualizes performance to identify bottlenecks and areas for optimization
- 37.
Editor's Notes
- Challenge Manage sites used for dispensing liquefied natural gas (clean fuel for commercial customers who do heavy-duty road transportation) Built LNG refueling stations across US interstate highway Stations are unmanned so they built 24x7 remote management and monitoring to track diagnostics of each station for maintenance or tuning Built internet-connected sensors embedded in 350 dispenser sites worldwide generating tens of thousands data points per second Temperature, pressure, vibration, etc. Data needs outgrew company’s internal datacenter and data warehouse Solution Chose Azure HDInsight, Data Factory, SQL Database Dashboards used to detect anomalies for proactive maintenance Changes in performance of the components Energy consumption of components Component downtime and reliability Future: Goal is to expand program to hundreds of thousands of dispensers How They Did It Collect data from internet-collected sensors Tens of thousands data points per second Interpolate time-series prior to analysis Stored raw sensor data in Blobs every 5 minutes Use Hadoop to execute scripts and Data Factory to orchestrate Hive and Pig scripts orchestrated by Data Factory Data resulting from scripts loaded in SQL Database Queries detect site anomalies to indicate maintenance/tuning Produced dashboards with role-based reporting Azure Machine Learning , SSRS, Power BI for O365 Provide users with customizable interface View current and historical data (day-to-day operations, asset performance over time, etc.) Leveraged Azure Mobile Notification Hub for real-time notifications, alarms, or important events
- JustGiving wanted to identify what was personal and relevant to people and what they cared about, so that they could suggest further causes that may inspire continual involvement. However, with 22 million customers this meant storing and processing huge amounts of data that their existing infrastructure simply couldn’t support. HDInsight, provided the scalable, ‘on-demand’ processing and analysis ability to assist JustGiving with its goal to constantly evolve the personalised experiences it provides to customers. JustGiving is a global online social platform for giving. It's a financial service (not a charity) that lets you "raise money for a cause you care about" through your network of friends. JustGiving's goal is to become "Facebook of Giving" JG preferred not to refer to Facebook as a way of describing themselves – they are using the term “social giving” and like to refer to themselves as a “tech for good” company. i.e. harness the effect to make charity a group activity which isn't just a onetime event but rather something you stay in touch with on regular basis. More details on charity goals in a blog post from JustGiving: http://blog.justgiving.com/creating-smart-targets-to-help-fundraisers-raise-more/ Technical Details: Workflow: There is one a set of daily HDInsight job that uses the data coming through SQL Server to build out the social graph and provides activity recommendations to the user. The input data is 20-30 GB/job but the output is in 'hundreds' of GBs as relationships are de-normalized/expanded. Azure Table Storage is used to serve 'News Feed' to users. The data in Table store come from two main sources: Real time activity feeds/events coming in from Azure Service Bus. (~50 events/second) Activity recommendation coming out of the daily HDInsight job There are several MR processes to create the graph; once that is done, further jobs create the denormalised activity feeds for all users. M/R jobs that do all the graph building.
- Wind turbines – used in windmills to turn wind into power These things are phenomenally complex devices They emit a ton of telemetry Every 25 milliseconds they emit 10 interesting datapoints 100x of wind mills (turbines) per windfarm They are managing about a hundred windfarms through their customers Their customers are the power companies SO yes this is a “IOT scenario” Couple of challenges How am I going to collect all this information – windfarms around the globe, some in remote areas, how to collect How to I store it cheaply – lot of data – honestly no single row interesting – only interesting in aggregate – but because there’s ns much of it putting in DW it is cost prohibitive Instead they landed data in cloud from around the world Cloud allowed storing it cheaply – just store no processing Then bring processing on top of it to distill and refine the data into something more powerful They cooked data down - refined, aggregate, duplicate detection Now they provided interesting metrics to power companies But since they had the source data around they can ask lots of interesting questions on *all* the data since they started collecting it Moved beyond charts and graphs bout how the windfarms were performing Now able to mash that together with maintenance data To be able to say what are some of the things correlated with my turbines failing These are very expensive pieces of machinery so if we can find things that are going awry *before* the whole thing breaks its usually much cheaper Now one of the interesting things they found was if they were more observant about dust conditions .. they were saving million dollar generators by replacing four dollar filters more often And so this is a interesting example of predictive maintenance
- Choose VM based on core size and memory
- Choose VM based on core size and memory
- Choose VM based on core size and memory
- Finding the right tools to design and tune your big data queries can be difficult. Data Lake makes it easy through deep integration with Visual Studio, so that you can use familiar tools to run, debug, and tune your code. Visualizations of your U-SQL, Hive, and Storm jobs let you see how your code runs at scale and identify performance bottlenecks and cost optimizations, making it easier to tune your queries.