How to Build a Successful Converged Data Platform
•Download as PPTX, PDF•
0 likes•701 views
Big data presents both enormous challenges and incredible opportunities for companies in today’s competitive environment. To deal with the rapid growth of global data, companies have turned to Hadoop to help them with performing real-time search, obtaining fast and efficient analytics, and predicting behaviors and trends. In this session, we’ll demonstrate how we successfully leveraged Hadoop and its ecosystem components to build a converged data infrastructure to meet these needs.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (8)
Similar to How to Build a Successful Converged Data Platform
Similar to How to Build a Successful Converged Data Platform (20)
More from MapR Technologies
More from MapR Technologies (17)
Recently uploaded
Recently uploaded (20)
How to Build a Successful Converged Data Platform
- 1. © 2016 MapR Technologies 1© 2016 MapR Technologies Xactly: How to Build a Successful Converged Data Platform with Hadoop, Spark, Solr and More
- 2. © 2016 MapR Technologies 2 Today’s Presenters Steve Wooledge VP, Product & Digital @swooledge Kandarp Desai Director of Engineering @kandarpdesai
- 3. © 2016 MapR Technologies 3
- 4. © 2016 MapR Technologies 4 Big Data Meets the As-it-Happen World 1 Billion 6 Billion 50 Billion 2000s: Mobile Internet 2020: Internet of People and Things 1990s: Fixed Internet Connected Devices Worldwide By 2020, 21% of all “high value” data will come from IoT - IDC
- 5. © 2016 MapR Technologies 5 Legacy • Complex and slow • Multiple versions of the truth • Difficult governance • High TCO • Real-time data to action • Single data copy • Easy governance • Low TCO – scales horizontally Analytics Operations Data Data HTAP (Hybrid Transaction/Analytical Processing) – Gartner 2015 Data Modern Apps
- 6. © 2016 MapR Technologies 6 Use Cases by Industry
- 7. © 2016 MapR Technologies 7 A Once-in-30-Year Shift in Data Architecture Critical infrastructure for next-gen business processes Next-Gen Applications Legacy Applications Open Source Analytic Innovations Legacy Disruptive Data Platform On Premise Private Cloud Public Cloud Heterogeneous Hardware Next-Gen Data Platform
- 8. © 2016 MapR Technologies 8 Leading Research Sees Data Platform Convergence “…we expect data-platform contraction to be driven by convergence of the various approaches to data processing and analytics. A number of 451 Research enterprise clients are already in the process of assembling what we might call ‘converged data platforms,’ combining operational and analytic databases with data grid/cache technologies, Hadoop and stream- processing technologies.” - Matt Aslett, 451 Research Toward a Converged Data Platform, Dec 2015
- 9. © 2016 MapR Technologies 9 Next-Gen Application Requirements Customer Experience Data Architecture Optimization Security Investigation & Event Management Operational Intelligence Managed Services & Custom Apps AppsProcessing Batch Interactive Streaming Transactions Storage Data Data Platform/Storage
- 10. © 2016 MapR Technologies 10 Silos – Point Solutions Customer Experience Data Architecture Optimization Security Investigation & Event Management Operational Intelligence Managed Services & Custom Apps AppsProcessing Batch Interactive Streaming Transactions Storage Data HDFS NoSQL Event Streaming RDBMS SAN / NAS
- 11. © 2016 MapR Technologies 11 MapR Solution: Common Data Services for All Applications Customer Experience Data Architecture Optimization Security Investigation & Event Management Operational Intelligence Managed Services & Custom Apps AppsProcessing Batch Interactive Streaming Transactions Storage Data HDFS NoSQL Event Streaming RDBMS SAN / NAS MapR Converged Data Platform • Combined analytic and operational data • Single copy of data not silos • Unified platform rather than separate point solutions
- 12. © 2016 MapR Technologies 12 Open Source Engines & Tools Commercial Engines & Applications Enterprise-Grade Platform Services DataProcessing Web-Scale Storage MapR-FS MapR-DB Search and Others Real Time Unified Security Multi-tenancy Disaster Recovery Global NamespaceHigh Availability MapR Streams Cloud and Managed Services Search and Others UnifiedManagementandMonitoring Search and Others Event StreamingDatabase Custom Apps HDFS API POSIX, NFS HBase API JSON API Kafka API MapR Converged Data Platform
- 13. © 2016 MapR Technologies 13 <1% MapR: the Production Choice for Big Data Applications Best Product High Growth >100% Growth 18% Customers with >50 apps** 382% Avg. 3-yr ROI* 700+ CustomersBig Data Converged Data Platform Apache Open Source Churn + Innovation * IDC – “The Business Value of MapR”, 2016. ** - TechValidate Research, 2015
- 14. KANDARP DESAI DIRECTOR OF ENGINEERING TWITTER @kandarpdesai How to Build a Successful Converged Data Platform with Hadoop, Spark, Solr and More
- 15. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. This presentation and the accompanying oral presentation contain “forward-looking” statements that are based on our management’s current expectations and projections about future events and trends that we believe may affect our business, financial condition, operating results and growth prospects. Forward-looking statements include all statements other than statements of historical fact contained in this presentation, including information relating to future events or our future financial or operating performance, such as our future product release dates. Forward-looking statements are subject to substantial risks, uncertainties and other factors. These factors, together with those that may be described in greater detail in a registration statement (including a prospectus) that we may subsequently file with the Securities and Exchange Commission (“SEC”) for the transaction to which this presentation relates, may cause our actual results, events, or circumstances to differ materially from those described in our forward-looking statements. You should not rely upon forward-looking statements as predictions of future events. Our forward-looking statements relate only to events as of the date on which the statements are made. We undertake no obligation to update any forward-looking statements to reflect events or circumstances after the date of this presentation or to reflect new information or the occurrence of unanticipated events, except as required by law. In addition to financial measures prepared in accordance with generally accepted accounting principles in the United States (“U.S. GAAP”), this presentation includes certain non- GAAP financial measures. We believe that these non-GAAP financial measures are useful as a supplement in evaluating our ongoing operational performance and enhancing an overall understanding of our past financial performance. The non-GAAP financial measures included in this presentation should not be considered in isolation from, or as a substitute for, financial information prepared in accordance with U.S. GAAP. A reconciliation between each non-GAAP financial measure and its nearest GAAP equivalent is included at the end of this presentation. We are an “emerging growth company” as defined under the Securities Act of 1933, as amended (the “Act”). This presentation and the accompanying oral presentation are intended to qualify as communications permitted pursuant to Section 5(d) of the Act. We may file a registration statement (including a prospectus) under the Act with the SEC for the transaction to which this communication relates. In the event we conduct an offering, before you invest you should read the prospectus in the registration statement and other documents we file with the SEC for more complete information about us and the offering. When available, you may get these documents for free by visiting EDGAR on the SEC website at http://www.sec.gov. SAFE HARBOR 15
- 16. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 16 COMPENSATION IS A GLOBAL PROBLEM
- 17. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 17 IT IS LARGELY A MANUAL PROCESS THAT Requires too much personal attention and takes too long Generates too many errors Specific Pain Points: Time consuming, manual process for mid-level analyst…takes 16-24 hours/month 3% Errors on $750,000 commission budget = $22,500 Reps lack visibility into performance, earnings Executive reporting is manual, as is accruals Increased plan complexity and growing sales team
- 18. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 18 BEST OF BREED APPROACH HCM ERP CPQ CRM
- 19. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 1919 Multi-level horizontal scalability App1 Calc1 App2 Calc2 App3 Appn Pod-1 Pod-2 Single SaaS System Multiple Pods Calculation Application Calc3 Calcn App1 Calc1 App2 Calc2 App3 Appn Calc3 Calcn RDBMS Storage Pod-N App1 Calc1 App2 Calc2 App3 Appn Calc3 Calcn DB Nodes DB Nodes DB Nodes Cache Cache1 Cache1 Cache1 Cache1 Cache1 Cache1 Cache1 Cache1 Cache1 Cache1 Cache1 Cache1
- 20. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 20 CRUD to MapR-FS or MapR-DB Async Event(s) Building Data Pipeline : RDBMS → Hadoop MapR Cloudera Hortonworks Apache Hadoop Core functionalities Storage 7 5 4 X Cluster Management 7 6 5 X Data Access 8 6 2 X Strategy & Market Presence Support & PS 8 6 X X Roadmap 9 5 X X Adoption X X X X Pricing & Company Customer Count X X X X Employee Base X X X X NOTES: • Scores in table on scale - 1: lowest; 10 Highest • Actual table contains many more criteria RDBMS
- 21. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 21 Building Data Pipeline : RDBMS Hadoop (MapR-FS) RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS POD 1 POD 2 POD N CRUD to MFS or MapR-DB • Billions of Calculation Results Per Month • Thousands of Events Per Minute • Homegrown Workflow and Event Management • Combines RDBMS ACID and MapR-FS Snapshots to Achieve Immutable Copy of Data • Evolution from Workflow Management to Pub/Sub System Such As Kafka
- 22. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 22
- 23. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 23 • Group Transaction Events In Logical Buckets • Efficient Data movement • Easier To Detect Missing Data ( If any ) • Re-Broadcasting Is Append Only Event • Platform First , Product Second Approach • Don’t Take Data Validation Lightly • Investing More Up-Front = Better Future • Choosing Perfect Hadoop Distribution Is Not TRIVIAL • Do Not Underestimate Power Of Snapshots Lessons Learned
- 24. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 24 Building Data Pipeline : Data Platform Processing RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS POD 1 POD 2 POD N CRUD to MFS or MapR-DB • Process 11 Years of Empirical Data • Billions of Rows ; 10s TBs of Data • Custom MapReduce Framework Running 1000s of Jobs • Types of Operations : Multiple Types of Joins, Aggregation at Lowest Level, etc. • Prepares Data for Product and Data Science Team Batch Map- Reduce / ETL Converged Data Platform
- 25. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 25 Building Data Pipeline : Batch SPARK Processing RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS POD 1 POD 2 POD N CRUD to MFS or MapR-DB • Batch Spark Jobs Running On-Demand and Weekly • Types of Operations : Multiple Types of Joins, Aggregation for application • 100s GBs of Data ; Billions of Records • Was Map-Reduce (~ 17 Hours) -> Now Spark Jobs (~6 hours) • Was Hive ( 10 Hours ) -> Now Drill ( ~ 2 hours ) • Off-line Data Science Models Batch Map- Reduce / ETL Batch SPARK Processing Converged Data Platform
- 26. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 26 Building Data Pipeline : Real-Time SPARK Processing RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS POD 1 POD 2 POD N CRUD to MFS or MapR-DB • Spark Data Processing for Real-Time Web Benchmarking App • Real-Time Data Science Models Processing Such as GLM • Types of Operations : Average, Percentile, Distributions, others • Long Running Spark Context • Varieties of RDD Caching Techniques to Speed Calculation Batch Map- Reduce / ETL Batch SPARK Processing Agg. Stored on MapR-DB Serving Data to Benchmarking Application Converged Data Platform
- 27. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 27 Xactly Insights : Web Application Real-Time Calculation Of Custom GLM Under 2-3 Seconds Response Time Calculates Percentiles Real-Time Under 3 Seconds Response Time
- 28. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 28
- 29. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 29
- 30. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 30 Building Data Pipeline : Spark Transaction Processing RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS POD 1 POD 2 POD N SQOOP transfers data to MapR-DB • Short Lived Spark Context Per Tenant • 100 GBs of Data Processing Per Business • Replacing Store Procedures • 2.5x Faster Processing Speed With Spark Spark Processing Generates Results RDBMSRDBMS RDBMSRDBMS RDBMSRDBMS POD 1 POD 2 POD N RDBMS RDBMS RDBMS Converged Data Platform
- 31. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 31 Building Data Pipeline : Real-Time Search RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS RDBMSRDBMSRDBMS POD 1 POD 2 POD N CRUD to MFS or MapR-DB • ~ Real Time Search On Thousands of Standard and Custom Fields • Types of Operations : Multiple Types of Joins an Mappings • 100s of Small Map-Reduce Jobs / 10 minutes • ~100 GBs Data Size / 5 minutes MapReduce Prepares Data for Solr Injection SOLR Engine Converged Data Platform
- 32. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. 32 • Design & Build For Future Not Only Present • Big Data Frameworks : Easy to Use > Extremely Hard to Master • MapR-DB vs MapR-FS • Never Settle for Default Configuration ; Customization Can Make Life Much Better • Do Not Use SPARK Without Proper Understanding • Simple Debugging Can Consume Entire Sprint Or More • Memory Management In SPARK May Surprise You • Many More • SPARK SQL Is Good Though Developers Must Prefer Power Of SPARK Scala API. • Retire Hive & Adopt Drill Lessons Learned
- 33. ©2016 Xactly Corporation. All rights reserved. Proprietary & Confidential. Xactly’s vision is to change the world of incentive compensation. For more information, visit www.xactlycorp.com • WE ARE HIRING ! • kdesai@xactlycorp.com or https://www.xactlycorp.com/company/careers/
- 34. © 2016 MapR Technologies Q&AEngage with us! 1. Get Case Studies: Big Data All-Stars https://www.mapr.com/when-streaming-becomes-strategic 2. Get Started: MapR Converged Data Platform https://www.mapr.com/get-started-with-mapr 3. Get Answers: MapR Converge Community https://www.mapr.com/big-data-all-stars
Editor's Notes
- We live in an on-demand world. TiVo, Snapchat, instant messaging, Slack, texting, and more. Here is a picture of one of my kids who one day asked me to fast-forward the TV past a commercial. When I explained to her that “you can’t fast forward live TV”, she responded, “who invented live TV, that’s stupid!” …. That is the mentality of the new generations and enterprises need to be able to respond to customers and internal insights/applications in the same way … RIGHT. NOW.
- I think most people understand the story, but to put it in context, think about the amount of data that's being generated. IOT, eclipses, anything we've seen from the internet or mobile devices so far, now that you're instrumenting every single device, whether it's a Fitbit or it's a car or airplane engines, production lines, there's just this tsunami of data that systems were never designed to handle. At the same time, we live in this as it happens world where everything is on demand, and people expect instant access and activity and actions. You've got these competing sources coming at each other. You can't architect around that easily. I think that's why you saw projects like Kafka get created at LinkedIn. There was simply no system out there that could do the types of real-time data flows that they needed to have to keep their system relevant and fresh and real-time. That's the type of technology that's been developing in our market. We're bringing it to the enterprises in one converged infrastructure so that they can take advantage of the IOTs.
- Historically analytics and operations systems have been separate. With dedicated processes to extract, prepare, and load data introducing delays and administrative and security issues. The benefits of a converged approach is to combine analytics and operations into a single platform. Not only does this drive efficiencies, and reduce data duplication, but more importantly it eliminates delays and latency enabling real-time applications.
- There are literally hundreds of use cases or applications enabled by big data analytics and applications….
- Cover Slide should have a brief but compelling title, presenter name, and their job title.
- - 3% Errors on $750,000 commission budget = $22,500
- 870 + companies ; 266,000 Subscriber
- Let data speak to us ; Provide feedback to customers ; Not trivial; strategic decision;
- This is a basic content page consisting of just a headline. There are also content pages like this one with the Xactly logo at the bottom in the master pages.
- All presentations should end with this slide. It reinforces our brand, has our legal mark and contact info.