Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IBM Smarter Analytics

1,216 views

Published on

  • Be the first to comment

  • Be the first to like this

IBM Smarter Analytics

  1. 1. © 2015 IBM Corporation IBM Smarter Analytics Big Data Adoption Adrian Turcu Big Data Architect IBM Client Innovation Centers RoCEB
  2. 2. © 2015 IBM Corporation2 Mobile Social Cloud Analytics The Mega Trends
  3. 3. © 2015 IBM Corporation3 Big Data: More than just volume Volume Terabytes to exabytes of existing data to process Velocity Streaming data, milliseconds to seconds to respond Variety Structured, unstructured, text & multimedia Veracity Uncertainty from inconsistency, ambiguities, etc.
  4. 4. © 2015 IBM Corporation4 Big Data & Analytics Value Proposition The primary value from big data and analytics comes not from the data in its raw form, but from the processing and analysis of it and the insights, decisions, products, and services that emerge from analysis.
  5. 5. © 2015 IBM Corporation5 IBM’s Commitment to Big Data  $16 Billion in Big Data acquisitions 35 new acquisitions in the last 5 years  More than 1000 developers focused on Big Data technology development  IBM joins Apple, Twitter, and the Weather Company in strategic partnerships  Largest patent portfolio in the industry  IBM has the largest commercial research organization on Earth ‒ 200+ mathematicians developing breakthrough analytics  IBM’s Big Data business grew over 150% in 2014 IBM CEO Says ‘Big Data’ Is Company’s Top Priority Commitment to Big Data Means… Commitment to Hadoop and Spark
  6. 6. © 2015 IBM Corporation6 IBM is Committed to Open Source  Open source technologies are the base for IBM software and solutions  IBM’s long history of deep open source commitment  Apache Software Foundation: Founding member in 1999  Cloud Foundry: #1 contributor; Basis for Bluemix  OpenStack: #4 contributor; Basis for IBM’s IaaS  Linux: #3 contributor; IBM first enterprise backer of Linux  Hadoop/Spark: Extensive investment in open source contribution; Integration with Analytics software Infrastructure Systems Application
  7. 7. © 2015 IBM Corporation7 IBM Investing in Four Catalysts for Big Data Adoption Familiar Interfaces & Integration with Established Tools Technical Standards New Analytics Capabilities Open Source Innovation
  8. 8. © 2015 IBM Corporation8 Apache Hadoop Ecosystem: Rapid Innovation, Few Standards  Distributions include different projects at different version levels “This proliferation of baskets [Hadoop distributions with different project versions] creates significant drag when it comes to building reliable applications ... makes it harder for customers to assess which basket of Hadoop that they need and harder for application developers to create solutions that work broadly.” – Raymie Stata, CEO, Altiscale  Even though the project versions match, there are interface differences If the industry is truly committed to developing big data technologies and solutions …, it will require an ecosystem of providers … to create a consistent framework around which everyone can develop. - Siki Giunta, SVP, Verizon  The Hadoop ecosystem is evolving at a faster pace than is comfortable “My personal speculation is that it comes from some who have been evaluating for a while seeing change occur so rapidly that they are dropping back for another look.” – Merv Adrian, VP, Gartner
  9. 9. © 2015 IBM Corporation9 Certify a standard “ODP Core” set of open source Hadoop family projects with specific versions and patch levels Develop tools and methods to help solution providers to test applications against the ODP Core. Contribute changes and fixes in the ODP Core Hadoop family projects to the ASF using the ASF processes. http://opendataplatform.org/
  10. 10. © 2015 IBM Corporation10 Open Data Platform Initiative Representation across the Hadoop ecosystem…  Hadoop distribution vendors  Software application providers  System integrators/consultants  Hardware vendors  Customers … who all believe in the need for a community-based effort to standardize Hadoop, which will lead to improved adoption
  11. 11. © 2015 IBM Corporation11 IBM Open Platform with Apache Hadoop (IOP)  100% open source code Apache Hadoop distribution  Commitment to currency: “days, not months”  Includes Spark  Free for production use  Decoupled Apache Hadoop from IBM analytics and data science technologies  Production support offering available Apache Open Source Components HDFS YARN MapReduce Ambari HBase Spark Flume Hive Pig Sqoop HCatalog Solr/Lucene IBM Open Platform with Apache Hadoop
  12. 12. © 2015 IBM Corporation12 Text Analytics POSIX Distributed Filesystem Multi-workload, multi-tenant scheduling IBM Biglnsights Enterprise Management Machine Learning on Big R Big R (R support) IBM Open Platform with Apache Hadoop (HDFS, YARN, MapReduce, Ambari, Hbase, Hive, Oozie, Parquet, Parquet Format, Pig, Snappy, Solr, Spark, Sqoop, Zookeeper, Open JDK, Knox, Slider) IBM Biglnsights Data Scientist IBM Biglnsights Analyst Big SQL BigSheets Industry standard SQL (Big SQL) Spreadsheet-style tool (BigSheets) Overview of Biglnsights (v4.x) . . .
  13. 13. © 2015 IBM Corporation13 IBM Open Platform with Apache Hadoop adopts ODP Core  BigInsights will include ODP certified Apache packages  ODP will initially target core packages of a Hadoop distribution  Packages will expand over time  First certification set expected this summer  Our goal for BigInsights on ODP  Better compatibility and less testing against ecosystem software  Enable IBM Hadoop capabilities to run on other ODP-certified Hadoop distributions HDFS YARN MapReduce Ambari HBase Spark Flume Hive Pig Sqoop HCatalog Solr/Lucene ODP * Candidate set of certified ODP modules – expected summer 2015 Apache Open Source Components IBM Open Platform with Apache Hadoop
  14. 14. © 2015 IBM Corporation14 Apache Spark is ideal for:  Machine Learning  Interactive analytics  Data Science http://spark.apache.org Spark is an open-source, in-memory compute engine that is highly versatile to any environment, enabling you to quickly build models, iterate faster, and apply deep intelligence everywhere. Apache Spark Overview Apache Spark Spark SQL Spark Streaming GraphX MLlib (machine learning) SparkR
  15. 15. © 2015 IBM Corporation15 IBM | Spark - The Start of Something Big in Data and Design Together, creating the platform for Data Science Understand Business Goal Data Profiling and Exploration Train Algorithms Consult Experts Prepare data App Dev, Deploy, Validate Go live. Refresh. +
  16. 16. © 2015 IBM Corporation16 IBM Analytic Platform Capabilities IBM Software Integrates and Extends Hadoop and Spark Data Warehousing PureData for Analytics, Operational Analytics Entity Extraction and Matching Big Match Security and Compliance Optim, Guardium Audit and Encryption Data Integration and Governance Information Server Enterprise Search Watson Explorer Real-time Analytics Streams Predictive Modeling and Descriptive Statistics SPSS, Big R and Scalable Algorithms Analysis, Reporting, and Exploration Watson Analytics, Cognos, BigSheets Fast, ANSI SQL 2011, and Secure SQL Big SQL Enterprise File System GPFS-FPO Cluster Resource and Workload Management Platform Symphony Large Scale Text Extraction Big Text IBM Open Platform with Apache Hadoop
  17. 17. © 2015 IBM Corporation17 What is IBM’s perspective on Spark?  IBM opens Spark Technology Center in San Francisco to foster innovation in the heart of the Spark community  IBM is forging key partnerships and building relationships with the creators of Spark  Big data university  Spark certification and Spark social badge  Databricks partnership  AMPLab partnership  IBM Analytics Platform will unify on and around Spark to ensure robust integration and ease of use for our clients  Biglnsights “Spark-Inside”  Spark as a Service on IBM Bluemix (beta in June 2015)  Streams and Spark integration
  18. 18. © 2015 IBM Corporation18 http://g01zcdwas002.ahe.pok.ibm.com/software/data/infosphere/hadoop/trials.html Free Quick Start (non production): • IBM Open Platform • Biglnsights Analyst, Data Scientist features • Community support
  19. 19. © 2015 IBM Corporation19 http://g01zcdwas002.ahe.pok.ibm.com/software/data/infosphere/hadoop/trials.html
  20. 20. © 2015 IBM Corporation20 IBM’s Investment in the Big Data Community Over 250,000 benefit from free Big Data skills training http://bigdatauniversity.com
  21. 21. © 2015 IBM Corporation21 Big Data ≠
  22. 22. © 2015 IBM Corporation22
  23. 23. © 2015 IBM Corporation23 Watson is creating a new partnership between people and computers that enhances, scales and accelerates human expertise.
  24. 24. © 2015 IBM Corporation24 Brief History of IBM Watson R&D Demonstration Commercialization Cross-industry Applications IBM Research Project (2006 – ) Jeopardy! Grand Challenge (Feb 2011) Watson for Healthcare (Aug 2011 –) Watson Industry Solutions (2012 – ) Watson for Financial Services (Mar 2012 – ) Expansion
  25. 25. © 2015 IBM Corporation25 IBM Watson is cognitive computing  Watson understands me.  Watson engages me.  Watson learns and improves over time.  Watson helps me discover.  Watson establishes trust.  Watson has endless capacity for insight.  Watson operates in a timely fashion. …built on a massively parallel Big Data scalable architecture
  26. 26. © 2015 IBM Corporation26 Many industries have a “discovery” challenge Drug discovery: ~12-15 yrs, $B per drug, 90+% fallout rate Lithium ion Battery: ~20 years development time Healthcare and Life sciences Chemical and Petroleum Drug Discovery New Biology Science New Bio- medical Research Oil Reservoir Discovery Crop Sciences New Energy Materials Product formation: based on Ad hoc manual trial & error Water filtration: Billions still do not have clean water today Consumer Goods and Products Semi-Conductor and Materials Product Innovation New Market Identification New Partnerships Nano Materials Energy Storage Water Filtration Existing Discovery is Slow, Expensive, Ad hoc and Manual
  27. 27. © 2015 IBM Corporation27 Bringing IBM Watson to market Watson Engagement Advisor Watson Discovery Advisor Watson Policy Advisor Watson Decision Advisor Offerings: Watson for Wealth Management Watson for Oncology Chef Watson Applications: Watson Explorer Watson Analytics Watson Curator Products Watson Zone on Bluemix Watson Developer Cloud Watson Tooling Platform:
  28. 28. © 2015 IBM Corporation28 Delivers the tools, methodologies, software developer kits and API(s) for ISVs to build the next generation of cognitive applications Provides sources of free and fee based content including public, industry and enterprise content Bridges developers resource gaps by providing a marketplace for critical cognitive skills • Cloud based sandbox • Hosting Services • Self-service portal – API / Tooling / SDK / Methodology • Starter Content • General content • Domain content • Taxonomies • Third-Party Content • IBM subject matter experts (500+) • Third-party specialists • Certification • Individual and project work Synopsis Offering WATSON DEVELOPER CLOUD WATSON CONTENT STORE WATSON TALENT HUB IBM Watson Platform
  29. 29. © 2015 IBM Corporation29 IBM Watson Services on Bluemix User Modeling Personality profiling to help engage users on their own terms. Language Identification Identifies the language in which text is written Machine Translation Globalize on the fly. Translate text from one language to another. Concept Expansion Maps euphemisms or colloquial terms to more commonly understood phrases Message Resonance Communicate with people with a style and words that suits them Question Answer Direct responses to users inquiries fueled by primary document sources Relationship Extraction Intelligently finds relationships between sentences components (nouns, verbs, subjects, objects, etc.) Visualization Rendering Graphical representations of data analysis for easier understanding
  30. 30. © 2015 IBM Corporation30 The Watson Experience Manager (WEM) With Watson Experience Manager: • Developers use APIs to access and test “Powered by Watson” apps • Data Scientists can manage their content used to enrich Watson • User experience developers can customize or create user interaction models with Watson • Domain experts can train and test their “Powered by Watson” apps WEM provides a role based set of tools for SME, Watson administrators, and Domain Experts
  31. 31. © 2015 IBM Corporation31 Access Watson Developer Cloud using Watson Experience Manager Develop app “Powered by Watson” using APIs Enrich Watson with content Train Watson using tools and experts Test app functional and non-functional Deploy application Building your “Powered by Watson” app
  32. 32. © 2015 IBM Corporation32 Let’s Get Started To partner with Watson, you need to:  Be committed to training  Have an accessible corpus of information  Identify a clear problem to solve  Get a Bluemix account  Try the Watson services free for 30 days  Take the next step towards development or production deployment Let’s Get Started with IBM Watson!
  33. 33. © 2015 IBM Corporation33 Investing and Educating www.ibmbigdatahub.com www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/ www.ibm.com/cloud-computing/bluemix/
  34. 34. © 2015 IBM Corporation34 zzzz z z z Questions?
  35. 35. © 2015 IBM Corporation35

×