Downloaded 391 times
Uploaded byIBM Sverige
Gene Villeneuve - Moving from descriptive to cognitive analytics
The document discusses the evolution of analytics from descriptive to cognitive analytics, detailing IBM's approach to big data and its application across various industries. It highlights the value of IBM's analytics solutions, including specific use cases, customer case studies, and the technologies involved in managing and analyzing big data. Additionally, it emphasizes the importance of cognitive analytics in making near real-time decisions, understanding data trustworthiness, and leveraging advanced technologies like natural language processing and machine learning.
More Related Content
Similar to Gene Villeneuve - Moving from descriptive to cognitive analytics
![제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계](https://cdn.slidesharecdn.com/ss_thumbnails/3-1boaz23rdconferencemboax-260203102709-9d519923-thumbnail.jpg?width=640&height=640&fit=bounds)
Gene Villeneuve - Moving from descriptive to cognitive analytics
- 1. © 2013 IBMCorporation A New Era of Smart Moving from Descriptive to Cognitive Analytics on your Big Data Projects Date: October 7, 2014 Gene Villeneuve Director & European Sales Leader Predictive & Business Intelligence
- 2. A New Eraof Smart Agenda Introduction and some clarification regarding terminology The evolution of analytics Descriptive Predictive Prescriptive Cognitive Analytics in the Context of Big Data Big Data & Analytics Reference Model Sample projects and customer case studies illustrating the evolution of analytics Current research & development areas © 2013 2 IBM Corporation
- 3. A New Eraof Smart INTRODUCTION & TERMINOLOGY © 2013 3 IBM Corporation
- 4. A New Eraof Smart Analytics: a Business Imperative across Industries LOB buyers are driving new demand for industry solutions At the point of impact Big Data and Analytics All perspectives All decisions All information All people The new era of computing enables new analytic methods Search Deterministic Enterprise data Machine language Simple outputs Programmatic Discovery Probabilistic Big Data Natural language Intelligent options Cognitive © 2013 4 IBM Corporation * Source: IBM Market Development & Insight – GMV 1H2013
- 5. A New Eraof Smart The Evolution of Analytics Cognitive Analytics Predictive Analytics Prescriptive Analytics Descriptive Analytics Descriptive “After-the-facts” analytics by analyzing historical data Provides clarity as to where an enterprise or an organization stands related to defined business measures Applied to all LoB for fact finding, visualization of success and failure Cognitive Pertaining to the mental processes of perception, memory, judgment, learning, and reasoning Range of different analytical strategies that are used to learn about certain types of business related functions Natural language processing Predictive Leverages data mining, statistics and ML algorithms, etc. to analyze current and historical data to predict future events and business outcome. Discovers patterns derived from historical and transactional data to optimize business measures Prescriptive Synthesizes big data, mathematical and computational sciences, and business rules to suggest decision options Takes advantage of a future opportunity or mitigate a future risk and shows the implication of each decision option © 2013 5 IBM Corporation
- 6. A New Eraof Smart The Scope of Advanced Analytics • IBM analytics breadth covers the full spectrum of decisions • IBM is the undisputed leader in advanced analytics Cognitive How can we learn dynamically? Prescriptive How can we achieve the best outcome? Predictive What could happen in the future? Descriptive What has already happened? Information Layer How is data managed and stored? How can everyone be more right… ….more often? BBuussiinneessss VVaalluuee Reasoning Learning Natural Language Optimization Rules Constraints Machine learning Forecasting Statistical Analysis Alerts & Drill Down Ad hoc Reports Standard Reports Big Data Platforms Content Management RDBMS and Integration IBM Big Data & Analytics © 2013 6 IBM Corporation
- 7. A New Eraof Smart Accelerating the Client’s Journey to Cognitive Win on Innovation COGNITIVE PRESCRIPTIVE Compete on time to business value – through context specific data, methods, workflow. Continuum PREDICTIVE DESCRIPTIVE Analytics The FOUNDATION INFORMATION Reasoning Learning Natural Language Optimization Rules Predictive Modeling Forecasting Statistical Analysis Alerts Drilldown Query Ad-hoc Reports Standard Reports Big Data Platforms Natural, Intuitive or Automated Interaction Context Specific Usage Opportunities to infuse cognition and collaboration in existing solutions and products for differentiation ECM Information Integration RDBMS © 2013 7 IBM Corporation
- 8. A New Eraof Smart Analytics: a Business Imperative across Industries Clients realize value through solutions * Source: IBM Market Development & Insight – GMV 1H2013 IBM Watson Engagement Advisor IBM Watson Engagement Advisor Transforms client experience with deep personalized Q&A Transforms client experience with deep personalized Q&A IBM Predictive Maintenance & Quality IBM Predictive Maintenance & Quality Improves productivity, prevents downtime and reduces costs Improves productivity, prevents downtime and reduces costs IBM Credit Risk Management IBM Credit Risk Management Derive competitive advantage from risk management processes Derive competitive advantage from risk management processes IBM Enterprise Marketing Management IBM Enterprise Marketing Management Discover and react in real time to how consumers are interacting Discover and react in real time to how consumers are interacting IBM Social Media Analytics IBM Social Media Analytics Uncover customer sentiment, predict behavior, improve marketing Uncover customer sentiment, predict behavior, improve marketing © 2013 8 IBM Corporation
- 9. A New Eraof Smart IBM’s Portfolio delivers Business Value Business value from automation of routine decisions, to transformative new usages of data Line of Business Leaders Industry Solutions Integrated by Design CPO CMO CHRO CFO CIO CRO Mayors Cloud Predictive Prescriptive Cognitive Mobile Social Big Data & Analytics Market-Growth Initiatives Client-Driven Capabilities and Platforms Big Data Infrastructure © 2013 9 IBM Corporation Supported by IBM expertise through BAO services Smarter Commerce Smarter Workforce Smarter Cities Smarter Analytics Cloud
- 10. A New Eraof Smart Power Systems enables next Generation Big Data and Analytics Applications Power Solutions Power Systems Industry Solutions Business & Predictive Analytics Cognitive Computing IBM Watson Natural Language Learning 1,000+ Concurrent Queries Real-time Analytics Parallel processing memory processing Stream Computing Massive IO bandwidth Continuous data load Design Open & flexible infrastructure - Available on premise or through the Cloud Large-scale ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 99.997% Availability 0 Incidents, Vulnerability 1.3M IOPS Scalability © 2013 10 IBM Corporation
- 11. A New Eraof Smart ANALYTICS IN THE CONTEXT OF BIG DATA © 2013 11 IBM Corporation
- 12. A New Eraof Smart Analytics in the Context of Big Data - The Big Data Analytics Challenge From noisy data to trustworthy insights VVeerraacciittyy Understand jargon and acronyms, eliminate spam Heterogeneous data VVaarriieettyy Combine, correlate information over 100’s of sources (sites, forums, message boards, newswires…) Timely Decision making VVeelloocciittyy Make decisions in near real-time over 10K+ messages/second <20% >80% Data Content Requiring overcoming the high volume, real-time, and unstructured nature of social media and Enterprise data streams Growing volume of data VVoolluummee Social media or other media source data Extract concepts from several 100M messages/day 100M+ active users per source Learning, NLP, Discovery • Auditory & visual processing • Logic & reasoning • Improve interventions Data Volume 360-degree Profiles • Micro-segmentation • Predict Behavior Listening and Monitoring • Sentiment, Buzz • Key influencers Analytics Complexity Manual Interaction • Polling & Extrapolation © 2013 12 IBM Corporation
- 13. A New Eraof Smart Analytics in the Context of Big Data - Key Drivers for Cognitive Analytics The need for cognitive analytics is driven by the confluence of SoLoMo (Social, Local, Mobile), Big Data, and Cloud VVeerraacciittyy VVaarriieettyy VVeelloocciittyy VVoolluummee Cognitive Systems © 2013 13 IBM Corporation
- 14. A New Eraof Smart Analytics in the Context of Big Data - Veracity / Trust / Sentiment Addressing the information trustworthiness of social media data Some dimensions of trustworthiness / Trustworthiness Sentiment – Jokes – Prosody – Sarcasm – Seriousness – Emotion – Mood – Ambiguity – Humor – Dialect – Social factors … – Social media languages – Context – etc. VVeerraacciittyy Information Provenance Author Classification Integrity Assumption Usage Intention Content Analysis Relevance Determination © 2013 14 IBM Corporation
- 15. A New Eraof Smart Analytics in the Context of Big Data DeepQA: The Architecture underlying Watson Generates many hypotheses, collects wide range of evidence, balances the combined confidences of >100 different analytics that analyze the evidence from different dimensions Answer Scoring Learned Models help combine and weigh the Evidence Models Models Models Models Models Candidate Answer Generation Answer Sources Evidence Retrieval Deep Evidence Scoring Primary Models Search Final Confidence Synthesis Merging & Ranking Answer & Confidence Evidence Sources Hypothesis Generation Hypothesis and Evidence Scoring Each year the EU selects capitals of culture; one of the 2010 cities was this Turkish “meeting place of cultures” Question & Topic Analysis Hypothesis Generation Hypothesis and Evidence Scoring Question Decomposition © 2013 15 IBM Corporation
- 16. A New Eraof Smart Analytics in the Context of Big Data - Watson drives optimized outcomes Generates and evaluates hypothesis for better outcomes 99% 60% 10% Understands natural language and human speech Adapts and Learns from user selections and responses 3 2 1 …built on a massively parallel probabilistic evidence-based architecture optimized for Linux on POWER7+ © 2013 16 IBM Corporation
- 17. A New Eraof Smart BIG DATA ANALYTICS REFERENCE MODEL © 2013 17 IBM Corporation
- 18. A New Eraof Smart Big Data & Analytics Platform An innovative, foundational big data platform can help tackle big data’s four V’s (volume, variety, velocity and veracity) with an integrated set of big data technologies to address the business pain, reduce time and cost, and provide quicker return on investment More cost-effectively analyze Analyze streaming data petabytes of structured and and large data bursts for unstructured formation near-real-time insights Access deep insight with advanced in-database analytics and operational analytics Big data platform Systems management Application development Discovery Apache Hadoop system Stream computing Data warehouse Information integration and governance Data Media Content Machine Social © 2013 18 IBM Corporation
- 19. A New Eraof Smart Big Data Analytics Reference Model - Key Capabilities Components to build a trusted information integration layer with ETL, data quality, real-time data processing, federation, metadata mgmt, … Business Analytics & Applications Layer Data Persistency Layer Infrastructure Services Data Transformation & Integration Layer Heterogeneous Data Sources Visualization & Reporting Layer Comprehensive Big Data advanced analytics layer with applications & research assets on heterogeneous source data Traditional reporting and BI analytics, with visualization & exploration of heterogeneous data Traditional DW system (SOR, ODS, marts) with MDM system, DW appliances, and augmented with Hadoop platform Common infrastructure services, such as systems management, security, backup, information governance, … Heterogeneous data landscape including existing data stored in BSS systems, from the network, external, customer touch points © 2013 19 IBM Corporation
- 20. A New Eraof Smart Cognitive Analytics Predictive Analytics Prescriptive Analytics Descriptive Analytics SAMPLE PROJECTS AND CUSTOMER CASE STUDIES ILLUSTRATING THE EVOLUTION OF ANALYTICS (IN THE CONTEXT OF BIG DATA) © 2013 22 IBM Corporation
- 21. A New Eraof Smart Predictive Analytics Demographics Enrichment for unknown Subscribers Gain analytical insight for pre-paid demographics Understand post-paid subscribers – Using post-paid demographics data (age, gender, income, …) – Gaining insight: propensity/predictive modeling, micro-segmentation, clustering, sentiment analytics, … from appl usage data, web browsing, CDR, social media Understand pre-paid subscribers – Gaining insight: propensity/predictive modeling, micro-segmentation, clustering, sentiment analytics, … – Demographics data isn't available or not sufficiently trustworthy Correlate post- with pre-paid subscribers and map demographics – Correlate post- with pre-paid segments, clusters, behavior, interest, … – Map known demographics for post-paid to corresponding pre-paid subscribers Required Data Sources Voice & data CDR (MSISDN & Usage) Behavioral data: – Web browsing & search (internal and external), user agent: browser, appl and/or device that made request, content type: type of data sent/downloaded Public sources (will be used, not required from CSP): – Wikipedia – IMDB http://www.imdb.com/ – Open Directory Project (ODP) © 2013 24 IBM Corporation Subscriber reference data (e.g. from CRM or EDW) Predictive Analytics
- 22. A New Eraof Smart Predictive Analytics Demographics Enrichment for unknown Subscribers CSP Analytical insight Visualization Consumption by Advertisement DATA SOURCES CSP & other Voice & data CDR (MSISDN & Usage) MSP (MSISDN & URL) Behavioral data (e.g. blogs, use of mobile apps, Web browsing & Web search ) Public sources (e.g. ODP) Metadata, e.g. time, size, … CRM or EDW IBM Singapore Data understanding Data transformation Data preparation Predictive Analytics PRODUCTS & Tools BigInsights (incl. BigSheets, SystemT, HDFS, Jaql, …) Customer Modeler SPSS Modeler NLP DB2 SaaS Correlation Predictive modeling Propensity modeling Micro-segmentation Clustering Sentiment IBM BigInsights Admin Customer Modeler Admin (Predictive Analytics) Data anonymization Data provisioning © 2013 25 IBM Corporation
- 23. A New Eraof Smart Predictive Analytics Demographics Enrichment for unknown Subscribers Pre-paid CSP Data Sources: Voice/Data CDRs Behavioral Data Source Data Transformation HDFS Analytical Model (pre-paid) DB2 Predictive Model (for pre-paid) HDFS Analytical Model (post-paid) Public Sources (not from CSP): Wikipedia IMDB ODP Post-paid CSP Data Sources: Voice/Data CDRs Behavioral Data • InfoSphere BigInsights • Customer Modeler • SPSS / DB2 / NLP GTS SmartCloud Enterprise Predictive Analytics Analysis/Insight Pre-paid: • Age • Gender • Income Used for gaining Analytical Insight Transformation Anonymization (to be validated) Post-paid CSP Data Sources: Subscriber Demographics Visualization Used for building Predictive Model © 2013 26 IBM Corporation
- 24. A New Eraof Smart XO Communications takes control of customer satisfaction 142 percent reduction in revenue erosion for customers at most risk of churning $10 million+ savings/year from increased retention and reduced customer service costs 5 months to achieve full return on investment Solution components The transformation: XO Communications had already taken the first steps in identifying customer retention risks through analytics; now it wanted to seize the opportunity to put these insights into action more effectively. By using IBM® SPSS® solutions to hone its predictive models, the company built a richer, more up-to-date picture of its client base and began delivering this data to a greater range of employees. “We are only just starting to realize the true potential that IBM analytics holds across the business.” • IBM® SPSS® Analytics Catalyst — Bill Helmrath, Director of Business Intelligence, XO Communications • IBM SPSS Modeler • IBM SPSS Modeler Server • IBM SPSS Statistics • IBM InfoSphere® BigInsights™ YTP03235-USEN-00 © 2013 27 IBM Corporation
- 25. A New Eraof Smart Fiserv cuts IT costs while enhancing analytics capabilities with software and infrastructure from IBM $8 million saved in IT costs over a five-year period 90% reduction in the number of midrange servers under management Boosts availability and improves the agility of service delivery Solution Components IBM® AIX® IBM Cognos® Business Intelligence IBM DB2® IBM InfoSphere® Warehouse IBM PowerHA® IBM PowerVM® IBM SPSS® IBM Tivoli® Storage Manager and System Automation for Multi- Platforms IBM WebSphere® Application Server IBM Power® 770 Business Challenge: Fiserv was seeking new ways to attract, retain and grow profitable customer relationships while helping its clients compete with newer and larger banks. Leveraging predictive analytics applications proved key to this goal, but Fiserv realised that it also needed a more agile, available and scalable IT infrastructure to support its new capabilities. The Solution: IBM information management and predictive analytic solutions enable Fiserv to transform billions of raw transactions into actionable insights that help small and midsize banks better target offers and maximize their marketing dollars. The use of cloud technologies to consolidate and virtualize servers helps reduce costs and accelerate time-to-market. “We have estimated a five-year-cumulative run rate reduction of about $8 million with the server consolidation and virtualization project.” —Leroy Hill, Manager, Midrange Engineering, Fiserv © 2013 28 IBM Corporation
- 26. A New Eraof Smart Cognitive Analytics Halalan 2013 Social Media Tracking BUZZ – candidates, topics, personalities, broadcasters Cognitive Analytics – How much / What is being said about the candidates (ongoing and for key “events” like debates, advertisements, etc.), different shows, news anchors. – How does this change over time, what is trending. SENTIMENT – popular opinion – What do voters like or dislike about the candidates, the parties, campaigns, constituents, etc. – How does this sentiment break down by the different groups (voters, political affiliation, news professionals, demographics, affinity groups, etc.) – Understand brand sentiment, i.e., whether ABS-CBN is being perceived as unbiased and trusted. How are the different news personalities being perceived: credible, neutral, fair? INTENT – action – What is the intent to act (support / vote) for each candidate. – What election outcomes can be predicted (shifts in candidate sentiment, voter intent, etc.) © 2013 29 IBM Corporation
- 27. A New Eraof Smart (just a few examples) CURRENT RESEARCH & DEVELOPMENT AREAS © 2013 30 IBM Corporation
- 28. A New Eraof Smart Cognitive Analytics: Technical Capabilities required Watson Solutions – Build on repeatable Assets Watson for Healthcare Watson for Financial Services Watson for Client Engagement Watson for Industry Solutions Sample Advisor Solutions Sample Advisor Solutions Sample Advisor Solutions Utilization Research Banking Insurance Call Center Oncology Care Mgt. Financial Markets Knowledge Help Desk Technical ASK Services DISCOVER Services DECISION Services NLP & Machine Learning 100111001 10010010010 1000101100101 10001010010 00110101 Data Analytics Cloud Mobile Workload Optimized Systems Capabilities Platform Content Tooling Methods Algorithms APIs Ready Build Teach Run Full Lifecycle © 2013 31 IBM Corporation
- 29. A New Eraof Smart Massive Scale SNA (X-RIME) over BigInsights Current Research Area Project Overview – X-RIME is a library that consists of MapReduce programs, which are used to do raw data pre-processing, transformation, SNA metrics and structures calculation, and graph / network visualization – Based on IBM InfoSphere BigInsights (Hadoop) – Goes beyond SPSS SNA for churn propensity modeling Reference – Commercial Solution: China Mobile enterprise blog analysis solution – ARL MSA on Power Benchmarking: Pageranking 390 millions of nodes on 10-nodes power7 cluster (2 hours per iteration) – Integrated to SystemG as GraphBase – Open Source X-RIME on SourceForge Selected X-RIME SNA Algorithms – Vertex degrees (in/out/both/average/max ) – Weekly connected components – Bi-connected components – Breadth first search (BFS) – K-core – Maximal clique – Community detection based on label propagation – Community detection based on scored label propagation – Community detection based on propinquity – Modularity evaluation – Hyperlink induced topic search (HITS) – Pagerank – Minimal spanning tree (MST) – Ego-centric network – Vertex clustering coefficient – Edge clustering coefficient SSNNAA lliibbrraarryy Message Passing Framework Graph Data Model (Object) MMaappRReedduuccee HHDDFFSS X-RIME Architecture © 2013 32 IBM Corporation
- 30. © 2013 IBMCorporation A New Era of Smart Thank you
Editor's Notes
- #5 To put cognitive systems into the proper context lets take a look at some of the differences over a more traditional programmatic approach to problem solving. What Google is to search Watson is to discovery. We all have entered key words into a search bar only to have millions of entries returned for our review. Unfortunately, the majority of the information retrieved is not what we were looking for so we start over. Watson looks to bring back relevant results, with confidence, putting content into context. Unlike deterministic outcomes Watson is probabilistic in nature. Take a simple question like 2+2. A precise answer is 4. That is exactly how a deterministic system would respond. However, Watson is not so sure. It may have a high confidence that 2+2=4 is the right answer, however if the context of the question was regarding automotive, 2+2 could have been a car configuration – two front seats, two back seats. If we had been talking to a family psychologist 2+2 could have been referencing a family unit with 2 parents and 2 children. You can quickly see how things have varying meaning which need to be analyzed and properly considered in the context of the broader questions being asked. Unlike traditional systems which thrive off structure, were information is stored in a binary fashion all neatly organized into rows and columns, Watson can tackle unstructured data spread across disparate sources to unlock patterns and possibilities. And we have already touched on the importance of working in natural language.
- #6 This chart illustrates the evolution from descriptive to predictive and presecriptive to cognitive analytics and lists the key characteristics of each phase or analytics domain. Important is to highlight the different possible analytics journeys and entry points. Clients will not always need to have a mature presecriptive analytics platform in place to launch a cognitive analytics initiative. We define Cognitive Systems as those systems that can navigate the complexities of human language and understanding, ingest and process vast amounts of structured and unstructured data, generate and evaluate countless possibilities, and scale in proportion to the task. These systems apply human-like characteristics to conveying and manipulating ideas, that when combined with the inherent strengths of digital computing can solve problems with higher accuracy, more resilience, and on a massive scale. Watson is an example of a Cognitive System. It is able to tease apart the human language to identify inferences between text passages with human-like high accuracy, and at speeds and scale far faster and far bigger than any person could do on their own. Watson doesn’t really understand the individual words in the language, but it does understand the features of language as used by people and from that is able to determine whether one text passage (call it the ‘question’) infers another text passage (call it the ‘answer’) with incredible accuracy under changing circumstances. In Jeopardy! we had to determine whether the question, “Jody Foster took this home for her role in ‘The Silence of the Lambs’” inferred the answer “Jody Foster won an Oscar for her role in ‘The Silence of the Lambs’”. In this case, taking something home inferred winning an Oscar. But it doesn’t always. Sometimes, ‘taking something home’ infers, ‘a Cold’, or ‘Groceries’, or any number of things. Context matters. Temporal and spatial constrains matters. All of that adds to enabling a Cognitive System to behave with human-like characteristics. A rules-based approach would require a near infinite number of rules to capture every case we might encounter in language.
- #8 This chart illustrates the various technical capabilities that characterize the various analytics areas. The reality of analytics-related use case sceanrios is that clients may have requirements on the entire continuum of analytics, ie. The focus may be on descriptive analytics with the need to implement all corresponding technical capabilities, but also analytical requirements from the remaining 3 analytics domains, including cognitive analytics. For instance, some clients may have a rather mature descriptive analytics platform, and require natural language processing capabilities for a sentiment analytics project to derive to brand sentiment, affinity analytical insight, without necessarily implementing a sophisticated predictive analytics platform. When walking through this chart, explain the individual technical capabilities for each analytics capability.
- #9 This chart lists some of the clients that IBM has worked with. In order to become familiar with the details of these projects, please visit the “IBM client reference Database”. Here is the link: http://w3-01.ibm.com/sales/references/crdb/ibmref.nsf/winsubmit?openform
- #10 The key message of this chart is the meaning of the blue areas to enable the various analytics domains. These blue areas are BI Data Infrastructure and Big Data Analytics with its capabilities to embrace the mobile needs, social media analytics, and cloud deployment models. These blue domains collectively enable the predictive, prescriptive and cognitive analytics (and descriptive analytics as well – although not explicitely mentioned on the chart). The capabilities transform into the key initiatives such as Smarter Commerce, Smarter Workforce, Smarter Analytics, and Smarter Cities and provide key business value to C-level stakeholders listed at the top of the chart. The business value is delivered for all Industries, which is illustrated by the 12 little symbols on the top of the figure. So the key message of the chart is the illustrateion – or rather transformation – of key technical domains such as BI Data Infrastructure and Big Data & Analytics capabilities to a broad industry-relevant set of business values.
- #11 Following are the 3 key messages of the chart: Clients are leveraging various types of analytics to solve real business challenges. They soon realize there is no single solution to address all of their analytics requirements. Businesses in different industries will have specific needs that applying analytics can address, but there is no one size fits all. This is why IBM offers many different analytics offerings. This includes industry specific solutions that address unique needs in major industries as well as optimized business and predictive analytics solutions. Cognitive computing, like IBM Watson is another example. IBM is delivering these solutions on Power Systems because of the platform’s design points and capabilities. Power was built from the ground up to handle data-related applications and analytics workloads.
- #13 Key Points of the chart is to highlight the 4 Vs that represent an essential way to characterize Big Data: veracity, variety, velocity, and volume. Volume is about rising volumes of data in all of your systems – which presents a challenge for both scaling those systems and also the integration points among them Variety is about managing many types of data, and understanding and analyzing them in their native form. Velocity is about ingesting data in real time and in-motion Veracity deals with the certainty, or truthfulness of big data. Veracity is a big issue – and one that directly relates to confidence. In fact, as the complexity of big data rises (the first 3 Vs grow), it actually becomes harder to establish veracity. The left part of the chart illustrates just 1 dimension (volume) in the context of increasing analytical complexity.
- #14 This chart puts into perspective three key areas that influence and drive the need of cognitive systems and analytics: Big Data: highlight the 4 Vs again as a key driver, especially the need to analyze text, speech, video content, and other non-structured data, such as LOGs, call center transcripts, etc. Also highlight the veracity – meaning trustworthiness – of the data that requires reasoning, and other cognitive analytical capabilities to put insight into context and provide contextual meaning Cloud: as a key deployment model, cloud represents a driving force to also take into consideration cognitive analytics in the cloud. Highlight the need to provide analytics capabilities that can be deployed and leverage in the cloud. As an example, point out IBM’s Social Media Analytics (SMA) v1.2 – the former Cognos Consumer Insight – which is not only cloud enabled but is offered as a cloud service by IBM. SoLoMo: still a rather term, SoLoMo (Social, Local, Mobile) is increasingly used to describe these 3 aspects as a combined area that characterizes today’s consumer lifestyles. As such, all 3 aspects drive specific requirements and influence the technical and business capabilities that cognitive analytics needs to deliver. Social means for instance to understand contributions to social media networks, the meaning in context, and to be able to analyze natural language and text in all languages, dialects, and also the sometimes unique style of communication that takes place in social media networks. Local requires the locality awareness for instance to deliver location based services, preferences and culture-awareness when running cognitive analytics. Mobility in regards to cognitive analytics requires the inclusion and understanding of the mobile lifestyle, mobility patterns and preferences.
- #15 This chart focuses on the veracity and trustworthiness of big data, and introduces some dimensions of trustworthiness and veracity (right side of the chart). One of the key aspects of big data is the analytics of social media networks. Contributions via social media networks, however, need to be analyzed by taking the listed dimension into consideration. For instance, what was the usage intention of a social media network contribution, what is its relevance for the given analytics in scope or the use case scenario. The left side of the chart lists some of the challenges that requires sophisticated and state-of-the-art cognitive analytics capabilities in order to for instance understand whether a statement/contribution is done in a certain mood or emotional state, whether it is meant as a joke, whether it represents a sarcastic statement, and so forth.
- #16 Watson – the computer system we developed to play Jeopardy! is based on the DeepQA softate archtiecture.Here is a look at the DeepQA architecture. This is like looking inside the brain of the Watson system from about 30,000 feet high. Remember, the intended meaning of natural language is ambiguous, tacit and highly contextual. The computer needs to consider many possible meanings, attempting to find the evidence and inference paths that are most confidently supported by the data. So, the primary computational principle supported by the DeepQA architecture is to assume and pursue multiple interpretations of the question, to generate many plausible answers or hypotheses and to collect and evaluate many different competing evidence paths that might support or refute those hypotheses. Each component in the system adds assumptions about what the question might means or what the content means or what the answer might be or why it might be correct. DeepQA is implemented as an extensible architecture and was designed at the outset to support interoperability. &lt;UIMA Mention&gt; For this reason it was implemented using UIMA, a framework and OASIS standard for interoperable text and multi-modal analysis contributed by IBM to the open-source community. Over 100 different algorithms, implemented as UIMA components, were integrated into this architecture to build Watson. In the first step, Question and Category analysis, parsing algorithms decompose the question into its grammatical components. Other algorithms here will identify and tag specific semantic entities like names, places or dates. In particular the type of thing being asked for, if is indicated at all, will be identified. We call this the LAT or Lexical Answer Type, like this “FISH”, this “CHARACTER” or “COUNTRY”. In Query Decomposition, different assumptions are made about if and how the question might be decomposed into sub questions. The original and each identified sub part follow parallel paths through the system. In Hypothesis Generation, DeepQA does a variety of very broad searches for each of several interpretations of the question. Note that Watson, to compete on Jeopardy! is not connected to the internet. These searches are performed over a combination of unstructured data, natural language documents, and structured data, available data bases and knowledge bases fed to Watson during training. The goal of this step is to generate possible answers to the question and/or its sub parts. At this point there is very little confidence in these possible answers since little intelligence has been applied to understanding the content that might relate to the question. The focus at this point on generating a broad set of hypotheses, – or for this application what we call them “Candidate Answers”. To implement this step for Watson we integrated and advanced multiple open-source text and KB search components. After candidate generation DeepQA also performs Soft Filtering where it makes parameterized judgments about which and how many candidate answers are most likely worth investing more computation given specific constrains on time and available hardware. Based on a trained threshold for optimizing the tradeoff between accuracy and speed, Soft Filtering uses different light-weight algorithms to judge which candidates are worth gathering evidence for and which should get less attention and continue through the computation as-is. In contrast, if this were a hard-filter those candidates falling below the threshold would be eliminated from consideration entirely at this point. In Hypothesis & Evidence Scoring the candidate answers are first scored independently of any additional evidence by deeper analysis algorithms. This may for example include Typing Algorithms. These are algorithms that produce a score indicating how likely it is that a candidate answer is an instance of the Lexical Answer Type determined in the first step – for example Country, Agent, Character, City, Slogan, Book etc. Many of these algorithms may fire using different resources and techniques to come up with a score. What is the likelihood that “Washington” for example, refers to a “General” or a “Capital” or a “State” or a “Mountain” or a “Father” or a “Founder”? For each candidate answer many pieces of additional Evidence are search for. Each of these pieces of evidence are subjected to more algorithms that deeply analyze the evidentiary passages and score the likelihood that the passage supports or refutes the correctness of the candidate answer. These algorithms may consider variations in grammatical structure, word usage, and meaning. In the Synthesis step, if the question had been decomposed into sub-parts, one or more synthesis algorithms will fire. They will apply methods for inferring a coherent final answer from the constituent elements derived from the questions sub-parts. Finally, arriving at the last step, Final Merging and Ranking, are many possible answers, each paired with many pieces of evidence and each of these scored by many algorithms to produce hundreds of feature scores. All giving some evidence for the correctness of each candidate answer. Trained models are applied to weigh the relative importance of these feature scores. These models are trained with ML methods to predict, based on past performance, how best to combine all this scores to produce final, single confidence numbers for each candidate answer and to produce the final ranking of all candidates. The answer with the strongest confidence would be Watson’s final answer. And Watson would try to buzz-in provided that top answer’s confidence was above a certain threshold. ---- The DeepQA system defers commitments and carries possibilities through the entire process while searching for increasing broader contextual evidence and more credible inferences to support the most likely candidate answers. All the algorithms used to interpret questions, generate candidate answers, score answers, collection evidence and score evidence are loosely coupled but work holistically by virtue of DeepQA’s pervasive machine learning infrastructure. No one component could realize its impact on end-to-end performance without being integrated and trained with the other components AND they are all evolving simultaneously. In fact what had 10% impact on some metric one day, might 1 month later, only contribute 2% to overall performance due to evolving component algorithms and interactions. This is why the system as it develops in regularly trained and retrained. DeepQA is a complex system architecture designed to extensibly deal with the challenges of natural language processing applications and to adapt to new domains of knowledge. The Jeopardy! Challenge has greatly inspired its design and implementation for the Watson system.
- #17 IBM Watson is the very embodiment of the new era of cognitive systems. It represents a new category of solutions that leverages deep content analysis and evidence-based reasoning to accelerate and improve decisions, reduce operational costs, and optimize outcomes. Cognitive Systems offer a whole new way of computing. Keeping pace with the demands of an increasingly complex business environment requires a paradigm shift in what we should expect from IT. We need an approach that recognizes today’s realities and treats them as opportunities rather than challenges. Main point: At the core of what makes Watson different are three powerful technologies - natural language, hypothesis generation, and evidence based learning. But Watson is more than the sum of its individual parts. Watson is about bringing these capabilities together in a way that’s never been done before resulting in a fundamental change in the way businesses look at quickly solving problems Further speaking points:. Looking at these one by one, understanding natural language and the way we speak breaks down the communication barrier that has stood in the way between people and their machines for so long. Hypothesis generation bypasses the historic deterministic way that computers function and recognizes that there are various probabilities of various outcomes rather than a single definitive ‘right’ response. And adaptation and learning helps Watson continuously improve in the same way that humans learn….it keeps track of which of its selections were selected by users and which responses got positive feedback thus improving future response generation Additional information: The result is a machine that functions along side of us as an assistant rather than something we wrestle with to get an adequate outcome
- #18 This section introduces the Big Data analytics reference model and serves as an introduction into the use case scenarios, which illustrate the various stages of analytics
- #19 Best in Breed Analytics Placed On Top Fuel all decision-making with powerful analytics & analytic adoption without silos Analyze all data wherever it lives Accelerate business value with solutions that leverage all data types, with predictive insight to let you know what has happened, what is happening and what is likely to happen next Delivering optimized decisions at point of impact through business applications Empower end business users with the information to deliver the best decision every time All touchpoints are managed in real time, via the appropriate channel Feedback loop ensures all decisions are accurate, dynamic Business rules integrated with analytics and optimization All of these different technologies come together (“an integrated platform”) to create decision services for the different LOB areas (e.g. marketing)
- #20 The depicted Big Data analytics reference model serves as an introduction into this section, and illustrates the key capabilities. In presenting this chart, explain the capabilities in the order listed here: Heterogeneous data sources Data transformation and integration layer Data persistency layer Business analytics and application layer Visualization and reporting layer Infrastructure services Highlight the message that these capability categories need to be addressed in every project. The focus, however, can vary depending on specific project requirements.
- #21 This figures describes the Big Data analytics reference model in just a slightly different way and lists the different technical capabilities within the various layers. We are listing essentially the same components as on the previous chart and highlight the breadth of different technical capabilities that each component - or layer – needs to be comprised of. Point out the fact that not all capabilities need to be included in every project. The concrete sublist of technical capabilities is derived from the concrete requirements and set of use cases for an individual project.
- #22 This chart contains a product mapping to the Big Data analytics reference model, which has been further customized for CSPs (Communication Service Providers). This chart and the 2 previous ones also serve as an introduction to the examples that are described in the following section. The presenter should make himself familir with all products and tools that are referenced in this product mapping chart.
- #23 This section describes – at a very high-level – sample projects for all 4 analytics areas: descriptive, predictive, prescriptive, and cognitive. All examples and corresponding use case sceanrios in this section are derived from real customer engagements in Asia Pacific.
- #24 This is an example of descriptive analytics project, where a Telco Service Provider is interested in competitive analytics based on CDR (Call Detail Records). Analysis of CDR records was optionally enhanced with analytics from social media networks. The data sources are depicted on the left side of the chart. The component in the center of the chart is comprised of the core capabilities that are derived from BigInsights and BigInsights applications, such as Customer Modeler (an IBM Research asset). Analytical insight is derived from the combined components in this central box. The analytics can optionally be enhanced with SPSS to deliver predictive analytics. In the real customer project, however, this was not part of the use case scenario. The left side of the chart illustrates the data warehouse and the descriptive BI analytics component, Cognos BI. This example also illustrates that descriptive analytics is very much a part of Big Data. CDR records are very large in volume, semi-structured, and especially the combination with non-structured data from social media networking sites, makes descriptive analytics very well Big Data use case scenario. It illustrates the changing paradigm that descriptive analytics plays in Big Data.
- #25 This sample project is geared towards determining demographics information for unknown pre-paid subscribers. The 1st main heading on the chart (gain analytical insight for pre-paid demographics) explains the logical flow and main steps that need to be performed. The 2nd main heading (required data sources) lists the data srouces such as voice and data CDRs, behavioral data and so forth. This is also a very nice example, which illustrates that predictive analytics – as well as descriptive analytics – are part of Big Data, ie. can be seen from a Big Data angle. The main step in this analytics flow is to predict demographics information for pre-paid subscribers by correlating and mapping post-paid with pre-paid subscribers. This sample project is further described on the following 2 charts with: a contextual diagram and an architecture overview diagram
- #26 This chart contains a high-level contextual digram with the key components, such as the data sources on the left, the cloud-based analytics system that leverages the IBM SmartCloud at IBM Singaporeand the key products on the right of the chart. The blue figure at the lower right corner is an illustration of the analytics and admin roles and responsibilities that exist in operating the environment. The yellow figure at the left upper corner illustrates the LoB user using the system and deriving to predictive insight.
- #27 This chart contains an architecture overview diagram that contains the key components and the component interaction at a high-level. Public data sources: will be used in the scenario to gain analytical insight and to leverage existing categorization of for instance websites that are visited by subscribers Post-paid data sources: will be used to understand preferences, interest, websites visited, performing micro-segmentation, etc. for post-paid subscribers Prep-paid data sources: the same data sources will be used from pre-paid subscribers, where the same analytical insight is derived for this category of subscribers Post-paid demographics information: will be used and correlated with the analytical insight that is derived from post-paid data sources. This allows a comprehensive view on post-paid subscribers, which includes knowledge on demographics. The analytics engine – depicted in the centre of the chart is used to correlate post- with pre-paid segments, clusters, behavior, interest, … and map known demographics for post-paid to corresponding pre-paid subscribers. This will allow prediction of demographics for pre-paid subscribers, e.g. age, gender, income, and other demographics measures.
- #28 Client Name: XO Communications Case study Link: http://www-01.ibm.com/software/success/cssdb.nsf/CS/STRD-9E4L7Y Pull Quote: &quot;We are only just starting to realize the true potential that IBM analytics holds across the business.&quot; —Bill Helmrath, Director of Business Intelligence, XO Communications Company Background: XO Communications is one of the United States’ largest communications service providers, offering a comprehensive portfolio of communications, network and hosted IT services through a 19,000-mile nationwide inter-city network and over 1,000 office locations. Priding itself on superior customer experience, the company is always looking for ways to raise the bar. Solution components: Software • IBM® SPSS® Analytics Catalyst • IBM SPSS Modeler • IBM SPSS Modeler Server • IBM SPSS Statistics • IBM InfoSphere® BigInsights™ Business challenge: XO Communications had already taken the first steps in identifying customer retention risks through analytics; now it wanted to seize the opportunity to put these insights into action more effectively. The benefit: 142 percent estimated reduction in revenue erosion for customers at most risk of churning. $10 million+ estimated savings per year from increased customer retention and reduced customer service costs 5 months to achieve full return on investment
- #29 Link to reference profile: http://w3-01.ibm.com/sales/ssi/cgi-bin/ssialias?infotype=CR&subtype=NA&htmlfid=0CRDD-8C53TV&appname=crmd Solution synopsis A global provider of information management and electronic commerce services for financial institutions in the United States anticipates increased revenue and increased competitive edge when it works with IBM Global Technology Services - Integrated Technology Services and IBM Software Services for Information Management to develop a powerful predictive analytics service for small to midsize banks comprising IBM Power Systems technology and IBM Information Management software
- #30 This chart describes at high-level a sentiment analytics project with ABS-CBN in the Philippines. The objective of the project was to analyze social media about election candidates and the issues that impact them: Buzz - Candidates, topics, personalities, broadcastersHow much / what is being said about the candidates (ongoing and for key &quot;events&quot; like debates, advertisements, etc), different shows, news anchors. How does this change over time – what is trending. Sentiment – Popular OpinionWhat do voters like or dislike about the candidates, the parties, campaigns, constituents, etc?How does this sentiment break down by the different groups (voters, political affiliation, news professionals, demographics, affinity groups, etc)?Understand brand sentiment – whether ABS CBN are being perceived as being unbiased and trusted. How are the different news personalities being perceived – credible, neutral & fair. Intent - What is the intent to act (support / vote) for each candidate?What election outcomes can be predicted (shifts in candidate sentiment, voter intent, etc)?
- #32 Main point: How does Watson work? It’s not a simple answer. But since Watson solutions are built on a set of repeatable assets that draw from decades of market leadership, research, and best practices. Beginning at the bottom: Watson solutions are implemented with customers with a full lifecycle of readiness preparation, building the solution itself, teaching Watson about the industry, use case, and data involved, and finally putting it into production during which it continues to improve through experience and feedback loops The basic platform of Watson operation is built on a core of ingestable natural language content, tooling to train and utilize Watson’s functionality, proven methods of successful lifecycle operation, algorithms for analytic parsing of language and identification of responses, and APIs to allow other modular functionality to interact with Watson. Built on this platform of core function is a set of capabilities used across Watson solutions. These include natural language processing capabilities to understand human communication (both from a user interface perspective and more importantly, as a source of information upon which to draw for evidence-based responses) and machine learning capabilities to learn from experience. Data is the fuel of Watson’s engine and a currated data corpus of structured and unstructured data is where Watson draws for evidence in its responses. Watson draws on IBMs’ leadership in analytics (predictive, business, etc.) to find patterns and relationships invisible to the naked eye. Watson solutions use cloud-based delivery to help scale their reach, optimize utilization of the infrastructure required, and help improve accessibility for users. With cloud-based delivery comes mobile accessibility since processing requirements on the user interface device itself are minimized. And finally, Watson infrastructure is optimized for the unique workloads it requires yet Watson runs on commercial off the shelf IBM p-series hardware. Drawing upon these capabilities are the Ask, Discover, Decide services discussed previously Actual Watson solutions are developed in close collaboration with industry and domain leaders. IBM has partnered with leaders in healthcare, financial services, and other areas to develop Watson Advisor Solutions to help professionals make better use of available information to improve outcomes. Early brainstorming has led to initial pilots which has led to full production Watson Advisor Solutions, which is leading to expansion into new use cases, industries and domains. The future of Watson and Cognitive Systems is as bold and compelling as the imagination itself.
- #33 This chart elaborates on an IBM research effort to use BigInsights as a platform for massive scale Social Network Analytics (SNA). Further description of X-RIME can be found here: