Downloaded 45 times
![Page 11
ML Sample Application
► Web search
► Computational biology
► Finance
► E-commerce
► Space exploration
► Robotics
► Information extraction
► Social networks
► Debugging
► [Your favorite area]](https://image.slidesharecdn.com/datadrivenantimomusone-160605221359/75/Predire-il-futuro-con-Machine-Learning-Big-Data-11-2048.jpg)
Uploaded byData Driven Innovation
Predire il futuro con Machine Learning & Big Data
The document provides an overview of machine learning and predictive analytics, outlining definitions, methodologies, and applications in various fields like business, finance, and healthcare. It differentiates between supervised and unsupervised learning, discusses model evaluation, and highlights tools such as IBM Watson and Azure Machine Learning for data processing. The presentation emphasizes the importance of data-driven innovation in enhancing decision-making and predictive capabilities.
More Related Content
Similar to Predire il futuro con Machine Learning & Big Data
Predire il futuro con Machine Learning & Big Data
- 2. Data Driven Innovation Codemotion Presentationtitle Antimo Musone IT Manager 20 Maggio 2016
- 3. Page 3 About Me ►AntimoMusone IT Manager / Architect presso EY Co - Founder Fifth Ingenum Srls. Ing. Informatica II Università degli Studi di Napoli email: antimo.musone@it.ey.com
- 4. Page 4 Indice ►What isMachine Learning ? ►Predictive Analytics ►Machine Overview ►Defining Predictive Analytics ►Supervised Learning ►Unsupervised Learning ►Watson Service ► Cortana Analytics Suite ►Demo
- 5. Page 5 What isMachine Learning ?
- 6. Page 6 Machine Learning/ Predictive Analytics Vision Analytics Recommenda- tion engines Advertising analysis Weather forecasting for business planning Social network analysis Legal discovery and document archiving Pricing analysis Fraud detection Churn analysis Equipment monitoring Location-based tracking and services Personalized Insurance Machine learning & predictive analytics are core capabilities that are needed throughout your business
- 7. Page 7 Machine LearningOverview ► Formal definition: “The field of machine learning is concerned with the question of how to construct computer programs that automatically improve with experience” - Tom M. Mitchell ► Another definition: “The goal of machine learning is to program computers to use example data or past experience to solve a given problem.” – Introduction to Machine Learning, 2nd Edition, MIT Press ► ML often involves two primary techniques: ► Supervised Learning: Finding the mapping between inputs and outputs using correct values to “train” a model ► Unsupervised Learning: Finding patterns in the input data (similar to Density Estimates in Statistics)
- 8. Page 8 Machine Learning Data: AB C D E F G H I J K L M N O P Q R S T U V W X Y Z Rules, or Algorithms: about, Learning, language – Spelling and sounding builds words Learning about language. – Words build sentences Learning, or Abstraction: Any new understanding proceeds from previous knowledge. Data + Rules/ Algorithms = Machine Learning
- 9. Page 9 Traditional programmingVS Machine Learning Computer Data Program Output Traditional Programming Data Output Program/Algorithms Machine Learning Program can predict the output! Computer
- 10. Page 10 ML :No, more like gardening Gardener = You Seeds = Algorithms Nutrients = Data Plants = Programs
- 11. Page 11 ML SampleApplication ► Web search ► Computational biology ► Finance ► E-commerce ► Space exploration ► Robotics ► Information extraction ► Social networks ► Debugging ► [Your favorite area]
- 12. Page 12 What isPredictive Analytics? Presentation title Wikipedia Definition: (http://en.wikipedia.org/wiki/Predictive_analytics) “Predictive analytics encompasses a variety of techniques from statistics, modeling, machine learning, and data mining that analyze current and historical facts to make predictions about future, or otherwise unknown, events.” Facts Predictions Predictive Analytics Techniques
- 13. Page 13 Breaking itDown Presentation title “Predictive analytics encompasses a variety of techniques from statistics, modeling, machine learning, and data mining that analyze current and historical facts to make predictions about future, or otherwise unknown, events.” Machine Learning Use of computer algorithms to derive complex formulations based on objectives and constraints Tools and Techniques Data visualization, segmentation, correlations Use in Predictive Analytics Predictive analytics is often applied in the context of datasets that are too large for manual analysis, so data mining techniques are required Statistics Focus on learning population characteristics based on samples of data Tools and Techniques p-values, confidence intervals, sampling, ANOVA Use in Predictive Analytics Underlying theory behind many parametric models – observed facts are a sample from a population including both known/historic and unknown/future events Modeling Representations of systems used to understand the underlying dynamics of the system Tools and Techniques Symbolic logic, proxies Use in Predictive Analytics Complex relationships can be simplified through modeling – these models can then be used to analyze relationships between factors
- 14. Page 14 What isa Model? A model is a simplified representation of observed effects Presentation title Key terms: Dependent or target variable – the variable of interest Independent or predictor variable(s) – variable(s) used for explanation/prediction Effect – the (quantitative) impact of an independent variable or combination of independent variables on the dependent variable Main Effect – The direct effect of a single independent variable on the dependent variable Interaction Effect – The effect of a combination of multiple independent variables on the dependent variable
- 15. Page 15 Two typesof model A model is a simplified representation of observed effects Presentation title Statistical Parametric Models Effects are well-quantified and can be examined An equation can be used to represent the model Emphasis on explanation “What causes the dependent variable to change?” Test hypotheses p-values, confidence intervals Machine Learning Non-parametric models Effects may be unquantified (“black box”) No representative equation Model may be stochastic, so results my vary Emphasis on prediction “What will the value of the next observation be?” Generate hypotheses
- 16. Page 16 Types ofLearning ► Supervised (inductive) learning ► Training data includes desired outputs ► Dependent variable is known ► May be statistical or non-statistical ► Unsupervised learning ► Training data does not include desired outputs ► No dependent variable ► Non-statistical ► Semi-supervised learning ► Training data includes a few desired outputs
- 17. Page 17 Machine LearningProblem Classification or Categorization Clustering Regression Dimensionality reduction Supervised Learning Unsupervised Learning DiscreteContinuous
- 18. Page 18 What isLogistic Regression? Regression Models are a form of supervised learning that attempt to fit “linear” functions to training data – the most common type of regression, linear regression, should be familiar to most of you as a “best fit line” Logistic Regression is closely related to linear regression, but fits a different shape function by using a binomial link function on the dependent variable
- 19. Page 19 Machine LearningExample Predict function F(X) for new examples X Discrete F(X): Classification Continuous F(X): Regression F(X) = Probability(X): Probability estimation Given examples of a function (X, F(X)) The probability of an event X, denoted F(X), represents the proportion of all events that have X as their outcome, and is typically represented as a decimal 0<P(X)<1
- 20. Page 20 Machine LearningExample Apply a prediction function to a feature representation of the image to get the desired output: • Training: given a training set of labeled examples {(x1,y1), …, (xN,yN)}, estimate the prediction function f by minimizing the prediction error on the training set • Testing: apply f to a never before seen test example x and output the predicted value y = f(x) output prediction function Image feature y = f(x) F( ) = «apple» F( ) =«tomato» F( ) = «dog»
- 21. Page 21 Supervised Learning Used when you want to predict unknown answers from answers you already have Data is divided into two parts: the data you will use to “teach” the system (data set), and the data to test the algorithm (test set) After you select and clean the data, you select data points that show the right relationships in the data. The answers are “labels”, the categories/columns/attributes are “features” and the values are…values. Then you select an algorithm to compute the outcome. (Often you choose more than one) You run the program on the data set, and check to see if you got the right answer from the test set. Once you perform the experiment, you select the best model. This is the final output – the model is then used against more data to get the answers you need
- 22. Page 22 Supervised Learning Car Not Car
- 23. Page 23 Unsupervised Learning Used when you want to find unknown answers – mostly groupings - directly from data No simple way to evaluate accuracy of what you learn Evaluates more vectors, groups into sets or classifications Start with the data Apply algorithm Evaluate groups
- 24. Page 24 Unsupervised Learning Example1 example A Example 2 example B Example 3 example C example A example B example C Example 1 Example 2 Example 3 The clustering strategies have more tendency to transitively group points even if they are not nearby in feature space
- 25. Page 25 Cross-Validation andModel Evaluation Cross-validation is a method of ensuring that models generalize to data they have not been trained to fit Given any collection of data points, a model can be developed that fits the data exactly; however, this model will have no predictive power
- 26. Page 26 Evaluating PredictiveModels Presentation title Model evaluation involves a combination of objective criteria and subjective judgment Objective Measures Gain or Lift Sensitivity Accuracy Others Subjective Considerations Business intuition Explainability Simplicity Usefulness
- 27. Page 27 Gain orLift Lift is a measure of the effectiveness of a predictive model calculated as the ratio between the results obtained with and without the predictive model. Cumulative gains and lift charts are visual aids for measuring model performance Both charts consist of a lift curve and a baseline The greater the area between the lift curve and the baseline, the better
- 28. Page 28 Sensitivity A ReceiverOperating Characteristic (ROC) curve is a plot of test sensitivity as a function of (1 - specificity) for several possible (arbitrary) cut off values. The curve illustrates the trade off between type I and type II errors in a given test. The closer the curve follows the left- hand border and then the top border of the ROC space, the more accurate the test, and the area under the curve is a measure of accuracy.
- 29.
- 30.
- 31.
- 32.
- 33. Page 33 Data Flowand Architecture Stream Analytics TransformIngest Web logs Present & decide IoT, Mobile Devices etc. Social Data Event Hubs HDInsight Azure Data Factory Azure SQL DB Azure Data Lake Azure Machine Learning (Fraud detection etc.) Power BI Web dashboards Mobile devices DW / Long-term storage Predictive analytics Event & data producers Azure SQL DW
- 34. Page 34 Process real-timedata in Azure using a simple SQL language Consumes millions of real-time events from Event Hub collected from devices, sensors, infrastructure, and applications Performs time-sensitive analysis using SQL-like language against multiple real-time streams and reference data Outputs to persistent stores, dashboards or back to devices Point of Service Devices Self Checkout Stations Kiosks Smart Phones Slates/ Tablets PCs/ Laptops Servers Digital Signs Diagnostic EquipmentRemote Medical Monitors Logic Controllers Specialized DevicesThin Clients Handhelds Security POS Terminals Automation Devices Vending Machines Kinect ATM
- 35. Page 35 Fully managedservice to support orchestration of data movement and processing Connect to relational or non-relational data that is on- premises or in the cloud Single pane of glass to monitor and manage data processing pipelines. Publish to Power BI Compose and orchestrate data services at scale C# MapReduce Trusted data BI & analytics Hive Pig Stored Procedures Machine Learning
- 36. Page 36 ML Algorithmsare best of breed and embrace OSS • MS + R + Python + BYOA ML Studio for productive development • Faster experiments results in faster improvements • Visual Workflows & ML Experiments ML Operationalization to remove deployment friction • Build entire ML Apps & Deploy as Cloud APIs ML Gallery • Provide ML applications like apps in an ‘app store’ • Publish/consume APIs in a 2 sided market Help organizations eliminate undifferentiated heavy lifting Powerful predictive analytics in Azure Azure Machine Learning
- 37. Page 37 Power BIinvestments New data visualizations and touch-optimized exploration in HTML5 Power BI mobile apps across devices including iPad and iPhone Support for new data sources including SalesForce.com, Dynamics CRM online and SQL Server Analysis Services Dashboard Tree Map Power BI dashboards and KPIs for monitoring the health of your business
- 38.
- 39. Page 39 Demo CortanaSuite
- 40. Page 40 Vehicle TelemetryArchitecture Event Hubs for ingesting millions of vehicle telemetry events into Azure. Stream Analytics for gaining real-time insights on vehicle health and persists that data into long-term storage for richer batch analytics. Machine Learning for anomaly detection in real-time and batch processing to gain predictive insights. HDInsight is leveraged to transform data at scale Data Factory handles orchestration, scheduling, resource management and monitoring of the batch processing pipeline. Power BI gives this solution a rich dashboard for real-time data and predictive analytics visualizations.
- 41. Page 41 Microsft AzureLearning Machine Data It’s all about the data. Here’s where you will acquire, compile, and analyze testing and training data sets for use in creating Azure Machine Learning predictive models. Create the model Use various machine learning algorithms to create new models that are capable of making predictions based on inferences about the data sets. Evaluate the model Examine the accuracy of new predictive models based on ability to predict the correct outcome, when both the input and output values are known in advance. Accuracy is measured in terms of confidence factor approaching the whole number one. Refine and evaluate the model Compare, contrast, and combine alternate predictive models to find the right combination(s) that can consistently produce the most accurate results. Deploy the model Expose the new predictive model as a scalable cloud web service, one that is easily accessible over the Internet by any web browser or mobile client. Test and use the model Implement the new predictive model web service in a test or production application scenario.
- 42. Page 42 Azure MachineLearning algorithms Classification algorithms These are used to classify data into different categories that can then be used to predict one or more discrete variables, based on the other attributes in the dataset. Regression algorithms These are used to predict one or more continuous variables, such as profit or loss, based on other attributes in the dataset. Clustering algorithms These determine natural groupings and patterns in datasets and are used to predict grouping classifications for a given variable.
- 43.
Editor's Notes
- #6 Machine learning can be described as computing systems that improve with experience. It can also be described as a method of turning data into software. “The goal of machine learning is to program computers to use example data or past experience to solve a given problem.” Introduction to Machine Learning, 2nd Edition, MIT Press Whatever term is used, the results remain the same; data scientists have successfully developed methods of creating software “models” that are trained from huge volumes of data and then used to predict certain patterns, trends, and outcomes.
- #7 Predictive analytics is can be simply defined as a way to scientifically use the past to predict the future to help drive desired outcomes.
- #8 Machine learning can be described as computing systems that improve with experience. It can also be described as a method of turning data into software. Whatever term is used, the results remain the same; data scientists have successfully developed methods of creating software “models” that are trained from huge volumes of data and then used to predict certain patterns, trends, and outcomes. Apprendimento Supervisionato Apprendimento Non Supervisionato
- #9 Tecniche alla base del machine learning sono utilizzate da noi tutti i giorni….ad esempio il linguaggio Che è composto da un Alfabeto ( dati ) e da regole come il suono di una vocale o di una parola o da regole come la costruzione di una frase La comprensione avviene da una conoscenza pregressa. I Dati + Rule = Machine Learning
- #10 Under traditional programming models, programs and data are processed by the computer to produce a desired output, such as using programs to process data and produce a report When working with machine learning, the processing paradigm is altered dramatically. The data and the desired output are reverse-engineered by the computer to produce a new program The power of this new program is that it can effectively “predict” the output, based on the supplied input data. The primary benefit of this approach is that the resulting “program” that is developed has been trained (via massive quantities of learning data) and finely tuned (via feedback data about the desired output) and is now capable of predicting the likelihood of a desired output based on the provided data.
- #11 A classic example of predictive analytics can be found everyday on Amazon.com; there, every time you search for an item, you will be presented with an upsell section on the webpage that offers you additional catalog items because “customers who bought this item also bought” those items. This is a great example of using predictive analytics and the psychology of human buying patterns to create a highly effective marketing strategy
- #12 Many examples of predictive analytics can be found literally everywhere today in our society: Spam/junk email filters These are based on the content, headers, origins, and even user behaviors (for example, always delete emails from this sender). Mortgage applications Typically, your mortgage loan and credit worthiness is determined by advanced predictive analytic algorithm engines. Various forms of pattern recognition These include optical character recognition (OCR) for routing your daily postal mail, speech recognition on your smart phone, and even facial recognition for advanced security systems. Life insurance Examples include calculating mortality rates, life expectancy, premiums, and payouts. Medical insurance Insurers attempt to determine future medical expenses based on historical medical claims and similar patient backgrounds. Liability/property insurance Companies can analyze coverage risks for automobile and home owners based on demographics. Credit card fraud detection This process is based on usage and activity patterns. In the past year, the number of credit card transactions has topped 1 billion. The popularity of contactless payments via near-field communications (NFC) has also increased dramatically over the past year due to smart phone integration. Airline flights Airlines calculate fees, schedules, and revenues based on prior air travel patterns and flight data. Web search page results Predictive analytics help determine which ads, recommendations, and display sequences to render on the page. Predictive maintenance This is used with almost everything we can monitor: planes, trains, elevators, cars, and yes, even data centers. Health care Predictive analytics are in widespread use to help determine patient outcomes and future care based on historical data and pattern matching across similar patient data sets.
- #15 Regression analysis is a statistical process for estimating the relationships among variables. It includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between adependent variable and one or more independent variables (or 'predictors')
- #17 Supervised learning is a type of machine learning algorithm that uses known datasets to create a model that can then make predictions. The known data sets are called and include input data elements along with known response values In the case of unsupervised machine learning, the task of making predictions becomes much harder. In this scenario, the machine learning algorithms are not provided with any kind of known data inputs or known outputs to generate a new predictive model. In the case of unsupervised machine learning, the success of the new predictive model depends entirely on the ability to infer and identify patterns, structures, and relationships in the incoming data set.
- #18 Classification algorithms These are used to classify data into different categories that can then be used to predict one or more discrete variables, based on the other attributes in the dataset. Regression algorithms These are used to predict one or more continuous variables, such as profit or loss, based on other attributes in the dataset. Clustering algorithms These determine natural groupings and patterns in datasets and are used to predict grouping classifications for a given variable. One of the most common unsupervised learning algorithms is known as which is used to find hidden patterns or groupings within data sets. Some common examples of cluster analysis classifications would include the following: Socioeconomic tiers Income, education, profession, age, number of children, size of city or residence, and so on. Psychographic data Personal interests, lifestyle, motivation, values, involvement. Social network graphs Groups of people related to you by family, friends, work, schools, professional associations, and so on. Purchasing patterns Price range, type of media used, intensity of use, choice of retail outlet, fidelity, buyer or nonbuyer, buying intensity. The other type of approach to unsupervised machine learning is to use a reward system, rather than any kind of teaching aids, as are commonly used in supervised learning. Positive and negative rewards are used to provide feedback to the predictive model when it has been successful.
- #19 In statistical modeling, regression analysis is a statistical process for estimating the relationships among variables. It includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between adependent variable and one or more independent variables (or 'predictors')
- #22 Features or vectors Known data that is used as an input element for making a prediction. Labels or supervisory signal Represents the known outcomes for the corresponding features for the input record. Not used (default) Not used by predictive algorithms for inferring a new predictive model
- #29 In statistics, a receiver operating characteristic (ROC), or ROC curve, is a graphical plot that illustrates the performance of abinary classifier system as its discrimination threshold is varied. The curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings.
- #30 Watson is the first cognitive system in the world Offer a lot of service like face, voice , image recognition , question & answer But in this moment don’t offer any types or configurations of tool to costumize these services.
- #31 Natural interaction with humans For example speech and emotions Not only used by web applications, but also in physical world Processing of semi-structured data and big amounts of data For example language classifications and image recognitions Trained and continuously improved via machine and deep learning For example search (retrieve and rank) Leverages context to improve service qualities
- #32 A fully managed Big Data and Advanced Analytics Suite that enables you to transform your data into intelligent action.
- #33 From data to decisions and actions A fully managed Big Data and Advanced Analytics Suite that enables you to transform your data into intelligent action.
- #38 Areas of investment for Power BI Power BI dashboards and KPIs – Power BI will provide new capabilities to allow for the creation of live dashboards and easy to create KPIs that can be pinned to the dashboard - from which you can drill into underlying Power View reports for additional detail and data exploration. New data visualizations – we will introduce a host of new data visualizations for Power BI. eg. Tree Map. Touch optimized data exploration in Power View HTML5 – Power View will gain new touch optimized data exploration features allowing users to explore data more easily on touch devices. Power BI mobile applications – beyond the mobile experiences provide through HTML5 we also continue to invest in providing native mobile apps and will soon support both iPad and iPhone. Support for new data sources such as SalesForce.com and Dynamics CRM – new data sources will be added such as SalesFoce.com and tighter integration with CRM online. Support for SQL Server Analysis Services – Power Query will support SQL Server Analysis Services as a data source. Power BI will also support interactive query directly to SQL Server Analysis services on-premises. This will be to both tabular and multidimensional cubes. This will allow organizations to managing and maintain their existing Analysis Services models without having to move their data to the cloud. Users will be able to view and explore their data from Power View in Power BI and the system will directly connect and query against SQL Server Analysis Service on-premises.
- #39 https://www.microsoft.com/cognitive-services/en-us/computer-vision-api http://vision.alchemy.ai/ https://tone-analyzer-demo.mybluemix.net/ https://text-analytics-demo.azurewebsites.net/
- #40 https://gallery.cortanaintelligence.com https://powerbi.microsoft.com/en-us/
- #42 The basic process of creating Azure Machine Learning solutions is composed of a repeatable pattern of workflow steps that are designed to help you create a new predictive analytics solution in no time. The basic steps in the process are summarized in Figure Data It’s all about the data. Here’s where you will acquire, compile, and analyze testing and training data sets for use in creating Azure Machine Learning predictive models. Create the model Use various machine learning algorithms to create new models that are capable of making predictions based on inferences about the data sets. Evaluate the model Examine the accuracy of new predictive models based on ability to predict the correct outcome, when both the input and output values are known in advance. Accuracy is measured in terms of confidence factor approaching the whole number one. Refine and evaluate the model Compare, contrast, and combine alternate predictive models to find the right combination(s) that can consistently produce the most accurate results. Deploy the model Expose the new predictive model as a scalable cloud web service, one that is easily accessible over the Internet by any web browser or mobile client. Test and use the model Implement the new predictive model web service in a test or production application scenario. Add manual or automatic feedback loops for continuous