Open source analytics

  • 1,994 views
Uploaded on

how open source has helped analytics

how open source has helped analytics

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
1,994
On Slideshare
0
From Embeds
0
Number of Embeds
9

Actions

Shares
Downloads
34
Comments
1
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Open Source in Analytics
  • 2. Introduction IIML ,DCE Founder Decisionstats.com Author R for Business Analytics
  • 3. Brief History of Analytics SAS and SPSS led from 1970-s to early 2000s SAS leads market but very expensive IBM bought SPSS but still not open source R, Python and Hadoop Challenged this
  • 4. Analytics Sub Components Data Storage Data Querying Data Summarization Data Visualization Statistical Routines
  • 5. Analytics Sub Components Data Storage Data Querying Data Summarization Data Visualization Statistical Routines Proprietary Open Source OracleDBMS SQL Server Business Objects SAP SQL, SAS,Crystal Reports Tableau SAS,SPSS
  • 6. Analytics Sub Components Data Storage Data Querying Data Summarization Data Visualization Statistical Routines Proprietary Open Source OracleDBMS SQL Server MySQL, NoSQL, Hadoop Business Objects SAP Pentaho, Jaspersoft SQL, SAS,Crystal Reports Still SQL,Pig, Hive Tableau R,Python,Javascript SAS,SPSS R,Python,RapidMiner
  • 7. Analytics using Python ● pandas http://pandas.pydata.org/ High-performance, easy-to-use data structures and data analysis tools ● scikit-learn http://scikit-learn.org/stable/ Simple and efficient tools for data mining and data analysis and built on NumPy, SciPy, and matplotlib ● NumPy http://www.numpy.org/ ● SciPy http://www.scipy.org/scipylib/index.html ● matplotlib http://matplotlib.org/ ● statsmodels http://statsmodels.sourceforge.net/# Statsmodels is a Python module that allows users to explore data, estimate statistical models, and perform statistical tests. An extensive list of descriptive statistics, statistical tests, plotting functions, and result statistics are available ● iPython http://ipython.org/ interactive computing
  • 8. Analytics using R http://www.r-project.org/ ● RStudio and Revolution Analytics ● sqldf https://code.google.com/p/sqldf/ and RODBC http://cran.r-project.org/web/packages/RODBC/index.html ● ggplot2 http://ggplot2.org/ and ggmap and shiny ● RHadoop et al https://github.com/RevolutionAnalytics/RHadoop ● car, stats, forecast, sna,tm ● rattle and Rcommander (with plugins) More at http://rforanalytics.wordpress.com/
  • 9. Analytics using R http://www.revolutionanalytics.com/
  • 10. Analytics using R http://www.revolutionanalytics.com/
  • 11. Analytics using R <blatant self promotion> http://www.amazon.com/R-Business-Analytics-A-Ohri/dp/1461443423 R for Business Analytics looks at some of the most common tasks performed by business analysts and helps the user navigate the wealth of information in R and its packages. With this information the reader can select the packages that can help process the analytical tasks with minimum effort and maximum usefulness . The use of Graphical User Interfaces (GUI) is emphasized in this book to further cut down and bend the famous learning curve in learning R. </blatant self promotion>
  • 12. Analytics using Rapid Miner Early adopter of open source analytics Recently moved from Germany to USA following PE infusion One of the first marketplace for analytics extensions http://marketplace.rapid-i.com/UpdateServer/ One of the best GUI - Drag and Drop using flow
  • 13. Analytics using Rapid Miner
  • 14. Analytics using Rapid Miner
  • 15. Analytics using other languages Julia- faster than R http://julialang.org/ Julia is a high-level, high-performance dynamic programming language for technical computing, with syntax that is familiar to users of other technical computing environments. It provides a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive mathematical function library. The library, largely written in Julia itself, also integrates mature, best-of- breed C and Fortran libraries for linear algebra, random number generation, signal processing, and string processing.
  • 16. IJulia !!
  • 17. IJulia !!
  • 18. Analytics using other languages Clojure- for JVM http://clojure.org/ Clojure is a dynamic programming language that targets the Java Virtual Machine . It is designed to be a general-purpose language, combining the approachability and interactive development of a scripting language with an efficient and robust infrastructure for multithreaded programming. Clojure is a compiled language - it compiles directly to JVM bytecode, yet remains completely dynamic. Every feature supported by Clojure is supported at runtime. Clojure is a dialect of Lisp https://bigml.com/gallery/models
  • 19. Analytics using other languages bigml.com (using clojure) https://bigml.com/gallery/models
  • 20. Analytics using other languages Scala- for big data analytics http://www.scala-lang.org/ ● A Scalable language ● Object-Oriented ● Functional ● Seamless Java Interop ● Functions are Objects ● Future-Proof ● Fun
  • 21. Analytics using Jaspersoft OLAP BIG DATA (offered through cloud, mobile)
  • 22. Analytics using Pentaho Basically Weka Reporting as well Complete BI and Analytics Stack
  • 23. Weka
  • 24. Hadoop http://hadoop.apache.org/
  • 25. Hadoop- evolving ecosystem
  • 26. Hadoop- evolving ecosystem
  • 27. Hadoop- evolving ecosystem
  • 28. R http://www.r-project.org/ Open Source Free 5000+ Packages Growing Faster >2 million users RAM constraints??
  • 29. R http://www.r-project.org/ Object Oriented has GUI and IDE has Commercial offerings
  • 30. R http://www.r-project.org/ Object Oriented has GUI and IDE has Commercial offerings
  • 31. R - Rattle- Data Mining GUI http://www.r-project.org/ Object Oriented has GUI and IDE has Commercial offerings
  • 32. R - R Commander http://www.r-project.org/ Object Oriented has GUI and IDE has Commercial offerings
  • 33. R -R Studio
  • 34. R -Revolution Analytics Free for Academics World Wide !! RevoScaleR package for Big Data Recommended Install - http://info.revolutionanalytics.com/free-academic.html
  • 35. R -Revolution Analytics Free for Academics World Wide !! RevoScaleR package for Big Data
  • 36. R -Big Data Packages http://cran.r-project.org/web/views/HighPerformanceComputing.html ● The RHIPE package, started by Saptarshi Guha and now developed by a core team via GitHub, provides an interface between R and Hadoop for analysis of large complex data wholly from within R using the Divide and Recombine approach to big data. ( link ) ● The rmr package by Revolution Analytics also provides an interface between R and Hadoop for a Map/Reduce programming framework. ( link ) ● A related package, segue package by Long, permits easy execution of embarassingly parallel task on Elastic Map Reduce (EMR) at Amazon. ( link ) ● The RProtoBuf package provides an interface to Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. This package can be used in R code to read data streams from other systems in a distributed MapReduce setting where data is serialized and passed back and forth between tasks. ● The HistogramTools package provides a number of routines useful for the construction, aggregation, manipulation, and plotting of large numbers of Histograms such as those created by Mappers in a MapReduce application.
  • 37. Terrific Data Mining using R GUI
  • 38. Great Data Visualization using R GUI
  • 39. So many packages- CRAN Views to the rescue http://cran.r-project.org/web/views/ Bayesian Bayesian Inference ChemPhys Chemometrics and Computational Physics ClinicalTrials Clinical Trial Design, Monitoring, and Analysis Cluster Cluster Analysis & Finite Mixture Models DifferentialEquations Differential Equations Distributions Probability Distributions Econometrics Computational Econometrics Environmetrics Analysis of Ecological and Environmental Data ExperimentalDesign Design of Experiments (DoE) & Analysis of Experimental Data Finance Empirical Finance Genetics Statistical Genetics Graphics Graphic Displays & Dynamic Graphics & Graphic Devices & Visualization HighPerformanceComputing High-Performance and Parallel Computing with R MachineLearning Machine Learning & Statistical Learning MedicalImaging Medical Image Analysis MetaAnalysis Meta-Analysis Multivariate Multivariate Statistics NaturalLanguageProcessing Natural Language Processing
  • 40. So many packages- CRAN Views to the rescue http://cran.r-project.org/web/views/ NumericalMathematics Numerical Mathematics OfficialStatistics Official Statistics & Survey Methodology Optimization Optimization and Mathematical Programming Pharmacokinetics Analysis of Pharmacokinetic Data Phylogenetics Phylogenetics, Especially Comparative Methods Psychometrics Psychometric Models and Methods ReproducibleResearch Reproducible Research Robust Robust Statistical Methods SocialSciences Statistics for the Social Sciences Spatial Analysis of Spatial Data SpatioTemporal Handling and Analyzing Spatio-Temporal Data Survival Survival Analysis TimeSeries Time Series Analysis WebTechnologies Web Technologies and Services gR gRaphical Models in R
  • 41. R in the Browser http://www.r-fiddle.org/#/ http://statace.com/ http://www.rstudio.com/ide/server/
  • 42. R -Hadoop Packages https://github.com/RevolutionAnalytics/RHadoop/wiki ● plyrmr - higher level plyr-like data processing for structured data, powered by rmr ● rmr - functions providing Hadoop MapReduce functionality in R ● rhdfs - functions providing file management of the HDFS from within R ● rhbase - functions providing database management for the HBase distributed database from within R http://amplab-extras.github.io/SparkR-pkg/ SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. https://github.com/nexr/RHive RHive is an R extension facilitating distributed computing via HIVE query. RHive allows easy usage of HQL(Hive SQL) in R, and allows easy usage of R objects and R functions in Hive.
  • 43. R - Cloud Computing http://cran.r-project.org/web/views/WebTechnologies.html
  • 44. R -Big Data Packages http://cran.r-project.org/web/views/HighPerformanceComputing.html Large memory and out-of-memory data ● The biglm package by Lumley uses incremental computations to offer lm() and glm() functionality to data sets stored outside of R's main memory. ● The ff package by Adler et al. offers file-based access to data sets that are too large to be loaded into memory, along with a number of higher-level functions. ● The bigmemory package by Kane and Emerson permits storing large objects such as matrices in memory (as well as via files) and uses external pointer objects to refer to them. . ● A large number of database packages, and database-alike packages (such as sqldf by Grothendieck and data.table ● The HadoopStreaming package provides a framework for writing map/reduce scripts for use in Hadoop Streaming; it also facilitates operating on data in a streaming fashion which does not require Hadoop. ● The speedglm package permits to fit (generalised) linear models to large data. ● The biglars package by Seligman et al can use the ff to support large-than-memory datasets for least-angle regression, lasso and stepwise regression. ● The bigrf package provides a Random Forests implementation with support for parellel execution and large memory. ● The MonetDB.R package allows R to access the MonetDB column-oriented, open source database system as a backend.
  • 45. Data Scientist Tool Kit ● web scraping ● visualization ● machine learning ● data mining ● modeling ● sna ● social media analytics ● web analytics ● reproducible research ● TS forecasting ● spatial analysis ● data storage ● data querying
  • 46. Data Scientist Programming Skills Java http://www.learnjavaonline.org/ Python http://www.codecademy.com/tracks/python SQL http://www.w3schools.com/sql/ R http://bigdatauniversity.com/bdu-wp/bdu-course/introduction-to-data-analysis-using-r/ http://www.statmethods.net/ Hadoop http://hortonworks.com/hadoop-training/ Linuxhttps://github.com/WilliamHackmore/linuxgems/blob/master/cheat_sheet.org.sh
  • 47. Other place to learn MOOCs 1 https://www.edx.org/ 2 https://www.coursera.org/ 3 https://www.udacity.com/ 4 https://www.udemy.com/ Books Courses Workshops
  • 48. Summary Open source has greatly helped cut down cost of software in analytics The benefits of analytics continue to be many Added with Big Data and Cloud and MOOCs -----total cost to geeks is much lower !!
  • 49. Thanks Contact and Feedback- ohri2007@gmail.com via http://linkedin.com/in/ajayohri