Extending the Reach of R to the Enterprise with TERR and Spotfire


Published on

An overview of how TIBCO integrates dynamic, interactive visual applications in Spotfire with predictive and advanced analytics in the R language, using TIBCO Enterprise Runtime for R--our R-compatible, enterprise-grade platform for the R language.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Extending the Reach of R to the Enterprise with TERR and Spotfire

  1. 1. Extending the Reach of R to the Enterprise Lou Bajuk-Yorgan Sr. Dir., Product Management TIBCO Spotfire 1 © Copyright 2000-2014 TIBCO Software Inc.
  2. 2. Extending the Reach of R to the Enterprise • TIBCO, S+, and embracing R in Spotfire • Challenges of R for Enterprise applications • TIBCO Enterprise Runtime for R (TERR) • Benefits for organizations (and individuals) who use R • Examples of TERR integration and performance • Learn more and try it yourself -2
  3. 3. Our Journey to TERR • John Chambers developed the S language at Bell Labs – Starting in the mid 70’s • Insightful (Statsci) founded to commercial S as S+ in 1987 – The “plus”: statistical libraries, documentation, and support – Later focus on commercial users, ease of use, server integration • R: development begun by Ross Ihaka and Robert Gentleman at University of Auckland in mid 90’s • Insightful acquired by TIBCO in 2008 – Spotfire (for Data Discovery and Visualization) acquired in 2007 • Focus shifted to applying Predictive Analytics in Spotfire – Step 1: Embrace R -3
  4. 4. Predictive Analytics with Spotfire Easily provide targeted, relevant predictive analytics to business users to improve decision making • Ensure compliance & proper usage • Share best practices and consistent workflows • Get the answer & do “What If?” analyses when needed • Leverage investments in R, S+, SAS, MATLAB, … Powerful Predictive Analytics tools for Spotfire analysts • Integrated into Spotfire workflows • Easily create, evaluate, and share Predictive Models • Add Forecasts with a single click Benefits of Predictive Analytics to a spectrum of users • Increase confidence & effectiveness in decision-making – – – Reduce uncertainty Discover meaningful patterns, important data Maximize ROI • Anticipate and react to emerging trends • Reduce/manage risk – • Scenario planning, forecasts, fraud detection Forecast specific behavior, preemptively act on it – Increase upsell, decrease churn
  5. 5. Embracing R • Spotfire Statistics Server – Integration of R & S+ into Spotfire applications • – Later added SAS® & MATLAB® Leverage the interactive visualizations of Spotfire • Contribute to the R community • Well received—but our Enterprise customers need more – – -5 R provides tremendous benefits to statisticians But large enterprises are often challenged to leverage that value
  6. 6. Enterprise Challenges for R • Core R engine struggles with Big Data – – • R was not built for enterprise usage and integration – – • Built as an academic tool for research and teaching Software vendors attempting to use R in ways it was never intended GPL great for statisticians, but limits enterprise innovation and investment – – • Customers don’t use R, or reimplement R code in specialized libraries or other languages Lose agility & consistency, delay time to production, lose opportunities Viral open source licensing risks commercial IP Large vendors avoid tight integration due to open source concerns Free to acquire, but costly to maintain – – Version incompatibilities, variable quality in packages Lack of enterprise-level technical support 6 © Copyright 2000-2013 TIBCO Software Inc.
  7. 7. TIBCO Enterprise Runtime for R (TERR) • Unique, enterprise-quality implementation of the R language – Fundamentally different • • • – TIBCO IP: Not open source/GPL • • • – Independent implementation Licensable for embedding and redistribution by partners Enables implementation of transparent big data handling Broad compatibility with R functions and 1400+ CRAN packages • • New architecture, developed from the ground up Based on our long history and expertise with S+ Faster, more robust and more memory-efficient than R Ongoing effort to broaden our coverage of R Extends the Reach of R to the Enterprise – – – Develop in R, deploy on TERR Rapidly iterate prototyping to production without recoding/retesting—more rapidly respond to changing business conditions Easily integrate R-language analytics consistently across organization—into grids, BI applications, event-driven analytics, etc. © Copyright 2000-2013 TIBCO Software Inc.
  8. 8. Leveraging TERR TERR in Spotfire TERR in Statistics Services Embeddable TERR Engine Ad hoc tools and interactive applications powered by advanced analytics • Spotfire Analytics platform: interactive visualization & data discovery, easily build and share applications, broad data access, etc. Distributed analytics • Managed pools of engines • Load balancing, queuing, failover, parallelization, etc. • High level APIs for loose integration, data i/o (C#, Java) • Central management of analytics, R packages Custom (tight) integration, batch, existing grids, etc. • Faster than R, more robust, better memory management, fully supported • Low level APIs for tight integration • Integrated into TIBCO products: CEP, Cloud Compute, … © Copyright 2000-2013 TIBCO Software Inc.
  9. 9. Providing Value for individuals who use R • Not seeking to displace R from statistician’s desktops – • Contribute to the R community – – • As we port from S+ or develop for TERR • Supports “Develop in Open Source R, Deploy on TERR” • E.g., splusTimeSeries, splusTimeDate, sjdbc TERR Developer Edition – – – -9 Sponsor useR conferences, contribute to R Foundation Contribute bug reports and propose fixes to R core Contribute packages to CRAN – • Enterprise platform for the deployment and integration of your work—without having to rewrite it! Full version of TERR engine for testing code prior to deployment • Compatible with RStudio & ESS Emacs Free for non-production use Supported through Community site
  10. 10. Example 1: TERR vs. R Raw Performance One specific example • Non-optimal, non-vectorized, real-world R script • For loop with row by row processing for (i in seq(1,length=nrow(df))) { …process each customer record… } Results • TERR is ~35x faster for 50K rows, 150x faster for 500K rows • No code modification required We are looking for more real-world performance tests! • On average 2-10x faster than R in microtests
  11. 11. Example 2: Spotfire Forecast Tool • Forecast Tool – Easily add Forecasts to Visualizations by right click menu – Advanced users can tune settings – Uses embedded TERR engine • Benefits – Extend the power of Predictive Analytics for ad hoc analysis to all Spotfire users – Easy entry point to Spotfire Predictive Analytics
  12. 12. TERR integration with TIBCO StreamBase • Event-Driven analysis in TIBCO Spotfire Event Analytics – • Apply predictive models in real-time decision making – – – – • Process monitoring, analysis, and optimization Best marketing offer Customer churn Predictive Maintenance Yield optimization Rapidly develop and iterate models in production – Respond to changing opportunities and threats 12
  13. 13. TIBCO Cloud Compute Grid • High performance computing on the cloud – Available on TIBCO Cloud Marketplace – TERR, Java and .NET computations • Robust DataSynapse GridServer architecture – Used by Wall Street to manage 10K’s nodes – Java, .NET, and REST APIs (JSON) • Perfect for pure computational work – Vastly easier to use for applications like Monte Carlo simulations than Map-Reduce – Run complex statistical models multiple orders of magnitude faster than open source R on a single computer – Unparalleled scalability without upfront capital investment • Easy to get started – Uses your Amazon EC2 account
  14. 14. Demos • TERR in Spotfire – Fraud Detection Application – Data Functions: using the R language in Spotfire – Forecast Tool
  15. 15. Learn more and Try it yourself • TERR Community at TIBCOmmunity.com – – – – • TERR Developer Edition – – • Full version of TERR engine for testing code prior to deployment Supported through TIBCOmmunity, download via tap.tibco.com TIBCO Cloud Compute Grid – • Resources, FAQs, Forums Details of R coverage Product documentation & download More info at spotfire.tibco.com/terr https://marketplace.cloud.tibco.com We want your feedback and input! – – – Real world performance tests Package & R coverage prioritization Via TERR Community, or contact me lbajuk@tibco.com or @loubajuk