TIBCO Advanced Analytics Meetup (TAAM) November 2015

TIBCO Advanced Analytics
Meetup : Q4 2015
Michael O’Connell
Chief Data Scientist
moconnell@tibco.com
@MichOConnell
November 2015

• TIBCO Analytics & Data Science (Michael)
• Data Analysis Pipeline
• Visual Analytics & Dashboards (Catalina Herrera)
• Building Dashboards on the fly
• Predictive Analytics (Peter Shaw, Anna Nowakowska)
• Simple Data Functions (R and TERR)
• Customer Analytics (Churn & Value)
• Data Access & APIs (Andrew Berridge, Mathew Lee)
• Iron Python
• Advanced GeoAnalytics
• Wrap-Up / Questions (Michael)
• Resources, Training, Links
Increase
Productivity
Grow
Revenue
Value
Reduce
Risk
ROI
TIBCO Advanced Analytics Meetup (TAAM) Agenda
© Copyright 2000-2015 TIBCO Software Inc.

Data Access
& Prep
Exploratory
Data Analysis
Features
Visual
Dashboard
Model &
Predict
Deploy
Champion
Model
Test &
Learn
Channel
Social
Loyalty
Campaign
Filter
Map
Merge
Shape
Propensity
Affinity
ImproveGuided -------- Deploy -------- In-LineExplore Data
Aggregate
Prepare DataBusiness Case
Increase
Productivity
Grow
Revenue
Ensemble
Forest
Regression
Additive
Models
Segment
Visualize
Pricing
Promotion
Challenger
Models
At Rest
In Motion
Value
Theses
Reduce
Risk
ROI
Value
Dashboard
Updates
Data a Insight a Action

Spotfire Platform
Spotfire
Desktop

Spotfire Platform

Immediate
Long-Term
Competitive AdvantageValue to the Organization
TIBCO is the only analytics platform that provides business
value across this Analytics Spectrum
Self-service
Dashboards
API and CEP
Predictive and
Prescriptive Analytics
Measure Diagnose Predict Optimize Operationalize Automate
Analytics Maturity
Insight to Action

Immediate
Long-Term
Self-service
Dashboards
Analytics Maturity
Predictive and
API and CEP
Insight to Action

Immediate
Long-Term
Self-service
Dashboards
Analytics Maturity
API and CEP
Predictive and
Insight to Action

Spotfire Visual Analytics - Interactive Brush-Linked

Spotfire Visual Analytics + R Extensions
3D rotate SurfacePolar
Contour Network Funnel

Spotfire + JS Extensions
Sankey
Venn
Chord
Donut
Gantt
Waterfall
Dials

Dashboards and Themes – KPIs

Dashboards and Themes – Company Specific

Dashboards and Themes – Color

Map Layer
Marker Layer Feature Layer
Image LayerWMS Layer
Map Layers

Roads
Labels
Borders
Standard map
Standard map - light
Basic map
Basic map - light
Base Map Options

• Color
• Shape
• Size
• Relative amounts
• Size
Marker Layer Feature Layer
• Color
Marker or Feature Layer
• Tooltip
• Labels
Map Elements

Building Dashboards
Catalina Herrera

Jaspersoft Pixel-Perfect Embedded Reports

Analytic Workspaces & Analytic Fabric
APIs
Search,Sharingetc.
Business Analysts Report Developers
Data Discovery Analytics Dashboards Reports Embedding
Analytic Workspaces
Analytic environments
Analytic Fabric
Connectivity layer
Data
Professionals
25%
Business
Professionals
75%
Business
Applications
Communication
& Collaboration
Platforms

Immediate
Long-Term
Self-service
Dashboards
Analytics Maturity
Predictive and
APIs and CEP
From Dashboards to Value

Big Data Analytics

Data a Insight a Action
Data Access
& Prep
Exploratory
Data Analysis
Features
Visual
Dashboard
Model &
Predict
Deploy
Model
Test &
Learn
Channel
Churn
Loyalty
Campaign
Filter
Map
Merge
Shape
Propensity
Affinity
ImproveGuided -------- Deploy -------- In-LineExplore Data
Aggregate
Prepare DataBusiness Case
Increase
Productivity
Grow
Revenue
Ensemble
Forest
Regression
Additive
Models
Segment
Visualize
Pricing
Promotion
Challenger
Models
At Rest
In Motion
Value
Theses
Reduce
Risk
ROI
Purchase
Dashboard
Updates
Fast DataBig Data

TIBCO’s Enterprise Runtime for R (TERR)
• TIBCO has rewritten R as a Commercial Compute Engine
• Latest statistics scripting engine: S a S-PLUS® a R a TERR
• Runs R code including CRAN packages
• Engine internals rebuilt from scratch at low-level
• Redesigned data objects, memory management
• High performance + Big Data
• TERR is licensed from TIBCO
• TERR Installs with Spotfire Analyst / Desktop and other TIBCO products (CEP, Stat Services)
• Spotfire Server can manage all TERR / R scripts, artifacts for reuse
• Standalone Developer Edition: www.TIBCOmmunity.com
• Supported by TIBCO

Spotfire and TERR local TERR on server
Spotfire-TERR – Local and Server
• Build models on data using local
TERR engine embedded in
Spotfire
• Build models on big data directly in TERR on
server and display results in Spotfire
• Run TERR as parallel sessions on Hadoop cluster,
controlled and visualized in Spotfire
Data Source TERR
TSSS
Spotfire
Results
ODBC
JDBC
SDC
File
Data
Function
Larger Data
Modeling
Spotfire
Local
TERR
ODBC
JDBC
SDC
File
Data
Data Source
Both Spotfire and TERR can load data from any ODBC or JDBC compliant source or from
Spotfire Data Connections (SDC) or Spotfire Information Links stored in the Spotfire library.

Date
RPackages
1/1/2002 1/1/2003 1/1/2004 1/1/2005 1/1/2006 1/1/2007 1/1/2008 1/1/2009 1/1/2010 1/1/2011 1/1/2012 1/1/2013
5000
4500
4000
3500
3000
2500
2000
1500
1000
500
0
Number of R- or SAS posts to Stack Overflow by week.
(copyright by r4stats.com)
Number of contributed packages on CRAN
(http://cran.r-project.org/)
> 6,000 Packages !
R Community
126 R User Groups in US and EMEA 6,000 User-Contributed Packages Very Active R Community

Contextual Analytics
- Right click
- Forecasting
Contextual Analytics
- Menu
- Machine Learning
Simple Predictive Analytics in Spotfire

Extensible Predictive Analytics – Expressions
Lines by TERR( contourLines(x,y) )
Color by TERR( kmeans(x1,x2) )
Interactive Spotfire Analytics with R/TERR
- Expression – on the fly R code
- Points: kmeans(), …
- Lines: contourLines(), spline(), …
- Expression Functions – sharing

Analysis Workflows – Data Functions
Interactive Spotfire Analytics with R
- Data Function
- Robust Cluster Analysis
- Any Analysis in R / CRAN
Variables driving segments
- Random Forest
Revenue by product
- Color by segment

Analysis Workflows – Route Optimization

Predictive Analytics Dashboards

Predictive Analytics
Customer Analytics Workflows
Anna Nowakowska
Peter Shaw

Immediate
Long-Term
Self-service
Dashboards
Predictive and
Analytics Maturity
Event Processing
From Dashboards to Value

BIG DATA
AT REST
FAST DATA
IN MOTION
Insight to Action

Spotfire Data Access
DATA
SOURCES
XMLRDBMS
Flat
Files
CubesSpread-
sheets
Hadoop &
Big Data
stores
Analytical
DWs e.g.
Exadata
Event Data
Streams
Active
Spaces
In-Memory
Load data from
source in to
memory
In-Database
Leave data in DB
Dynamically load and
discard data to visualize
On-Demand
Dynamically swap
data in and out of
memory.
SQL
MDX
1010
0110

Custom GUI-driven
data access via SDK
Siebel
eBusiness
Local data sources
AccessExcel STDF
Drag-and-drop
MySQL
SQL Server
Oracle
Information Services
(join, transform, reusable,
parameterized, dynamic query
for in-memory use)
Databases
JDBC/ODBC
Hadoop
SFDC
PostgreSQL
Teradata
Netezza
Etc.XML
RDBMS
Flat
Files
Spread-
sheets
Web
Services
Oracle
E-Business
RDBMS
RDBMS
RDBMS
SAP BWSAP R/3 D
A
T
A
F
A
B
R
I
C
Salesforce
ODBC
OLE DB
SqlClient
Direct
connection
Oracle
TeradataAsterMS SSAS
Teradata
Direct Query
(dynamically query and retrieve data
for visualization and analysis)
Databases
MySQL
Etc.
OBIEE
Netezza
Hadoop
Spotfire Data Access

Winner of 2014 Strata Cloudera Award
For Best Advanced Analytics Application
Hadoop & Spark

TERR on the nodes of Hadoop Cluster
TERR in Action
• Hadoop cluster compute
• TIBCO Cloud Compute Grid
• TIBCO Streambase
• TIBCO Business Events
• KNIME
• Lavastorm
• Rstudio
• Teradata
• TIBCO Statistics Services
• TIBCO Spotfire
Hadoop & Spark

Non Geographic Data – Map Chart Layers

Non Geographic Data – Interactive Spotfire Charts
Airplane Seating Chart
Football Field Seating Chart
Spotfire Interactive Charts
created by TERR script

Spotfire API’s
Spotfire MapChart & GeoAnalytics
Andrew Berridge
Mathew Lee

Optimize
pricing Check for
fraud
Make offer
to customer
Restock
inventory
Reroute
transport
Give customer
service
Proactively
maintain machines
Fast Data Analytics – Interventions

Contact – Follow me on Twitter
Thank you!
Michael O’Connell, PhD
Chief Data Scientist
TIBCO
moconnell@tibco.com
@MichOConnell
http://about.me/moconnell
+1-919-7401560

Creating a TERR data function in
Spotfire
Peter Shaw

• Example shown here: Customer Churn
• Quick look at data
• Create data function in RStudio to draw curve on scatter plot
• Linear
• Quadratic
• Apply model to new data: Current customers.
• Create new column: Probability of churn
Outline

R code
For this demo, the data is read into the R environment directly.
Here Rstudio is configured to use the TERR engine as will be used in the data function.
Alternatively, the data function can use a save() command to save any incoming objects
to a disk file, and then these can be loaded into the interactive R environment.

Data function development: linear model

Quadratic model

Streamlined data function for prediction

• Thanks to Ujval Kamath and Ian Cook for developing early form of this demo
Thanks

TIBCO Advanced Analytics Meetup
Product Update Including APIs
Andrew Berridge, Michael O’Connell
TIBCO Data Science

65
Product Updates and APIs
• IronPython and APIs
• Why Use IronPython with the Spotfire APIs?
• Custom Sort Order API Example
• IronPython Authoring Improvements in Recent Versions of Spotfire
• Spotfire Updates
• Styling and Themes
• Annotation and Collaboration
Agenda

66
Two Books of Interest
A Picture is Worth a Thousand Tables
• Graphics in Life-Sciences
• Many useful lessons and analysis
examples for other industries
TIBCO Spotfire –
A Comprehensive Primer
• Covers how to use Spotfire
• Practical real-world examples
• A big section on IronPython, including documentation, examples
and code snippets
• Covers predictive, geoanalytics and many other areas of Spotfire

67
Why Use IronPython with the Spotfire APIs?
Apply Customizations and Automate Multiple Tasks With a Single Click!
• Apply customizations:
• Change axes
• Create new visualizations
• Set colorings, update color rules
• Produce analyses customized for users
• Extend features of Web Player (Consumer)
• Roll up multiple user actions into a single click
• Work with data:
• Set custom sort orders
• Set filters, mark data with “where” clauses
• Load data, refresh data
Using IronPython Automation
Services extension,
automate IronPython scripts
with Automation Services

68
Custom Sort Order API
Introduced in Spotfire 7.0
• Important and not well known!
• In the past, it was not possible to programmatically set a custom sort order
• This could lead to unexpected sorting:
• If not all values are known during analysis authoring;
• Then the data is refreshed, any new values will not be sorted accordingly
• API now allows sort order to be set (e.g. via an algorithm or external lookup
table)
• Can automate this via Automation Services Run Script extension; or via an
OnLoad IronPython script (triggered from JavaScript)

69
Significant Improvements Since Spotfire 6.0
• Improved debugging
• Line numbers are shown when exceptions are thrown (when using the script
editor)
• New dialog for managing scripts, accessed from Edit->Document
Properties menu
• One click to “Trust All” scripts (IronPython or JavaScript)
IronPython Authoring Improvements

70
Demo!
IronPython Authoring Improvements

71
Spotfire Desktop Client only
GUI for altering the sort
order…
Here we see fruits sorted in
size order
Refresh the data, for
example, to add “Plum”, then
it will automatically appear at
the bottom of the sort order
New API allows us to set a
custom sort order

72
Demo!

73
Annotate Visualizations; Comment on and Discuss Visualizations
• New ability to attach annotations to visualizations
• Annotations can be styled and placed anywhere on the visualization
• Collaboration allows conversations between users
• Conversations are recorded alongside the visualizations
Spotfire 7.5 Updates – Collaboration and Annotation

74
Spotfire 7.5 Updates – Collaboration and Annotation
Demo!

Spotfire 7.0 Updates – Theming and Styling
Spotfire can now be themed and styled to suit YOUR design!
• Every aspect of the visual styling is customizable
• Set colors and fonts
• Background images, styles and colors
• Borders, visual layout
• Apply the style of one DXP file to another

Examples

77
Demo!

78
Spotfire 7.5 Updates – Infographics

79
Thank you!

TIBCO Advanced Analytics Meetup (TAAM) November 2015

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (11)

Similar to TIBCO Advanced Analytics Meetup (TAAM) November 2015

Similar to TIBCO Advanced Analytics Meetup (TAAM) November 2015 (20)

Recently uploaded

Recently uploaded (20)

TIBCO Advanced Analytics Meetup (TAAM) November 2015

Editor's Notes