Interactive business intelligence visualizations with R Shiny and beyond with scalable big data architectures. Going beyond MS Excel and other non-scalable proprietary solutions.

1. Dashboards for
Business Intelligence
Interactive visualizations
with R Shiny and beyond with
scalable big data
architectures
Petteri Teikari, PhD
Singapore Eye Research Institute (SERI)
Visual Neurosciences group
http://petteri-teikari.com/
Version “Tue 31 July 2018“
Sankey diagram; income and spending

2. R
Competition with Python as the top dog (Excel / SAS / Tableau, forget about it :p)
https://qz.com/1063071/the-great-r-versus-python-for-data-science-debate/
https://www.kdnuggets.com/2015/05/r-vs-python-data-science.html
https://www.datasciencecentral.
com/profiles/blogs/r-python-or-
sas-which-one-should-you-learn-
first

3. R or Python
No need necessarily to choose one over other. Use them together
https://www.r-bloggers.com/r-or-python-python-or-r
-the-ongoing-debate/
reticulatepackage,acomprehensivesetoftoolsforinteroperabilitybetweenPythonandR.
TranslationbetweenRandPythonobjects(forexample,betweenRandPandasdataframes,or
betweenRmatricesandNumPyarrays).
https://blog.rstudio.com/2018/03/26/reticulate-r-interface-to-python/
http://www.sanaitics.com/UploadedFiles/html_files/7601AL-Working_on_R_with_Python.html
rpy2 is a Python module which offers an interface to run embedded R in a Python
process. rpy2 module provides two interfaces: a low-level interface (rpy2.rinterface) and a high-
levelinterface(rpy2.robjects).Wewillusehigh-levelinterface, rpy2.robjects.

4. Example of Interoperability
Code most of the stuff in Python, and use statistics methods from R
https://medium.com/bigdatarepublic/contextual-changepoint-dete
ction-with-python-and-r-using-rpy2-fa7d86259ba9
We’ll talk about the last option and show an example of how you can
combine Python and R to perform contextual changepoint detection. rpy2
is a Python package that provides access to R from Python. It provides the
capability to convert Python objects into R objects and vice versa. Thus,
with the help of rpy2, you can integrate R’s functionality into your Python
workflow.
https://longhowlam.wordpress.com/2017/04/10/test-driving-python-integration-in
-r-using-the-reticulate-package/
Python dominates the deep learningrepositories and
luckily you can use them in your R project via
Reticulate
Clarifai provides a set of computer vision API’s for image
recognition,facedetection,extractingtags,etc.
pytorch is a python package that provides tensor computations and deep
neural networks. There is no ‘R torch’ equivalent, but we can use reticulate in
R.
The pattern.nlmodule contains a fast part-of-speech tagger for Dutch,
sentiment analysis, and tools for Dutch verb conjugation and noun
singularization & pluralization. At the moment it does not support python 3.
That is not a big deal, I am using Anaconda and created a Python 2.7
environment to install pattern.nl. The nice thing of the reticulate package is
thatitallowsyoutochooseaspecificPythonenvironmenttobeused.
Steven Reitsma
LonghowLam

5. Exploratory Data Analysis
Easy with R | Easy to switch from Excel if you are not interested in coding too much
Recently, I came across this package DataExplorer that seems to be doing
the entire EDA (at least, the typical basic EDA) with just one
function create_report() that generates a nice presentable rendered
Rmarkdown html document. That’s just a report automatically generated
andwhatifyouwantthecontrolofwhatyouwouldliketoperformEDAon.
https://towardsdatascience.com/simple-fast-exploratory-data-analysis-i
n-r-with-dataexplorer-package-e055348d9619
EDA isnotaformalprocesswithastrictsetof rules.Morethananything, EDA is astateofmind.
DuringtheinitialphasesofEDA youshould feelfreetoinvestigateeveryideathatoccurstoyou.
Someoftheseideaswillpan out, and somewillbedead ends.As yourexploration continues, you
willhomein on afewparticularlyproductiveareasthatyou’lleventuallywriteupandcommunicate
toothers.
http://r4ds.had.co.nz/exploratory-data-analysis.html

6. R Develop in Practice
RStudio the most popular IDE, easy to get started
https://www.slideshare.net/KIRENZ_CONSULTING/introduction-to-r-66118918
Most popular visualization library is ggplot2
https://github.com/rstudio/cheatsheets/blob/master/data-visualiza
tion-2.1.pdf
http://r-statistics.co/Top50-Ggplot2-Visualizations-Mast
erList-R-Code.html

7. Transition from Excel?
Get rid of the “fax machine” a.k.a. Excel which scales badly to bigger problems
https://www.jessesadler.com/post/excel-vs-r/
byJesseSadler
https://www.amazon.com/Excel-Users-Introduction-Analysts-ebook/dp/B01K3HFOZU
There is no doubt that the learning curve for R is much steeper than
producing one or two charts in a spreadsheet. However, there are real long-
term advantages to learning a dedicated data analysis tool like R. Such
advice to learn a programming language can seem both daunting and
vague, especially ifyoudonotreallyunderstandwhatitmeanstocode. For
this reason, after discussing why it is preferable to analyze data with R
instead of a spreadsheet program, this post provides a brief
introduction to R, as well as an example of analysis and visualization of
historicaldatawithR.

8. R Shiny
Make interactive browser-based apps quickly from your R code
https://shiny.rstudio.com/gallery/
https://www.showmeshiny.com/
https://shiny.rstudio.com/gallery/retirement-simulation.html
In other words, make interactive reportsfor
your colleagues, boss, clients,etc.
Quicklydo sensitivity analysis with adjustable
sliders.
See also:
IntrotoShinyAppswithRStudio'sJoe Cheng

9. R Shiny Dashboard
Present everything on an interactive web app instead of PDF reports or Excel crap
https://github.com/rstudio/shinydashboard
https://nycdatascience.com/blog/student-works/project-2-shiny-dashboard-app-data-scientist-salary-comparator/
https://rstudio.github.io/shinydashboard/get_started.html
One of the beautiful gifts that R has (that Python missed,until dash) is Shiny
. Shiny is an R package that makes it easy to build interactive web apps
straight from R. Dashboards are popular since they are good in helping
businesses make insights out of the existing data. In this post, we will see
how to leverage Shiny to build a simple sales revenue dashboard.
https://medium.freecodecamp.org/build-your-first-web-app-dashboard-u
sing-shiny-and-r-ec433c9f3f6c

10. Microservice Architecture
Language-agnostic approach if all your favorite algorithms are implemented in different
languages
https://www.youtube.com/watch?v=0FT8EB9gQoA
BusinessIntelligence(atscale)in
MicroservicearchitecturebyDebarshi Basak
https://berlinbuzzwords.de/16/session/busine
ss-intelligence-scale-microservice-architect
ure
ExperiencesinUsingR andPython inProduction
MarkusOjala |May12,20167:55:48AM
PhD,ChiefDataScientistatSmartly.io
https://www.slideshare.net/PetteriTeikariPhD/deploying-deep-learning-models-with-docker-and-kubernetes

11. Think in Processes
That have inputs and output, with something intelligent happening between
INPUTIngestdatato warehouse
OUTPUT data readforanalysis https://www.youtube.com/watch?v=-1w-6uEfV6FW
Y
Justreplace thevisualization,e.g.with D3.js

12. D3.js instead of R Shiny
Keep the model (backend) the same but use non-R frontend for fancier visualization
https://www.quora.com/Which-is-the-best-dashboard-framework-which-is-an-open-source
https://www.packtpub.com/mapt/book/big_data_and_business_intelli
gence/9781785885433/4
https://github.com/topics/business-intelligence?l=javascript
https://d3js.org/
Likevisualizationand creativecoding?TryinteractiveJavaScriptnotebooksin Observable!

13. Time-series Anomaly Detection
For example your best time-series decomposition algorithm might not be written on the
language that you used for the rest of your code
https://github.com/rob-med/awesome-TS-anomaly-detection
http://www.business-science.io/code-tools/2018/04/08/introducing-anomalize.html
Twitter hasmadeanopensourceanomalydetectionpackageinR.Itsgoalistodetect
anomaliesin seasonaltimeseries,aswellasunderlyingtrends.
Findthe AnomalySourceCodeonGitHub

14. Automate the Analytics
Let’s say you are acquiring continously data (IoT, sales database in e-commerce, etc.)
https://www.oreilly.com/ideas/apache-kafka-and-the-four-challenges-of-production-machine-learning-systems
Thisintheend,the
easy partifyouget
goodquality data

15. Automate the Analytics
Another view of the same idea
https://www.infoq.com/presentations/big-data-agile-analytics
KenCollieristheauthor of"AgileAnalytics:A
Value-DrivenApproachtoBusinessIntelligence
andDataWarehousing"andisafrequent
speaker atconferencessuchasTDWIWorld
Conference2010&2011andHEDW2011&
2012.Kenpioneeredtheadaptationofagile
techniquestocomplexdataanalyticssystem
development.Hiscurrentfocusat
ThoughtWorksisonadvancedanalyticsinbig
dataecosystems.

16. Managerial & Culture Problem
Legacy systems and legacy mindsets keeping workflows at stoneage level in practice
https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/three-keys-to-building
-a-data-driven-strategy
Itis sure good tohave team/ department/ company
-level strategies, but itdoesnotreally getyou far
● Ifthe managementjustthrowsoutbuzzwords
withoutknowing whattheymean
- In other words there is no CTO
● ifyoudonothave people whoknow how to
execute atlow-level
- You do not want people entering the
data to random Excel data sheets that
are a nightmare to analyze

17. How to get to Agile Data Science
Legacy systems and legacy mindsets keeping workflows at stoneage level in practice
http://shop.oreilly.com/product/0636920051619.do
Agiledata sciencewithscala. AndyPetrella,CEO&Founderat Kensu.
https://www.slideshare.net/noootsab/agile-data-science-with-scala

18. Case: Internet of Things
Microservices Architecture for the Internet of Things (MSA-IoT)
https://internetofthingsagenda.techtarget.com/blog/IoT-Agenda/Five-things-to-know
-about-the-future-of-microservices-and-IoT
Cyber-physicalmicroservices:AnIoT-basedframeworkformanufacturingsystems
KleanthisThramboulidis- 2018 - Cited by 1 - Related articles
BuildingIoTApplicationsUsingMicroservicesandAPIs–
RealWorldExamples
bySachin Gadre
https://youtu.be/Uga8fCXxnvohttps://publications.opengroup.org/g187