IoT hardware is becoming easier to prototype using tools like Onion Omega. Companies are using sensors in retail stores to improve customer experiences. Cloud adoption of Hadoop analytics is growing rapidly at 84% annually, led by offerings from Amazon, Microsoft, IBM, and Salesforce. Real-time analytics options are expanding with services from Google and Microsoft that can process streaming data faster than competitors. Many companies are considering moving legacy ETL pipelines to the cloud to reduce time spent cleaning raw data and prepare it for analytics. Major tech companies like IBM and Amazon are heavily investing in improving Apache Spark for large-scale data processing.

1. Snapshot of trends reshaping Big Data Analytics & IoT
H/W is back (IoT) – Onion Omegs makes IoT h/w prototyping as easy as creating & installing s/w apps.
Industry devices: Scanalytics – retail floor sensor analytics promotes in-store conversions, customer retention.
Cloud Adoption in Analytics : Global Hadoop as a Service (HDaaS) market to grow at 84.81% CAGR through
till 2019 + Existing offerings from Salesforce, Microsoft, IBM, Amazon, Accenture, CSC, DataBricks.
Real Time Analytics Options: Google Cloud Dataflow: pipelines to ingest, transform & analyze data – works
w/ Spark. MSFT Trill: streaming analytics engine processes data @ 2-4 orders of magnitude over competitors.
BI-aaS & Data Prep: 33% BI folks spend 50-90% of Time “Cleaning” Raw Data for Analytics; 97% said ETL
critical - 51% polled use on premise vs 49% cloud. On prem. ETL folks: 51% “strongly considering” cloud
Spark: IBM to invest a few Hundred Million dollars/year in Apache Spark - putting 3,500+ researchers to
improve the engine, fix bugs, test it in production & submit new features. Amazon EMR follows suit.
Pricing Machine Learning: AWS-ML priced @ $.42/hr for model builds, $0.1/1000 batch & $0.0001 per real time
prediction. MSFT’s Cortana Analytics: subscription bundle includes ML, Big data stores, PowerBI all-in-one.
Insights Products: First Data xforms payment data to insights on sales & customers. Thomson Reuters quants
create models for institutional investors to predict security prices & influencing events w/ non traditional data
Analytics App Re-use: Teradata App Center - analytics app template s - 29 frameworks in 6 industries: retail,
telco, healthcare, hospitality. Incl. pre-built logic, visualizations + config. Templates do 80% of finished app.
Arindam Banerji (banerji.arindam@gmail.com)