Hyper-Converged Infrastructure: Big Data and IoT opportunities and challenges, April 2019_6

1Center for Research in
Intelligent Storage Andrei.Khurshudov@gmail.com
HCI BIG DATA/IOT OPPORTUNITIES
AND CHALLENGES
Andrei Khurshudov, PhD

• The Fourth Industrial Revolution refers to new
ways in which technology becomes embedded
within society and even the human body
• The Fourth Industrial Revolution is marked by
emerging technology breakthroughs in such
fields, as
– Robotics
– Nanotechnology
– Biotechnology
– Quantum computing
– 3D printing
– Artificial intelligence
– Big Data Analytics
– The Internet of Things (IoT)
– Augmented/Virtual reality (AR/VR)
Industrial Revolutions: IoT, AI, and Big Data
PRODUCTIVITY WILL BE INCREASING. STANDARDS
OF LIVING WILL BE IMPROVING
Reference

• Lots of data, data
storage and compute
• Fast, sophisticated data
processing, decision
making and control
algorithms
• Global network of
connected sensors,
devices, and machines
• Full integration of all of
the above
The Future:
Essential Building Blocks
Big Data

4
Data Generation. “New Data” is Different
Changes:
• Mostly unstructured data (80%+ of all the data)
• No-SQL vs. SQL
• Key-value pairs vs. relational data structures
• Distributed storage and computing
• Storage and computing on commodity hardware
• Lots of open-source solutions
• Object storage vs. block storage
• New analytics technologies (Hadoop/MR, 2005)
• New algorithms (e.g., Random Forest, 2001)
• New visualization techniques (multi-D)
• Cloud-based analytics vs. local analytics
• HCI instead of traditional IT architecture
• Lots of machine-generated data à
UNSTRUCTURED DATA IS FUNDAMENTALLY CHANGING THE INTERNET TECHNOLOGY (IT) AND
REQUIRES DIFFERENT STORAGE, COMPUTE, AND ANALYTICS
Users of the Internet generate 2.5 Exabytes
of data each day (2.5E18 Bytes)*
* Source

5
Data Generation: Machine Data
THERE'S PLENTY DATAAT THE BOTTOM
180 HDDs/year
(10 TB drives, No redundancy)
Process and store…
250,000,000 HDDs/year
(10 TB drives, No redundancy)
Process at the Edge, partially
store, and delete most of it…
Nearly 90% of
data collected in
the industry is
never acted upon
(per IBM)

6
Future Market Impact: The Data Gravity Concept
• “Data gravity” is an analogy for the nature of data and its
ability to attract additional applications and services. The
Law of Gravity states that the attraction between objects is
directly proportional to their mass.
• Data gravity describes the phenomenon in which the
number and speed at which services, applications, and
even customers are attracted to data increases as the mass
of the data increases.
• This also means that as data size (mass) increases, the
data becomes harder and harder to move for financial and
technical reasons (inertia).
• For example, selecting one specific cloud-based data
storage and analytics platform (such as AWS or Azure) and
using it for a while makes it expensive and difficult to
move to another similar platform in the future.
+ cost
Source
THINK CAREFULLY HOW AND WHERE YOU
STORE YOUR DATA TODAY!

storage and compute
making and control
algorithms
connected sensors,
the above
The Future:
Big Data Analytics
Storage and Compute Infrastructure
AI/ML/DL

8
Big Data Analytics (BDA) vs. Traditional Analytics
Structured, well-organized data
with low dimensionality
Traditional Statistical Data Analysis and Visualization Techniques
Lot’s of high-dimensionality,
incomplete, and noisy data
• High data dimensionality
• No-SQL databases, key-value pairs
• Machine learning/AI
• Results verification on a ”test group”
• Distributed storage and computing
• Graph databases and analytics
• Hadoop, Spark, Storm
• R, Python, Java, Scala, D3
• Real-time streaming & batch processing
• Novel visualization techniques & tools
Big Data Analytics New Ways to Analyze and Visualize Results
What makes BDA possible today?
- Abundance of analyzable data
- Abundance of cheap computing
power
andrei@alchemyiot.com

9
Availability of Compute
1999: First Google Rack
2016: Council Bluffs Google Data Center, Iowa, USA
Google today: 40+ data centers
Hadoop is the leading analytics platform that keeps
evolving with time, and growing at a CAGR of 65%
STORAGE AND ANALYTICS PLATFORMS HAVE EVOLVED DRAMATICALLY, MAKING FAST, COMPLEX,
PARALLELIZED ANALYTICS GENERALLY AVAILABLE

10
The Future of Analytics: Artificial Intelligence (AI) vs. Machine
Learning (ML) vs. Deep Learning and Cognitive Analytics
Cognitive Analytics refers to a class
of automated, autonomous, self-
learning algorithms capable of
data collection, analysis,
interpretation, pattern discovery,
and event forecasting that evolves
over time and mimics the way a
human would collect, analyze, and
interpret data, as well as discover
patterns and forecast future events
of interest
Cognitive Analytics
1956
RELATED FIELDS, SAME GENERAL GOAL: TO MAKE THE COMPUTER “THINK” MORE LIKE A HUMAN

11
Analytics/AI/ML Trends
• Automated (Augmented) Analytics and Data Management
• Analytics migration to the Cloud unless the decision-making latency is
critically important or connectivity is poor (at the Edge)
• Shift from batch analytics à streaming, real-time analytics
• Continuous BI (periodic BI à continuous BI)
• In-memory computing (DB, Data Grids, Analytics, OS, etc.)
• AI-driven product development
• Data privacy, security, governance and more regulations
• Explainable AI (vs. Black-Box AI)
• Data Fabric (vs. Data Lakes)

storage and compute
making and control
algorithms
connected sensors,
the above
The Future:
The Internet of Things

13
The Internet of Things (IoT)
We are giving our world a digital
nervous system. Location data
using GPS sensors. Eyes and ears
using cameras and microphones,
along with sensory organs that can
measure everything from
temperature to pressure changes.
These inputs are digitized and
placed onto networks that range
from personal area networks like
Bluetooth to large area networks
like Cellular and Satellite
These networked inputs can then
be combined into bi-directional
systems that integrate and store
data in the Cloud, deploy
complex rules and Analytics,
and enable people,processes
and systems with better
decision making
SENSORS +
and enable people, processes
decision making
CONNECTIVITY +
and enable people, processes
decision making
ACTIONABLE INSIGHTS AND ACTIONS

14
The Internet of Things: An Explosive Growth
ALL OF THESE DEVICES WILL GENERATE AND CONSUME A LOT OF DATA
Smart homes and appliances
Smart factories and machines
Smart wearables
Phones
and
tablets
Smart cars
IoT is born

IoT Solutions Architecture
ALL STAGES ARE REQUIRED FOR A COMPLETE IoT SOLUTION
15
Source: HPE
Measure Collect & Transmit Process, Decide, Store

16
Powerful Edge/Mesh Analytics and Efficient Data
Storage for IoT: Great Opportunity for HCI
HCI Offers Plenty of Benefits at the Edge:
• Optimized Cost of Ownership
• Simplified management
• Simple scalability and upgrades
• Workload agility and diversity of applications
• Availability, Reliability
HYPER-CONVERGENCE WILL EMPOWER END-USERS TO RUN REAL-TIME ANALYTICS AND MAKE
DECISION AT THE INTELLIGENT DYNAMIC MESH USING MULTIPLE SERVICES, SYSTEMS, AND
APPLICATIONS

storage and compute
making and control
algorithms
connected sensors,
the above
The Future:

18
IoT Data/ HCI/Data Analytics/AI/ML/AR: Current INTEGRATION
+ remote assist and step-by-step guidance
Alert!EDGE HCI
• 25-45% PRODUCTIVITY IMPROVEMENTS
• ~50% IMPROVEMENT IN QUALITY OF REPAIRS
Consult with a remote expert!
Receive and view step-by-step
repair instructions
Repair
Report
Visualize, review, and diagnose
Analytics
Modeling
Dispatch a Tech

IoT Data/ HCI/Data Analytics/AR INTEGRATION
• 2 devices generating IoT data
• 1 AR headset from Realwear
• Device data flows to AWS and Azure
• Cloud-based health status decision-
making
• Data and device health status are
accessed by the headset over API
• Data and health status are displayed in
the headset when a particular device is
recognized
• Then, a voice-controlled, hands-free
diagnostic and repair workflow is started See full demo at AWE conference
in Santa Clara, CA on MAY 29 - 31
AWS Azure
(Fusion AR IoT product from Jujotech)

IoT Data/Analytics/Decision Integration inside AR
Start step-by-step
Repair Process
Jujotech Jujotech
Diagnostics: Step 1

21
Trends and Opportunities
• The number of IoT devices, the amount of generated data, and new IoT user
experiences will growth rapidly – greater need for storage capacity and compute speed
• Edge solutions are expected to take up to 40% of the future IoT market – considering
the total size of the future IoT market - great opportunity for HCI solutions at the Edge
• There will be a shift from the intelligent edge to the intelligent mesh (a wide range of
devices, people, and services connected) – a great place for HCI to handle lots of data
and lots of diverse applications, including AI/ML/Analytics
• Data brokering will become popular – it might be worth collecting and storing more
data if it can be monetized
• In-memory computing (DB, Data Grids, Analytics, OS, etc.) will grow – an opportunity
for in-memory HCI products
• AR / VR will be used for data visualization, documentation access, and hands-free
operations (as the cost of AR/VR devices declines rapidly) – even more data storage
and compute will be needed to support this

22
General Challenges
• Security risks will remain high
• Implementation costs of new technologies might be higher than
expected
• Customer ROI benefits must be made clear
• Complex solutions could be difficult to understand (and to sell)
• Trusting in black-box solutions (for AI/ML as well as for HCI) might
be a challenge
• Too many choices might slow technology adoption

The Future
In the near future, IoT will
adopt fully-automated,
autonomous, self-learning Big
Data Analytics to enhance
itself
relying heavily on HCI
solutions
to become one giant intelligent
network of devices, sensors,
data, and decision-making
algorithms to run this world on
our behalf

24
Backup

Size of the IoT / IIoT (Industrial IoT) Market
(from multiple sources)
McKinsey estimates the total IoT
market size in 2015 was up to $900M,
growing to $3.7B in 2020 attaining a
32.6% CAGR
IHS forecasts that the IoT market will
grow from an installed base of 15.4
billion devices in 2015 to 30.7 billion
devices in 2020 and 75.4 billion in 2025
Bain predicts that by 2020 annual
revenues could exceed $470B for the IoT
vendors selling the hardware, software
and comprehensive solutions
Accenture estimates the Industrial
Internet of Things (IIoT) could add
$14.2T to the global economy by 2030
IoT MARKET WILL GROW FAST IN THE NEAR FUTURE
25

26
Industrial IoT: Cloud Analytics and Services
SaaS– Software as a Service
Cloud IoT Aggregator &
Analytics
Desired characteristics:
• SaaS offering (subscription-based)
• Data encryption at rest and in-flight
• Powerful multi-cloud data collection,
aggregation, and data governance
• Seamless interface with legacy
systems (Historians, etc.)
• Automated analytics
• Actionable insight that results in
increased uptime and revenues
• Root-cause analysis combined with
in-field documentation and step-by-
step repair guides decrease
equipment downtime
• Just-in-time issue notification and
reaction
Cloud 1
Cloud 2 Cloud 3
Cloud 4

Industrial IoT: Edge Analytics and Services
IoT device
Operator
Next to the device
Device telemetry
Diagnostic
Commands
Cloud Analytics
Device telemetry
Analysis
Instructions
Big-picture view
Recommendations
27
Desired Characteristics:
• Low-latency analytics, diagnostics,
and decision-making
• Automated actions that can be pre-
programmed
• Alerts issued to the Operator
• Cloud connectivity for a “big-picture
view” of the entire fleet
• Device might look normal to the
Operator or local analytics but be
anomalous at the entire fleet scale
• Most data is stored locally and
only a small subset is moved to
the Cloud

Hyper-Converged Infrastructure: Big Data and IoT opportunities and challenges, April 2019_6

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Hyper-Converged Infrastructure: Big Data and IoT opportunities and challenges, April 2019_6

Similar to Hyper-Converged Infrastructure: Big Data and IoT opportunities and challenges, April 2019_6 (20)

More from Andrei Khurshudov

More from Andrei Khurshudov (8)

Recently uploaded

Recently uploaded (20)

Hyper-Converged Infrastructure: Big Data and IoT opportunities and challenges, April 2019_6