This document discusses predictive maintenance solutions using EMC and Virtustream technologies. It describes how raw sensor data is acquired from sources like railway sensors and processed. This involves data preparation, model training and testing. The data is then stored in various types and locations depending on how often it needs to be accessed. This includes hot/real-time data stored in SAP HANA, warm historical data stored in SAP IQ, and cold archived data stored in Hadoop. Combining SAP, EMC storage, and Virtustream cloud services can provide solutions for predictive analytics use cases involving large volumes of both structured and unstructured data.

1. © Copyright 2015 EMC Corporation. All rights reserved. 1© Copyright 2015 EMC Corporation. All rights reserved.
PREDICTIVE MAINTENANCE
EMC/VIRTUSTREAM SOLUTION
RICCARDO ROMANI

2. © Copyright 2015 EMC Corporation. All rights reserved.
PREDICTIVE MAINTENANCE PROCESS
1. Data acquisition and processing:
A. Data acquisition from raw data
sources ( i.e. railways sensors )
B. Long times for data preparation
and transformation into data set
be used as structured data model
and historical data base
C. Long times for model training and
testing iterations
D. Some batch calculations required
1. Data Storage : Huge amount of data
generated at both acquisition as well
as historical stages
MOST COMMON ISSUES

3. © Copyright 2015 EMC Corporation. All rights reserved.
DIFFERENT TYPES OF DATA IN PREDICTIVE ANALYTICS
FROM RAW DATA SOURCE…
• Example of Data Sets created starting from the raw data
• Structured Data
• Training data: It is the engine run-to-failure data.
• Testing data: It is the engine operating data without failure events
recorded.
• Ground truth data: It contains the information of true remaining cycles
for each engine in the testing data.
• Predictive data used to predict when an in-service machine will fail, so
that maintenance can be planned in advance.
• Responds to the question :” Given these aircraft engine operation and
failure events history, can we predict when an in-service engine will fail?”
• Regression: Predict the Remaining Useful Life (RUL), or Time to Failure
(TTF).
• Example of raw data gathered from a Stream Processing System.
• Millions of txt files
• Unstructured/semi-structured data
…TO DATASET

4. © Copyright 2015 EMC Corporation. All rights reserved.
SAP HANA Sybase IQ Sybase IQ
Hot Data Warm Data Cold Data
• Analytics run on Data Model
− Modern in-memory
platform
− Transact/analyze in real-
time
− Native predictive, text,
and spatial algorithms
• Warm Historical Data Model
− Disk backed, smart column store (
HANA on disk or IQ )
− It helps in offloading HANA from
huge amount of data that are
dinamically stored on disk instead
of in-memory
− Excels at queries on structured
data from terabyte to petabyte
scale
• Cold Historical Data Model
− Less frequently accessed data is
archived in time partitions on IQ
− The data is static and used
primarily for read access
− data resides in cost- efficient
storage with fewer backups to
reduce operational costs
− Lower SLA requirements
HOW COMBINING SAP AND EMC/VIRTUSTREAM CAN HELP
HADOOP
Raw data
• Data acquired from Sensors
− New type of User Defined
Function for data federation
− Direct access to HDFS without
need for the package, mapper,
and reducer specification
− Invoke custom Map Reduce jobs
Solution: EMC with Isilon,
Hadoop-native storage
combined with Virtustream
Cloud Storage (EMC ECS Object
Storage )
DIFFERENT TYPES OF DATA IN PREDICTIVE ANALYTICS
Solution : Virtustream with SAP HANA Cloud IaaS/PaaS

5. © Copyright 2015 EMC Corporation. All rights reserved.
SCENARIO #1
SAP HANA, SAP IQ @ VIRTUSTREAM , DATA ACQUISITION ON ISILON @ HYPERCED
HISTORIC/COLD
Gateway
Rolling Stock
Raw data
RAW DATA
1. Raw Data acquisition done at gateway
level– serialization could be a
bottleneck as well as raw data history
growth
2. Predictive model computation done
at Hot Data level, in Hana
3. Warm data on SAP IQ
4. Cold data archived ( could be SAP IQ
) for compliance and outlier data from
faulty sensors and for statistical
purposes
• EMC can control rolling stock data
growth during acquisition with Isilon
storage certified by SAP and sitting at
gateway layer
HISTORIC/
WARM
HOT DATA
Datacenter
Almaviva “Hyperced” Datacenter

6. © Copyright 2015 EMC Corporation. All rights reserved.
SCENARIO #2
ADDING HADOOP COMBINED WITH ISILON @ HYPERCED
1. Raw Data acquistion done in
parallel streams. Hadoop
speed up parsing, and
creation of reduced dataset to
be loaded and historicized
2. Predictive model computation
can be done accessing a wider
dataset accessing Warm data
on IQ and Hadoop
3. Cold data stored on hadoop
filesystem as “intelligent”
archiving for statistical
analysis and retrieval
• EMC can control and improve
Data growth management at
both Hadoop and Storage level
− Hadoop for unlimited capacity for raw data
processing
− Apache Kafka or Storm for distributed stream
processing
Gateway/Data Lake
Rolling Stock
Raw data
INTELLIGENT
ARCHIVING FOR
RAW DATA AND
HSISTORIC
HISTORIC/
WARM
HOT DATA
Almaviva
“Hyperced”
Datacenter

7. © Copyright 2015 EMC Corporation. All rights reserved.
HADOOP AND SAP 1/3
HADOOP IN ACTION @ SAP PREDICTIVE MAINTENANCE BLUEPRINTS
Where SAP and EMC combined technologies fits in a predictive
maintenance project

8. © Copyright 2015 EMC Corporation. All rights reserved.
HADOOP AND SAP 2/3
Data acquisition
• Real time events streams coming in at a rate of
thousands of raw events per seconds.
• The stream processing system should be able to
process those events in a fault tolerant and distributed
manner and with parallel processing
• Streaming processing systems should also keep record
of old data for some reasonable amount of time before
they are archived or destroyed in order to :
A. build pattern recognition and statistical model
building methods
B. Cope with local country laws.
C. to identify any potential outliers in the
streamed-in data from the sensors. While
monitoring for the faults in the assets it is
possible that the sensor that is taking the
readings, being a machine itself, could fail and
start sending faulty records. Intelligent
CBM management systems capable of detecting
such outliers will try to isolate these faulty
sensors and notify
We have evaluated two popular open source technologies; Apache Kafka , which is the
distributed messaging system, and Storm which is a distributed stream processing engine.
Both having HADOOP as repository for managing data growth.
SAP provides a product called “ESP – Event Stream Processor” that integrates HANA ad
Hadoop
HADOOP IN ACTION @ SAP PREDICTIVE MAINTENANCE BLUEPRINTS

9. © Copyright 2015 EMC Corporation. All rights reserved.
Build, train and validate the model – creating Data Sets
• To detect failure in a given stream of sensor data, we need to first
define normal behavior.
• For this we need to build model around the historical sensor data.
• Predictive models analyze current and historical data on individuals
to produce metrics.
• A model is reusable and is created by training an algorithm using
historical data and saving the model for reuse purpose to share the
common business rules which can be applied to similar data, in
order to analyze results without the historical data, by using the
trained algorithm
• The process involve running one or more algorithms on the data set
where prediction is going to be carried out. This is an iterative
processing and often involves training the model, using multiple
models on the same data set and finally arriving on the best fit
model based on the business data understanding.
• Raw data, once prepared, are organized in Data Sets, needed to be
stored to be used by different type of models and algorithms
HADOOP AND SAP 3/3
HADOOP IN ACTION @ SAP PREDICTIVE MAINTENANCE BLUEPRINTS

10. © Copyright 2015 EMC Corporation. All rights reserved.
VIRTUSTREAM
SAP HANA
DT
ALMAVIVA
HYPERCED
Servers & Network
DATA STORAGE powered by EMC
HDFS or NFS
EMC ISILON , VNX
FibreChannel or NFS
EMC VMAX, VNX or XtremIO
EMC/VIRTUSTREAM HYBRID ARCHITECTURE
HOT
Extended
Storage
IQ
NLS
Backup / Archive
Best in class Cloud IaaS/PaaS
for SAP workloads , through
Almaviva White Label Option
EMC Scale Out Unstructured
Storage with de-duplication
features and native HDFS
support
From 15% to 35 % of data
storage efficiency depending
on data specific nature GATEWAY
Unstructured Storage
Raw Data

11. © Copyright 2015 EMC Corporation. All rights reserved.© Copyright 2015 EMC Corporation. All rights reserved.

12. © Copyright 2015 EMC Corporation. All rights reserved.
1. Acquire OT data in parallel
streams
2. Clean data and import only the
validated data into SAP HANA
3. Historic OT data in IQ and/or
Hadoop
4. Use Hive to build a
relational/historical view of the
OT data also at Hadoop level
5. Unify OT view with IT data, if
needed
6. Save HANA resources for
realtime predictions only
7. De-duplicate and store raw OT
data for future use or statistic
purposes
Gateway/Data Lake
HADOOP AND SAP
PDMS BLUEPRINT

13. © Copyright 2015 EMC Corporation. All rights reserved.
HADOOP AND SAP
GATHERING REAL-TIME EVENTS AT SCALE