1.
1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Data & Analytics in Insurance
Can you have one without the other?

2.
2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
P&C Insurance trends in big data/analytics
Use of Predictive Models in P&C New applications, New Methods
• Source: Willis Towers Watson 2016 Predictive Modeling Benchmark Survey (U.S.)
• The survey was fielded from September 7 to October 24, 2016. Respondents comprise 14% of U.S. personal lines carriers
and 20% of commercial lines carriers.

3.
3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
P&C Insurance trends in big data/analytics
 “Big data, notably from vehicle
telematics and the IoT, are opening up
many new potential avenues for
investigation and improvement. These
opportunities apply as much to carriers
that have invested recently in
improved policy administration and
quote systems as it does to others.
Whatever the available level of
hardware and software within a
business, a lack of accompanying
investment in data and analytics is
rather like driving a sports car
without fully revving up the engine.”
Uses of Big Data
• Source: Willis Towers Watson 2016 Predictive Modeling Benchmark Survey (U.S.)
• The survey was fielded from September 7 to October 24, 2016. Respondents comprise 14% of U.S. personal lines carriers
and 20% of commercial lines carriers.

4.
The Liberty Mutual
Insurance Data Lake
One small Hadoop footprint …
One giant leap to understanding
#TechAtLiberty

5.
5Liberty Mutual Insurance
Empower Liberty Mutual to leverage the vast data and
amazing talent that we have
Make analytics as easy as it can be
Allow data to be free and secure
Foster a culture of quick iterative experiments, failing
and learning as fast as possible
Remove the separation between IT and business
Our North Star: What we strive for

6.
6Liberty Mutual Insurance
75th

7.
7Liberty Mutual Insurance
Agenda
• How do we think about analytics?
• How do we work as a team?
• Who/what is a data scientist?
• How does a data lake help us?

8.
8Liberty Mutual Insurance
How we think about analytics and machine learning (ML)
Obtaining the data
from source systems
and devices
Storing the data in a
format and location so
that it can be studied
Studying the data to
gain insight and
business value
GET LAND STUDY
• ML is an extension of STUDY
• ML programs need to access
data that’s in “LAND”

9.
9Liberty Mutual Insurance
Who/what is a data scientist?
How do we work as a team?

10.
10Liberty Mutual Insurance
What makes up a data scientists?
True data scientists are extremely rare
because of the unique combination of
skills required.
We believe in investing in data science
teams made up of energized engineers
with various roles:
• Software developers
• Data engineers
• Data analysts
• Data scientists
You don’t need a PhD to be a data
scientist!!!
Business analyst
Engineer/
Developer
Mathematician
Data
Scientist

11.
11Liberty Mutual Insurance
We heard common frustrations
Analytics is hard!
Tools are too hard to
use; Requires many
types of skills
Security and Analytics
have competing goals
IT/business collaboration
needs to improve

12.
12Liberty Mutual Insurance
Information Technology Business
Source system Data
scientists/analysts
MS SQL
Teradata
DB2
Mysql
MS SQL
Oracle
Mongo
Postgres
DATA
Mart
Information management (IM)
Ent. Data
Warehouse
DATA
Mart
2
EDW
Cognos
Tableau
SAS
OBIEE
Micro
strategy
SharePoint
PowerBI
Sybase

13.
13Liberty Mutual Insurance
PYTHON
R
SAS
H2O
R Shiny
Excel
PowerBI
Source system Data
scientists/analysts
IM evolving into Data analytics
MS SQL
Teradata
DB2
Mysql
Oracle
Mongo
EDW
Sybase
Iterate and learn
Information technology Business
Unstructured Data

14.
14Liberty Mutual Insurance
Text
Analytics
Streaming
Analytics
Predictive
Analytics
Data Engineer
Data Engineer
IT Data
Scientist
Software
Developer
Software
Developer
Form one team with business and IT together
Data Scientist
Data Scientist
Data Scientist

15.
15Liberty Mutual Insurance
How does a data lake help us?

16.
16Liberty Mutual Insurance
HORTONWORKS DATA PLATFORM (HDP®)

17.
17Liberty Mutual Insurance
Enterprise data lake security
Security: Centrify / AD / Kerberos / Ranger/ HDFS Encryption /SSL
Kerberos HDP Data Lake on-Premises
AD Server as KDC
Secured Zone
HDFS
Secured Zone
HDFS
Secured Zone
HDFS
/Legal
| user:grp
| __1
| __2
/HR
| user:grp
| __1
| __2
/Finance
| user:grp
| __1
| __2
Ranger Policies & Plugins
HDFS Permission & ACL
System Admins
Power BI Users
Data Scientists
ETL Developers
Ambari Server
Spark Thrift Server
HDP Edge Node
Kerberos
Kerberos
NAS/Local HDD
SSL
ODBC
SSL
SSL
RMDBS on-Premises
Sqoop
Security Options Available:
1. Kerberos
2. SSL Enablein Connection String
3. Encryption=true on database
Zeppelin  Livy Server
Layers of Defense
Perimeter Level Security: Apache Knox for REST API
Authentication : Kerberos
Authorization: Ranger
OS Security : HDFS Permission, encryption on HDFS
ApacheKnox

18.
18Liberty Mutual Insurance
Security challenges and alternatives
• Security implementation requires existing tools reconfiguration
• Need to use the combined security mechanisms
• Testing is painful and something doesn’t work
• Not all BI Tools Build-in Drivers Support Kerberos
• Spark Security
⎻ Kerberos for Authentication
⎻ AD Groups for HDFS ACLs
⎻ SparkSQL, Ranger, and LLAP via Spark Thrift Server for Authorization

19.
19Liberty Mutual Insurance
Data lake BI & analytics example
User’s Desktop / Laptop /VDEApplications & Databases
PowerBI Desktop
Dashboard
(data embedded)
Sources of Cost
Information
PowerBI
Hive/Data
Transformation
Kerberos / ODBC
S3: csv Files
Centrify / AD / Kerberos/ Ranger/Encryption
Publish
Text Files / API
License
Counts from
Office 365
Daily
HDP Cluster
PullData from Hadoop
Report& Data
AWS Keys
Upload Data
PowerBI Services
DataAutomation
PowerBIGateway
Report Developers
Report Consumers
ETL Developers
Other Data Sources
on-premises
Sqoop
Data Lake on-Premises
AD Server
Rest API

20.
20Liberty Mutual Insurance
Integrate Elasticsearch and Spark in data lake
Enterprise Data Lake
Master & Data Nodes
HDP Edge/ES Node 1 HDP Edge/ES Node 2 HDP Edge/ES Node 3
ES Repo
/experian
| index
| __1
| __2
ES Repo
/experian
| index
| __3
| __4
ES Repo
/experian
| index
| __5
| __6
ElasticSearch
Hadoop Plugin
ElasticSearch
Hadoop Plugin
ElasticSearch
Hadoop Plugin
REST API – Elasticsearch Queries
End Users
NAS
spark-submit --master yarn --num-executors 4 --executor-
memory 1G --executor-cores 1 esspark-assembly-1.0.jar
hdfs:///data/BRICK_2016_Q3_masked.csv
curl -XPOST "http://localhost:9200/gs/_search" -
d'{"query": {"match" :{ "CITY": {"query": "Yiqing",
"fuzziness": "AUTO"}}}}'
Data Volume: 1 data brick
100GB csv file
Fuzzy Match: company name,
street address, city, state
Results: match score and all
500+ attributes
IT Developer

21.
21Liberty Mutual Insurance
Integrate Elasticsearch and Spark in data lake (cont.)

22.
22Liberty Mutual Insurance
Data archiving example
Apache Flume
Syslog Server 1
Syslog Server 3
Syslog Server 2
Apache Flume
Apache Flume
VirtualIndex
Enterprise Data Lake (5 data nodes total 120TB)
Analytics, trend
Hot Data Storage
OneMonth
 1TB uncompressed Data
 100GB Compressed Data
SharePoint Logs
HDP Edge Node
SharePoint Logs
IT Developers
Data Analytists
Kafka
Warn Data Storage
OneYear
 60TB uncompressed Data
 6TB Compressed Data
SharePoint Logs
Kafka Kafka
SIEM, Alerts, Real Time Monitoring
Kerberos
NifiMergeContent: Holds data
until the flow file reaches a
suitable sizeto be loaded to HDFS
Logs

23.
23Liberty Mutual Insurance
Sample DataFlow

24.
24Liberty Mutual Insurance
Conclusion
Just get started!
Don’t be afraid to fail!
Invite your “business” partners into the process
A small lake is still very beneficial!

25.
25Liberty Mutual Insurance
Thank you