Presentation of Hadoop implementation at LifeWay. Starting point and some future possibilities.

1. Jim Forrester
Director, Information Services
jim.forrester@lifeway.com
@jrforrester
HADOOP @ LIFEWAY

2. ABOUT LIFEWAY
• Founded in 1891
• 4,200+ employees
• One of the world’s largest providers of Christian
products

3. ABOUT LIFEWAY (CONT.)
• Operate more than 185 LifeWay Christian Stores
across the United States
• Trade publishing through B&H – direct publishing
and events through LifeWay Resources

4. OUR ISSUES
• Long ETL process in EDW
• Disjointed data warehouse
LifeWay.com
Database Server
WSPRODDBProduct Attribute Repository
JDA Test
Server
JOPPA
JDA Prod
Server
GAZA
Enterprise Data Warehouse
UID / RCP
Tomcat Server
KETESH
ScanUS
App Server
SAMSON
App Server
SOCO
Business Objects
SAS
UNICA
Epiphany
B&H Dashboard
UID/Customer Insight
Maximizer
Database Server
MAXSQLPROD
Database Server
lwSQLPROD
Web App Server
URBANE
App Server
INTWEB01
UID Database Server
ABRAM
Business Objects App Server
BOBAPP
Epiphany App Server
EPIAPPPROD
Unica App Server
UNICAAPP
Enterprise Data Warehouse
(EDW)
BISQLPROD
STAGING PROD
BISQLPROD2
Used for running SSIS load packages
against BISQLPROD
UNICA
EPIPHANY
.txt
files
Unica Database Server
UNICASQL
Unica
Staging
(SQL Server)
Unica
Data Mart
(SQL Server)
Epiphany Database Server
EPISQLPROD
Epiphany
(SQL Server)
Business
Objects
SAS App Server
NINEVEH
BaseSAS
SAS-EG
Client App
SAS-Web
Reporting
Studio
HK Inv
(oracle)
SBDS
(SQL Server)
Ministers Disc
(monthly)
UID Staging
(Sybase) UID
Web App
Tri-Media
RTS/Alliant
(SQL Server)
WORDSearch
(SQL Server)
LW Worship
(MY SQL)
RCP
(SQL Server)
Vendor LW
Commission
MOSAIC
MRI
Simmons
Vista
(static)
SAS
Libraries
Base SAS App
Server
ZERUBBABEL
BaseSAS
JDA
SAS
Information
Maps
Retail Acq RCI
(static)
Unity Mail
(static)
Freq Buyers
(static)
SBDS
(.asp)
Church leaders,
organizations, ACP
UID
(Sybase)
Discoverer
Prospects
(access)
DataFlux
Client App
NCOA
Natl Chg of Addr
(90 Days)
PeachTree
[TD]
AEC
Addr Elmt Corr
(Semi-Annual)
Post Office
[TD]
UNC
Universal Coder
(Java)
Unique Customer ID
B&H
Dashboard
(Java)
Customer
Insight
(asp)
C
ustom
erAccount
Customer
Insight
(SQL Server)
My Study Bible
/ Oracle
Campaign
Campaign
Account / Transaction / Product
Store / Inventory
Account / Commission
Royalty
Shipping / Receiving / Inventory
CRD Prospect
Account / Transaction
Lifestyle Segmentation
Church Demographics
Account / Transaction / Product
GL / Account / Transaction / Product
People
Church Leaders
Email Addresses
Maximizer
(SQL Server)
App Server
MAXAPPPROD
Maximizer
(asp)
Account / Transaction / Product
Account / Transaction / Product
All Data
All Data
All Data
CRD
Data
M
M
S
/C
ustom
erH
ousehold
ing
/S
ale
s
Address
Address
Address
Address
App Server
GUARDIANNEW
BlueFusion
(VB6)
LW Std
Uniq Cust
(VB6)
Customer
Customer
Custo
mer
C
ustom
er
Customer
Household / Customer / Profile
Customer
All Data
All Data
Customer
Name
B&H
Sto
re
/In
ventory
Address
Address /
Name
Unique
Customer
ID
Group 1
Address
Standardization
(Householding)
[Vend LW
Comm/TD]
Unique
Customer
ID
Address
Business Intelligence
Technology and
Integration Flowchart
Address
Vendor
LW
Commission
New or
Amended
Site
Account / Transaction / Product
Sales
Customer
Customer
Customer
Customer
Glorietta
(static)
Ridgecrest
(static)
Maximizer
(static)
Conference Attendees
Conference Attendees
Assigned Consultant
Household
Segm
entatio
n
à
/ß
Sale
s
ScanUS
Client App
.txt
files
App Server
THOMASSCANUS
ScanUS
Network App
Network Computer
[CRD/CSD/B&H]
Notebook Computer
[CRD/CSD/B&H]
.txt
files
Any Data
All Data
.txt
files
Customer
Budget/Proje
ctio
ns
Budget / Meeting Category
Store
Demographics / Price Event
JDA
(MY SQL)
Coupon
Customer
JDA
(DB2)
[CRD/Retail]
[CRD]
[B&H]
Campaigns &
Mail Lists
[Retail]
Campaigns &
Mail Lists
[CRD]
SAS Reports
[Retail]
SAS Reports
[Retail/CRD]
SAS Reports
[Retail/CRD]
WebIntelligence
Reports
[all divisions]
[TD/CRD/Retail]
[TD]
[Orgs/TD/LW Rsch]
Buying
Behavior
Buying
Behavior
JDA
(DB2)
Non EDW
Data
Non EDW
Data
Any Data
Any Data
Aptify
Product
Contribution
(CRD / B&H)
BHPOS
Aptify
(SQL Server)
Oracle eBus
(oracle)
Price Events
New Seed List Update
Store
Data
Lifestyle Segmentation
App Server
HAZOR
MSC
Mktg Sppt Cntr
(Java)
[TD/CS]
Mail Lists
Salesà/ßAssignmentsUID
Address
PAR
ItemPopularity
Websphere
Commerce
for
Lifeway.com
WCP01
(DB2)
Address
Business
Objects
Universes
Product Class
1/24/2014
Mosaic
My Study Bible
SSRS Reports
All Data
Big TX
Church leaders,
organizations, ACP
Church leaders,
organizations, ACP
Tomcat Reports
(TD)
Changed Names
and Addresses Unique Customer ID
Customer / UID

5. OUR ISSUES (CONT.)
• Analysts spending 80% of time gathering data
sources
• Data is limited to structured sources
• A lot of HiPPOs
“Without data you’re just another person with an
opinion.”
– W. Edward Deming

6. WHY HADOOP?
• ETL -> ELT
• Centralized schema development
• Analysts get to be analysts
• More and different types of data
• “Win with data” – new analytics culture

7. FIRST STEPS
• POCs on several distributions
• Ran POCs on a 5 node VM cluster
• Chose Hortonworks
• Pure play distribution
• Engineering expertise
• Support model

8. INVESTMENT
• Started with a 12-node physical cluster
• Cisco UCS hardware to meet infrastructure
standards
• Training for 1 administrator and 2 developers
• Two week professional services engagement
• Got environment up
• Successful ETL offload for two data marts
• Templated framework for remaining data marts

9. EMPLOYEE SKILLS
• SQL skills translate well for Hive
• Pig can be picked up quickly through training
• Forward thinking DBA for administration

10. HADOOP ECOSYSTEM TECHNOLOGIES
• Currently using:
• Hive
• Pig
• Sqoop
• Oozie
• Ranger

11. TYPES OF DATA
• Structured Systems
• ERP
• Logistics
• POS
• Merchandising
• Unstructured
• Wifi analytics
• Price API
• Clickstream
• International weather
• CPI
• Census

12. ARCHITECTURE
Enterprise
Data Hub
Cloud
Data
Structured
Data
Unstructured
Data
A
P
I
Apps

13. ARCHITECTURE (CONT.)

14. BUSINESS COLLABORATION
• Socializing impacts of Hadoop
• Data gathering time for analysts
• New data sets
• Improved schemas
• Successful implementation of Hadoop
• Lays the groundwork for new BI&A tooling
• Creates an Agile BI framework

15. USE CASES
• Segmentation/Targeting of customers
• Omnichannel customer views
• Pricing optimization
• Product clustering
• Supply chain optimization
• Cannibalization of products and customers
• Fraud detection on AP

16. NEXT STEPS
• Continued growth of cluster
• HA/DR planning
• Cloud vs On-premise
• EMC Isilon

17. FUTURE OF HADOOP AT LIFEWAY
• Data science
• Machine learning
• Process optimization
• Heat mapping for store optimization
• Event log and sensor aggregation – predicting failure

18. Jim Forrester
Director, Information Services
jim.forrester@lifeway.com
@jrforrester
HADOOP @ LIFEWAY