HP ships millions of PCs, printers and other devices every year to customers in all market segments. Many of these systems have had various generations of data collection and reporting, going back as many as 16 years. That has led to a significant sprawl of custom data formats, specialized code and numerous brittle legacy systems collecting, analyzing and reporting data. This session will focus on samples of HP’s journey to find, catalog and ultimately eliminate these systems by migrating to Apache Spark with Databricks in the cloud. Hear about HP’s challenges dealing with legacy systems (some even located under engineers desks) and how the power of AWS, Spark, and visualization tools has significantly simplified their migrations. You’ll also learn how the success of this endeavor is not just in counting the number of systems deprecated, but also how the process is evolving into building companywide shared Spark libraries, notebooks and web services that are accelerating future migrations and analysis using Spark.

1. Our Journey of
Migrating Dozens of
Oddball Telemetry &
Analytic Systems to
Apache Spark
Herding Cats

2. John Cavanaugh
Master Architect/Strategist
cavanaughwww@hp.com
linkedin.com/in/john-cavanaugh-2307513
20 Years in R&D, ~5 in BigData
MSEE / MBA
Technology Savvy Business Leader
Business Savvy Technology Leader
Passionate for all things
Data, Advanced Analytics
and Business Value that
can be extracted…
2017
NBA
Finals
Bottom Line, Im a
Data Junkie…
Feel free to stop me
in the hall and talk
shop…
Who you got??

3. ~330K Employees WW

4. ~330K Employees WW
~50K Employees WW
Dozens of Sites WW
Print, Personal Systems &
3d Printing Business Groups
Products and Services

5. + +
Expectation Reality

6. History & Challenges
•Coming to Terms with Harsh Realities
Envisioning the Future
•Spark/Databricks as a Cornerstone
Building Together
•Renovating the House, While Living In It…

7. History & Challenges

8. 1999-2009
Data Stone Ages
2010-2014
BigData Wild West

9. What have
we done…
Inconsistencies
Lots of
Problems
Circa 2015

10. Envisioning the Future
www.tesla.com/solarroof

11. Catalyst
• Acknowledging
Harsh Reality
• Emergent Systems
Thinking
• HP Global Virtual
Community
• HP Corporate Split
Strategy
• Tangible Business
Results & Efficiencies
• Enablement Focus
• Drive Alignment only
on the Critical Few
• Architect for
Empowerment &
Data Democracy

12. Cloud Platform & Data Lake
Data Analysis Framework
Data Format & Warehouse
Data Analysis Tooling

13. BusinessManaged
ITManaged
Cloud Hosted
OnPrem Hosted
GOAL

14. Building Together

15. BusinessManaged
ITManaged
Cloud Hosted
OnPrem Hosted
Vertica
Hadoop
2
Hadoop 1
High ROI Targets
Scary Long Tail…
GOAL

16. Talking to Business Teams & Engineers
Scary Things for
Them to Hear
Hi, I'm from IT,
and I'm here to help.
Hi, we have a new platform
that you need to adopt/pilot.

17. BusinessManaged
ITManaged
Cloud Hosted
OnPrem Hosted
NEW
PIONEERS
High ROI Targets
Scary Long Tail…
GOAL

18. BusinessManaged
ITManaged
Cloud Hosted
OnPrem Hosted
High ROI Targets
Scary Long Tail…
GOAL

19. OnPrem Hosted
JBOSS
JBOSS
GLB LLB
JBOSS
JBOSS
LLB
Hadoop
1HDFS
Vertica
9 REST Endpoints
8 Formats
40+ SW Clients
20+ HW Clients
FTP
HTTP
Legacy
Example – Print Telemetry with ~3 Billion Data Points / Day

20. Cloud Hosted
OnPrem Hosted
JBOSS
JBOSS
GLB LLB
JBOSS
JBOSS
LLB
Hadoop
1HDFS
Vertica
9 REST Endpoints
8 Formats
Telemetry
AWS
ElasticBeanstalk40+ SW Clients
20+ HW Clients
FTP
HTTP
Kinesis
Legacy Modern
Example – Print Telemetry with ~3 Billion Data Points / Day
HTTP

21. Cloud Hosted
OnPrem Hosted
Clone
Proxy
JBOSS
JBOSS
GLB LLB
JBOSS
JBOSS
LLB
Hadoop
1HDFS
Vertica
9 REST Endpoints
8 Formats
Adapters
AWS
ElasticBeanstalk
Telemetry
AWS
ElasticBeanstalk
B
A
40+ SW Clients
20+ HW Clients
FTP
HTTP
HTTP
Kinesis
Legacy Modern New
Example – Print Telemetry with ~3 Billion Data Points / Day

22. BusinessManaged
ITManaged
Cloud Hosted
OnPrem Hosted
High ROI Targets
Scary Long Tail…
GOAL

23. Overcoming
Fear
Internalizing
Benefits

24. Discovered Gaps
• Shared Services (Geo)
• Dataset Registry
• Schema / DataDictionary Registry
• Foreign Data Replication
Strategic Siblings
• Data Telemetry Initiative
• Machine Learning Emphasis
• DataScience Education & Careers
• University Partnerships
Journey not a Destination

25. Ultimately Data/Analytics are culture,
but start with a Tribe & Community
Focus on Enablement, Architect for
Empowerment & Data Democracy
Tools/Technology are important, but
people are what make things happen
Change is hard, but a Community will
have the change agents you need

26. Thank You
for Attending
Any
Questions?

27. Acknowledgements & Media References
• TomCruise –
thatmomentin.com/2016/12/02/minority-
report-2002/
• MikeRowe - Dirty Jobs via Discovery Channel
• HerdingCats –
blogs.uoregon.edu/cscenter/tag/kelsey-
zlevor/
• Rear View - www.banyantreatmentcenter.com
• ClintEastwood –Wikimedia.org
• StoneKnife –www.imgarcade.com
• Whack a mole - coderchronicles.org
• Chaos –braidmission.org
• Hangover – blog.supercareer.com/how-to-
deal-with-hangover-at-work/
• FutureSolarHome – tesla.com
• BuildingTogether – www.af.mil
• FearofHeights – imgur.com/NBA97x3
• Fulcrum -
upload.wikimedia.org/wikipedia/commons/d/
d8/Lever_drawing.gif
• Drowning –
www.crosswalk.com/faith/women/12-
survival-tips-for-when-you-feel-like-you-re-
drowning.html
• TakeAway –
http://www.indianlodge.co.uk/takeaway.html
•