Democratizing Apache Spark for the Enterprise with Jonathan Gole

Editor's Notes

• #2 Hello! Thank you to Databricks and the spark summit organizers for having us here. I want to share our story of unifying analytics by successfully spreading Apache spark across our enterprise
• #3 You may know us for our credit cards and our catchy commercials.. Sorry I couldn’t bring Samuel L Jackson However, we are a diversified financial services leader. corporate offices nationwide – headquartered outside of Washington DC, office one block away in San Francisco. Cap One is relatively new company, IPO’d in 1994. First FinTech unicorn. To understand us, look to our long-term CEO and co-founder Rich Fairbank.
• #4 Founded to disrupt the industry by leveraging the technology and data revolution Embraced rapid experimentation and rigorous data-driven decision making Data technology was and is still at the heart of Capital One’s success Apache Spark is the latest technology revolution transforming the way we use data to drive our businesses.
• #5 PdM team delivering new products and services for transforming our ability to succeed with Data. In this role, my team has built the largest Spark projects in Capital one tell you the story of how we transformed our business through the unifying capabilities of Apache Spark
• #6 Like other financial companies…Built a mature, growing business on prior technology paradigms As data systems aged, limited in ingenuity and innovation in our core businesses - company uses data at the heart of everything, the cost was meaningful - needed to change this paradigm
• #7 associates were clearly impacted - less efficient, less able to focus on innovative, less able to test and learn. testing & experimentation were core to the foundation of Capital One, so technology limitations were challenging the nature of our culture shortcomings as a team - people were isolated from each other, working in different technology paradigms Create a new paradigm – both eliminate technology limitations and our culture around data
• #8 typical evaluation process – work with experts, get hands dirty build around Spark - Mot active project, Unifies batch and streaming, flexible APIs and support for multiple languages more important - product-centric focus: Unified ownership, real high value use cases, not abstract technology principles Focus on our associates as customers Learn through doing - learn quickly and altering approach as needed Do not start by building an enterprise platform, first build use cases ourselves to deeply understand needs
• #9 challenges in first use case: rolled our own infrastructure - several months to master Associates working in new paradigms: AWS & public cloud, distributed system, use of scala, separation of storage and compute quickly realized we could not upfront design one system that would meet everyone’s needs Diversity & complexity of our business - 80M customers, long-term relationships, regulatory burdens. Aware of our own ignorance, need to continue to learn through use cases & our customers
• #10 Operations: business critical applications, fulfill customer & regulatory promise Enable a high degree of flexibility and customization by engineers Yet maintain a degree of efficiency in development, standards to well govern our critical capps Answer: build a custom in-house shared application framework, leveraging shared code libraries and tooling. Analytics – insight generational, data science, and machine learning Iterate on new ideas quickly, collaborate effectively Ignore infrastructure - It should just work, and let me be creative Answer: deploy a shared data science platform – a Unified Analytics Platform. We knew we didn’t have the expertise, so we partnered with Databricks
• #11 iteratively expanded on that strategy Invested continuously into improving the products core product philosophy to drive our decision making – similar to Suffering Oriented programming Focus first on enabling and proving out new transformative functionalty – use real business use cases to validate progress - ( highlight Spark 2.0) set the foundation for growth. Improve performance and scalability. Reduce technical debt. Build tooling to attracting future customers (Highlight – JSON DSL, Inner-Sourcing) Expand through good user experience – invest in ease of use and efficiency (Ensure a great on-boarding expience for UAP, created self service training in notebooks)
• #12 Over time, this product and customer focus organically led to a large ecosystem operating around our Apache Spark products. Spark as a primary data compute layer for many of our systems We didn’t have control over this whole ecosystem, but we embedding flexibility in our platforms and allowed for emergent design Apache sparks reflects this ecosystem approach- continue to deepen the connectivity across the data and software ecosystem
• #13 Truly shows the massive impact of Spark- a design that unifies domains can have a massive ROI TO call out a few accomplishments, our developer & DS community: Re-building 100s of ETL jobs in the cloud self-service marketing automation platform improving our fraud defenses, protecting customer identities Enabled rapid testing of ML approaches, detecting changing credit trends, automating manual processes
• #14 transforming what our associates could achieve leveling-up in their productivity Enable shift to more creative ideas. Given the shared technology paradigm, associates organically work together, collaborate more effectively. growing importance of small, empowered, cross-functional or cross-skilled teams.
• #15 Highlight and reemphasize several learnings from our efforts with Spark: 1) see colleagues as customers, understand & solve for both broad platform needs & enable unique problem solving 2) Learn quickly through doing, instead of abstract evaluation. 3) enable community development, collect feedback, and provide training 4) unified approach – accelerate innovation by removing barriers Thank you for having me here, I’m proud to participate in such a great Spark community