Innovating with
Developer Experience
Why DX cannot be an afterthought
Sangeeta Narayanan
@sangeetan
https://www.linkedin.com/in/sangeetanarayanan/
#Innovation
WHY?
Counter/respond to disruptors
Stay ahead of competition
Keep up with evolving customer
expectations
Respond to changes & threats
e.g. regulatory, economic,
security
HOW?
Reducing Time to Market
Lowering Cost
Accelerating Biz Activities
Ingredients
For
Innovation
Velocity
Velocity Risk
Velocity Risk Ambiguity
#Innovation
Powered By
Technology
#Innovation
Powered By
Technology
Starring
Developers
INNOVATION
INGREDIENTS DEVELOPERS
INNOVATION
INGREDIENTS DEVELOPERS
What can we learn from companies who have
been successful at empowering their
developers to innovate?
CULTURE &
TECHNOLOGY
CULTURE
https://rework.withgoogle.com/p
rint/guides/5721312655835136/
https://rework.withgoogle.com/p
rint/guides/5721312655835136/
How much autonomy do people have?
Are they empowered to take risks?
What is the appetite for failure?
Are diverse opinions encouraged?
Velocity Risk Ambiguity
CULTURE
TECHNOLOGY
“Controlled experimentation is becoming
the norm in advanced software
companies”
The evolution of continuous experimentation in software
development by Fabijan et al ICSE '17
But change is risky
High performers have
440x faster commit-to-deploy lead time
170x faster MTTR from downtime
5x lower change failure rate
Velocity Risk Ambiguity
TECHNOLOGY
Culture &
Technology in
practice @Netflix
190+ Countries
130+ Million Members
1000+ Device Types
140+ Million Hours Streamed DAILY
Edge powers core
product experiences
Edge powers core
product experiences
Edge powers core
product experiences
Edge powers core
product experiences
Compelling product
Reliable experience
Compelling product
Reliable experience
At SCALE
Netflix
CULTURE
People over Process
If you want to build a ship, don’t drum up the
people to gather wood, divide the work, and give
orders. Instead, teach them to yearn for the vast
and endless sea.
- Antoine de Saint-Exupéry, the author of The Little Prince
Dream Team
Freedom and Responsibility
Informed Captains
Disagree Openly
Context, not Control
Highly Aligned, Loosely Coupled
Seeking Excellencehttps://jobs.netflix.com/culture
Netflix
TECHNOLOGY
INNOVATION
INGREDIENTS DEVELOPERS
Developer
Experience
Shortest, Safest route to Production
and beyond
UI
Developers
Server
Developers
Certification
Engineers
Infra
Engineers
About the “Developer”
Full cycle devs @netflix
Productivity
UX
ReliabilityScalability
Ops Insights Biz Insights
Architecture
Feedback Loops
Insights
Architecture
Customizable API enables rapid
innovation
Spot the difference
Zuul
Cloud
Gateway
Mid-Tier
Services
Netflix Edge Architecture
API
Service
Layer
NodeQuark
Edge
Device
Services
AuthN
Services
DEVELOPER PRODUCTIVITY OPERATIONAL INSIGHTS
Playback
Licensing
Netflix
Micro-
services
Customized API
Fine grained insights
Isolation
Capacity Management
Feedback Loops
Building confidence in changes
Handling server responses
Simulate special behaviors from server
Validate app response
Triggered externally by app developers
Transparent to app
Simone - A Distributed Simulation Service
Simulation Testing
Insights
Staying in the know - always
Quality of Experience
Time to interact with UI take on a given device
Data throughput for members in a certain region
Delay between pressing Play and playback start
Rebuffer rates
etc.
Self-enablement
CULTURE &
TECHNOLOGY
What is the culture of your Dev
community?
How well does your Developer
Experience support innovation?
Food For Thought
Thank you!
Sangeeta Narayanan
@sangeetan
https://www.linkedin.com/in/sangeetanarayanan/

Innovating with DevEx - Developer Relations Conference March 2019

Editor's Notes

  • #2 Anecdote + intro
  • #3 Reflecting on innovation strategies and creating an environment that fosters it
  • #4 Let’s start with they Why. Why focus on innovation? To stay relevant and grow, businesses need to Defend against disruptors - think Nordstrom competing with newer online only retailers Stay ahead of competition - no time to rest on laurels, amazon, netflix Ever changing customer needs - Netflix, Google, FB, Spotify Regulatory & compliance threats -
  • #6 It’s clear that innovation is has become a priority for more and more companies. Businesses have recognized innovation as a key to staying relevant and growing in a fast changing world.
  • #7 There are 3 ingredients for innovation
  • #8 Three Must-Haves for innovation to succeed: Ability to move fast
  • #9 Three Must-Haves for innovation: Ability to move fast Appetite for smart risks
  • #10 Three Must-Haves for innovation: Ability to move fast Appetite for smart risks Comfort with ambiguity
  • #11 Now we are living in a digital world, therefore technology is a powerful driver of innovation.
  • #12 And the people who leverage technology to generate business value are developers. It follows that developers play a key role in executing a successful innovation strategy.
  • #13 So there are 3 ingredients to innovation and developers play a lead role.
  • #14 It stands to reason that businesses should empower developers to move fast, take calculated risks and navigate ambiguity.
  • #15 What can we learn from companies who have been successful at empowering their developers to innovate?
  • #16 A common thread is using combination of culture and technology to support an innovation focus. Let’s dig into these aspects for a bit.
  • #17 Starting with culture. Most companies, teams and communities have a culture. The question is whether it has been shaped intentionally or has developed organically over time. Culture is more than a set of value statements. The real values of a company are shown by the behaviors and actions that get rewarded and those that are penalized. Being intentional about what those are is a great way to encourage the types of behaviors you want to see in your company, organization, team or community. Successful companies make it a priority to shape their culture.
  • #18 To underscore the importance of culture, a study to answer the question: “What makes a team effective at Google?” found that the #1 factor that high performing teams had in common was Psychological safety. Code-named Project Aristotle - a tribute to Aristotle’s quote, "the whole is greater than the sum of its parts" (as the Google researchers believed employees can do more working together than alone) - the goal was to answer the question: “What makes a team effective at Google?”
  • #19 This insight helps teams within Google optimize for an environment that creates and promotes psychological safety. That’s an example of being intentional about the culture. Additionally, as we saw earlier, the ability to take risks is a key must have for innovation. Unsurprisingly, Google is a great example of a company that nurtures innovation.
  • #20 Questions to consider when assessing the culture of a company. How much autonomy do people have? Are they empowered to take risks? What is the appetite for failure? Are diverse opinions encouraged? Answers to these questions will help assess how well a company is set up for innovation.
  • #21  Businesses whose culture encourages people to be nimble, take calculated risks and be comfortable navigating ambiguity put themselves in the best position to succeed at innovating.
  • #22 On to Technology. How do those innovation ingredients apply to technology?
  • #23 Successful companies take an iterative approach to innovation as opposed to aiming for instant perfection. It is very hard to get things right the first time. An iterative approach requires the ability to support experimentation, which in turn puts an emphasis on the velocity of software delivery. Image credit: https://svgsilh.com/image/576604.html
  • #24 At the same time, we know that change is inherently risky. A study of contributing factors and root causes of incidents across the industry will typically look like the above (example from Netflix). You can see that the leading cause of issues is change.
  • #25 If you needed further proof...
  • #26 This of course poses an interesting and hard challenge - how can software organizations support innovation by moving fast while maintaining reliability that customers expect? A study by the authors of Accelerate shows that high performing tech orgs are able to move FAST and stay SAFE most of the time. When things go wrong, they are able to recover quickly.
  • #27 Businesses who invest in technology that encourages people to be nimble, take calculated risks and be comfortable navigating ambiguity put themselves in the best position to succeed at innovating.
  • #28  Let’s look at a case study from Netflix
  • #33 Edge Engineering provides functionality and metadata to power core Netflix product experiences
  • #34 Edge provides functionality and metadata to power core Netflix product experiences
  • #35 Edge provides functionality and metadata to power core Netflix product experiences
  • #36 Edge provides functionality and metadata to power core Netflix product experiences: Signup, Discovery & Playback
  • #37 We are constantly dealing with the challenge of balancing creating a compelling product while maintaining a reliable service.
  • #38 Doing it at scale adds to the challenge because it doesn’t leave much room for error. How do we use a combination of Culture & Technology to support to create an environment for our developers that fosters innovation?
  • #41 We are very intentional about our culture. It is a core part of our hiring process and permeates all aspects of working at Netflix.
  • #42 The core philosophy of Netflix culture
  • #43 Our CEO Reed Hastings was inspired by the above quote, aspects of which made their way into our culture. Empowering people yields far better results than constraining them with processes and policies. If you want to build a ship, don’t drum up the people to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea.
  • #44 Giving people the freedom to make decisions, as opposed to telling them what to do. Balancing that out by giving them full responsibility of their actions. Equipping them with vast amounts of context in the form of data, strategy choices and any information. This allows them to build the muscle needed to take smart risks and navigate a world of ambiguity.
  • #45 Our technical decisions support our commitment to innovation by supporting risk taking and rapid iteration.
  • #46 How do we empower our developers to support innovation?
  • #47 By placing a major emphasis on Developer Experience
  • #48 For us it means two things: Give developers the shortest, safest path from idea to production Maximize the time and energy spent by a developer on their core mission - art and science of building a great product, vs. the process of developing, deploying, operating and supporting it.
  • #49 A word on DevEx (tangent). Who is a developer? With cloud, microservices, mobile, everyone is a developer. All communication happens through APIs You provision a server through an API You get data through APIs You page someone through an API So, who is a developer? EVERYONE
  • #50 Why is DevEx important? We think of developers as different from end users. The philosophy for Dev Tools is often function over form because developers are considered power users. The reality is that they *could* be power users, but *should* they be? Their time is best spent focusing on innovation, rather than learning tools. Any cycles spent on the latter is energy taken away from the former. As such, we should be placing an equal emphasis on all the other aspects of our tools and processes, in addition to function. How easy and quick is it to get onboarded, make a change, gain confidence in the change and roll it out? What does the feedback loop look like? Don’t make developers spend time on gathering and understanding context and learning tools.
  • #51 With that context in mind, let’s look at some examples of how we try to shorten the path to production for our developers and free up their cycles to focus on making an amazing service for Netflix members worldwide. https://www.flickr.com/photos/16210667@N02/25300692725
  • #52 The examples are in 3 categories
  • #53 It is important to make DevEx a first class consideration as part of your architecture. Bolting things on top will only take you so far.
  • #54 Remember we support well over 1000 device types? Form factors, input modes, network and device constraints vary widely across devices. Powering the UI takes a lot of data because of the highly personalized experience. How do we optimize the API to service the varying data needs of all these devices?
  • #55 Zuul: Internet facing service, provides routing, traffic shaping, security and more NQ: Translation layer to optimize experience for each device API: Orchestration and abstraction over other Netflix services PBL: Handles content licensing DP & OI: Tools and infrastructure to enable engineers to develop and operate complex systems at scale
  • #56 The NQ tier allows each UI team to have a dedicated service to customize data based on the needs of their device/app. They have complete control over the interactions between client and server. In effect, it acts as a translation layer between our API and UI tiers. This pattern is also referred to as the BFF pattern. Resources: https://samnewman.io/patterns/architectural/bff/ https://nordicapis.com/building-a-backend-for-frontend-shim-for-your-microservices/
  • #57 This architecture gives us a highly customizable API which frees up our UI developers to innovate on the experience. At the same time gives us the operational controls to maintain safety.
  • #58 I’ve talked a lot about pushing out changes rapidly. As a change makes its way through the system, we want to provide quick feedback on the quality of the change so we can increase our confidence of it succeeding in production.
  • #59 We want our apps to be able to handle any response that the server sends.
  • #60 One technique we use is Simulation testing. The idea is to trigger a particular type of behavior in a service by asking it to override its normal behavior with something that’s specified by the caller. This allows app developers to test various scenarios.
  • #61 Imagine a client developer wanted to test their app’s behavior if the server responded with this check. They can easily simulate this behavior by asking the server to return this response. The server advertises the various types of behavior that it supports for overrides. Clients use that to customize the specific behavior they want returned. This allows client developers to exercise various edge cases and scenarios that might otherwise be hard to create and maintain.
  • #62 The third category I’d like to touch upon is related to insights.
  • #63 App developers are responsible about the user experience on their platforms. They need to be able to access key business and operational metrics in a quick and easy to consume fashion. For instance they might want to know the uptake of a newly launched feature or they might be interested in understanding the performance profile of their app. And of course, failure is a part of life in a distributed environment. Ease of debugging is a critical piece as well. Let’s look at a couple of examples oh how we provide visibility into the environment for our developers.
  • #64 Examples: What is the QoE on various devices? How long before the user can interact with the app?
  • #65 Playback statistics and alerts on specific titles. Helps answer questions such as “What’s the error distribution of members experiencing playback issues watching The Triple Frontier?” If a specific UI version or device type is seeing large errors, we can dig into the issues.
  • #66 Server side telemetry is a rich source of information. Surface actionable insights derived from that data to client developers, with context is another way we help them troubleshoot problems. Making it self-service is another aspect we place high emphsis on.
  • #67 This was an example of how we combine culture and technology to empower our developers at Netflix and create an environment that fosters innovation.
  • #68 What is the culture of your Dev community? How well does your Developer Experience support innovation?