Punctuated Equilibrium, Celestial Navigation, and APIs

Punctuated Equilibrium,Celestial Navigation,and APIs
Competing through dynamic adaptation
Sam Ramji, Apigee @sramji
Dan Jacobson, Netflix @daniel_jacobson
Michael Hart, Netflix @michaelhart

PUNCTUATED
EQUILIBRIUM

Darwin formulated his theory of evolution about 150 years ago

Based on observations he made in the Galapagos Islands 15 years earlier

A wild diversity of creatures existed in a new environment

Starting from an ancestor which looked like this

Geospiza Fulginosa

Finches evolved that looked like this

Geospiza Fortis

and this

Camarhynchus Pallidus

and this

Camarhynchus Pauper

and this

Geospiza Conirostris

and this

Certhidea Olivacea

and this

Geospiza Scandens

For many years the belief was that this change happened slowly and gradually.

In 1972, Stephen Jay Gould and Niles Eldredge proposed a new idea

that evolution is not slow and gradual

but sudden and violent.

Applying this view to the observations of finches

G. Scandens
C. Pallidus
G. Magnirostris
G.
Conirostris
C. Heliobates
G. Fortis
G. Fulginosa
C. Psittacula
C. Pauper
C. Olivacea
C. Parvulus
Certhidea
Camarhynchus
Geospiza

“Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows.”
Charles Darwin
On Origin of Species

So while it may look slow and gradual in hindsight

Evolution is experienced in punctuated bursts.

If you’re living in a punctuated burst of evolution
it feels like a revolution

CELESTIAL
NAVIGATION

Exploration

like evolutionary change

only looks smooth in hindsight

Living through it is usually chaotic

Karen James
The Beagle Project Blog

To navigate, you need a map and instruments

Maps exist for transferring knowledge

and they too have evolved over time.

They started as oral traditionand were written down in a form called a periplus

Periplus of Hanno
Courtesy of Heidelberg University

Periplus of Hanno
Courtesy of Cornell University

Far less efficient knowledge transfer than
a modern map of the same journey

Map of Hanno’s Journey
Courtesy of Bourrichon/Wikipedia

Exploration was dramatically held back for want of a map

In the two thousand years between Hanno’s journey on a Phoenician trireme

And the Mediterranean caravel of the 15th century

Maps had only evolved to be graphical descriptions of coastlines

That was a map published a few years before
Columbus crossed the Atlantic to find India

After his crossing, his expedition shared their knowledge
in a new map

Still far from perfectbut much improved.

The biggest challenge in this kind of exploration
was determining their location on the Earth

Instruments for measuring latitude had beenused and improved for centuries

Longitude was the hard problem.

You needed to know not just the angle of the sun and stars

you also needed to know the precise time.

Regardless of your sailing technology
without the proper measurement
you were lost

We are not promising a perfect map of the new world

But it should be more like this

than this

Periplus of Hanno

and we will show you what we know how to measure.

There are more niches today than we’ve seen before, so we need to borrow from nature

If we start with an API we can explore all the niches around our business

Visualization by Apigee

The leaders of today’s Internet

clearly understand this mechanism.

They understand that the distribution model for value has changed in the Internet era.

Packaged Goods
Consumer
Retail Store
Producer
Internet Services
App
Developer
Consumer
Provider

Developers took their APIs and explored the niches for them

The providers and the developers both benefited from this adaptation

Suddenly this seems obvious to everyone.

Data from Programmable Web

And developers are racing to pack the niches.

Data from Wikipedia

This is a sudden, material shift in competition.

It only looks gradual if you’re losing.

BUSINESS MODELS
From 1st Party to Partner to Platform

We’ve seen punctuated equilibrium in business models over the last hundred years

App Developer
Device App
Big Box Retailer
Mobile App
National Chain
Web Retail
Department Store
Web Catalog
Specialty Store
Indirect Sales
Direct Sales

What’s the environmental stress driving the current rapid change?

Hardt’s Theorem: The Internet Power Law
The first wave of the Internet demonstrated the economic impact of web-based business models.
99:1
95:5
The API-driven Internet is demonstrating the next concentration of power and is reflecting a “99:1” distribution (examples: Twitter, Facebook) due to the high switching costs and effective lock-in through software.
The HTML-driven Internet showed new business models that focused revenue in the 3rd standard deviation (examples: Amazon, EBay). Reality reflected a “95:5” rule where 5% of companies dominated the transactions and profits.
80:20
Previous eras of business showed a normal distribution for revenue, with most firms getting most of the revenue. In the later half of the 20th century, business model innovations focused revenue in the 2nd standard deviation above the mean. The “80:20” rule became conventional wisdom.
The next wave of the Internet is demonstrating the economic impact of API-based business models.

But you need to tackle it in a way that fits your business

1st Party
Open
Open
Open
Platform
Partner

These are complementary and distinct.
Open is different for each one.

1st Party Apps
Partner Apps
Platform Ecosystem

1st party
1st party is about offering direct access to your core business via apps that you make or contract out

1st party
Here open means all the business is accessible to internal developers and contract specialists

partner
Partner is about enabling directed development of apps that extend your business model towards your business partners

partner
Here open means existing partners have access to your business via APIs and can innovate asynchronously

platform
Platform is about enabling unknown developers to build brand new apps and businesses that will surprise and inform you

platform
Here open means enabling business models and allowing developers to support each other at massive scale

open
Open is attractive

open
Open is Biz Dev 2.0

1st Party
Open
Open
Open
Platform
Partner

open
Open lets you navigate across the possible business models when your first model doesn’t work as planned

1st Party
Open
Open
Open
Platform
Partner

To get your API strategy properly grounded

“
John Musser
Programmable Web

But how?

Let’s break it down

Establish Target Segments
Engage Developer Channel
Set Industry Goal

target segments
An API should extend your core business
into a new part of the market

target segments
Your core business already has
key performance indicators

target segments
So apply your KPIs to the new market segment you’re targeting with your APIs

target segments
What is the market impact you need to create in order to succeed as a business?

target segments
What does the targetsegment need that it is not getting from you today?

target segments
The answer will be the foundation of your API strategy.

developer channel
In most cases the channel for your API will be developers, but what do they need?

developer channel
A profit motive.

developer channel
Here are the leading profit models for developers today

developer channel
In-app purchases
Affiliate royalty
Your advertising spend
Market awareness of their offering
App sales

industry goal
If you don’t know where you’re going, you definitely won’t get there

industry goal
Partnerships and platform businesses are very different things.

industry goal
Partnerships are formed to serve a known set of entities

industry goal
A partner API should be traceable to each partner’s relationship

industry goal
And support end-to-end business processes

industry goal
A platform exists to create massive and unpredictable opportunities

industry goal
All your technology, support, and community decisions will be about surviving the scale of adoption

That’s the strategy dimension.

The execution dimension is what you already know.

Planning.
Management.
Organization.

Putting all this in context gives us a map for our API strategy

The instruments will be your KPIs and your core API metrics: performance and adoption

ARCHITECTURE
From Mainframe to Mobile

Connected
Devices
Smartphone
Web App
N-tier
Personal
Computer
DCOM
CORBA
Website
Minicomputer
Client/Server
Mainframe
Integrated
Distributed
Computing

“
The Web is Dead. Long Live the Internet.
Chris Anderson
Wired Magazine

The Web is Dead. Long Live APIs!

Twitter traffic distribution shows what he means

Twitter Traffic in 2010

Netflix traffic distribution is nearly the same.

The majority of Netflix traffic comes from
API-drivenconnected devices.

Like Columbus, Netflix started with a map of the coastline

Build an open API as a platform
and let a thousand flowers bloom

But they had left the coastline far behind

And the instruments indicated that there were fewer flowers than expected

Netflix API Requests by Segment

But partners started building apps for connected devices and the business took off

Wii
iPad
PS3
iPhone
Apple TV
LG TVs
Samsung TVs
Roku
XBox

Architecture should
reflect the business model

So Netflix has drawn the following map

Wii
Roku
PS3
iPhone
iPad
Google TV
Apple TV
XBox
LG TVs
Samsung TVs

Instruments show that API traffic has grown tremendously in a short time

Growth of Netflix API

20,000,000,000 API requests per month.
Is that a cause for celebration?
Or for concern?

When you’re navigating uncharted waters,
speed is not your friend.
Perhaps it’s time to slow down and avoid risking unknown reefs.

Navigating this growth challenge for Netflix means that the next API revision will focus on reducing overall traffic.

Part of this redesign is reviewing conventions

Punctuated Equilibrium: REST
Data sourced from
ProgrammableWeb

REST seems obvious but assess what makes sense for your business.

Tiered architecture helps you navigate different problems with agility

Controller
Model
View
iPhone Wrapper
Recommendations
R12n Service
Xbox Wrapper
UNIFIED LIST/TITLE API
Wii Wrapper
User Info
User Service
PS3 Wrapper
Similar Movies
Similar Movie Service
SHARED API INTERFACES
Roku Wrapper
Movie Metadata
DataNormalization
&
Resiliency
iPad Wrapper
USER API
Viewing History
Apple TV Wrapper
Ratings
PC / Mac Wrapper
TiVo Wrapper
API Repository
Layer
Wrapper
Layer
Web Service
Layer
Shared
Layer
Source Data
Layer
App Layer
API Layer

Stable
Flexible
Agile
iPhone Wrapper
Recommendations
R12n Service
Xbox Wrapper
UNIFIED LIST/TITLE API
Wii Wrapper
User Info
User Service
PS3 Wrapper
Similar Movies
Similar Movie Service
SHARED API INTERFACES
Roku Wrapper
Movie Metadata
DataNormalization
&
Resiliency
iPad Wrapper
USER API
Viewing History
Apple TV Wrapper
Ratings
PC / Mac Wrapper
TiVo Wrapper
API Repository
Layer
Wrapper
Layer
Web Service
Layer
Shared
Layer
Source Data
Layer
App Layer
API Layer

Server architecture should support both crests and troughs of the waves of demand

Instance Architecture Based on Specialization
List Creation
Dependency Service
ELASTIC INSTANCE LAYER
API METADATA CACHING LAYER
Meta
Data
Dependency Service
METADATA SERVICE

Instance Architecture Based on Specialization
Handles Request/Response
List Creation
Dependency Service
ELASTIC INSTANCE LAYER
Caches Dependency Data
API METADATA CACHING LAYER
Meta
Data
Dependency Service
Populates and Manages Cache
METADATA SERVICE

Map out your usage patterns and cache your data accordingly

Vertical Caching

Vertical Caching: Netflix Full Movie Data

Horizontal Caching

Horizontal Caching: Netflix Basic Data

Combining horizontal and vertical caching may be the best approach when building for multiple geographies

Two-Dimensional Caching

Design for where you’re going

Not for where you are

You may be starting here

But you must design for here

You don’t need to implementfor massive scale

But you must design for it or you will follow your successful ocean crossing with a massive shipwreck.

Navigation also means constantly adjusting your course to ensure you arrive at your final goal

Sometimes adjusting course on an API means you must change your version

Versioning means supporting multiple applications, all of which basically do the same thing

Versioning
1.0
1.5
2.0
3.0?
4.0?
5.0?
Today

If possible, go versionless

Version-less API
1.0
1.5
2.0
Here there be dragons
versionless
Today

Rules for going versionless
in your production APIs

Extend your API by extending data types
Addition is not version-worthy
Better to be incomplete than inaccurate
Withhold implementation if you are unsure

With APIs emerging, we need better tools help us navigate

The service level agreement is your sextant

Set reasonable service level agreements

What is reasonable will vary by API and use case
but you must communicate them to your users.

Measure average latencies,
error rates and types,
and respond when SLAs are broken

Having visibility into performance means you can tack immediatelyrather than after your users threaten to mutiny.

Longer-term navigation requires higher-level metrics

Request-based metrics such as what endpoints were called and what parameters were passed can show you what aspects of your API are popular.
Response-based metrics such as what was delivered, how quickly, and whether the response was valid can show you what aspects of your API need work.

System trace metrics that track what underlying systems were called and how they responded can show you where you need to evolve your internal architecture.
Business performance metrics such as how much revenue or how much customer engagement is occurring through the API show you how close you are to delivering on the business plan.

DATA
From Silos to Social

Domain-specific
Data APIs
Private Cloud DBs
Caching DBs
Data API
Data
Warehousing
Mainframe
RDBMS
Flat file
Shared
Silos
Data

Punctuated equilibrium in data sharing

App
Org
Cross-org
Cross-business
Mainframe
Databases
Middleware
APIs

So we are evolving to cross-business sharing

There are a few challengesin making this journey

Sharing your data via an API entails a different set of considerations than APIs which expose your services

loss of control
only recourse is legal
enforcement is expensive

That said, it may be time to get over
your control issues

Be honest about the value you could get from sharing your data outside your corporation

instead of just the costs and risks of sharing it.

In opening up its movie data warehouse

Netflix found that the cost was the same as any API,
the risks could be managed through rate limits and access control,
but that now others could build great movie discovery experiences that led to increased Netflix viewing hours.

Sharing data as a service means making a few course corrections on your API

enable larger downloads for fewer queries
more liberal retention policies means less API traffic, higher performance, and less cost
push incremental updates
limited access to richer queries

Looking forward, how are we going to work through this wave of shared data?

Two different dimensions are apparent

First, no one wants all of your data,
just some of it.
Just the right parts of it for their particular need.

Where have we seen this before?

App
Org
Cross-org
Cross-business
Mainframe
Databases
Middleware
APIs

Last time the right solution was a
query language
Perhaps it’s the right solution now

Instead of a static REST call, we could pose a query like “what are the highest rated movies from the 1980s?”
http://odata.netflix.com/Catalog/Titles?$filter=ReleaseYear le 1989and ReleaseYearge1980and AverageRatinggt4
&$expand=Awards

Caveat structor does apply
This is new ground and we haven’t seen anyone do this at massive scale

Second, people want just the right parts of your data for their particular need
But they need the right parts of other people’s data to have a complete context

Where have we seen this before?

App
Org
Cross-org
Cross-business
Mainframe
Databases
Middleware
APIs

Last time the right solution was middleware for distributed queries
Perhaps it’s the right solution now

But the rules have changed a bit since the data is laying all over the Internet

[{
"id": null,
"name": null,
"type": "/dining/restaurant",
"/business/business_location/address": [{
"street_address": [],
"citytown": {
"id": "/en/toronto"
}
}],
"cuisine": [{
"/dining/cuisine/region_of_origin": [{
"!/film/film/featured_film_locations": [{
"id": "/en/the_italian_job"
}]
}]
Instead of a single domain query, we could ask for a list of “Toronto restaurants with cuisine from a filming location of ‘The Italian Job’”

Caveat structor still applies.
This is very new terrain indeed.

The bigger question than what should you be sharing out
Is what should you be sharing in

What data APIs should your business be using?

What could you offer your customers
if you knew which of them were friends with each other?

What does that logo mean to you today?
Perhaps it’s time to think differently.
Facebook is a massive data API
that lets you correlate your customers
with their true context

You could move beyond a flat-earth view where all your customers are their own islands of data
And each of them were connected in ways that makes your business more valuable to them

What could you offer your customers
if you knew where they like to go?

The more of your customer’s context that you can understand
The more time you can save them
And that makes your business more valuable to them

The good news is that there are already data APIs to get this context
Now you need to focus on sharing in

InCLOSING

We are going through a period of rapid change in business models, architecture, and data

Navigating based strictly on the stories of others

Periplus of Hanno

Will not give you the clear map that you need

Develop instruments around your API
that help you understand where you’re going

so that you can correct your course
and beat your competition in the race for the future.

THANK YOU
Questions and ideas to:
@michaelhart
@daniel_jacobson
@sramji

http://blog.apigee.com	6511
http://www.web2expo.com	574
http://blog.sonoasystems.com	79
https://twitter.com	73
url_unknown	14
http://www.linkedin.com	12
http://translate.googleusercontent.com	3
http://webcache.googleusercontent.com	3
http://kred.com	1
https://abs.twimg.com	1
http://dev.en.oreilly.com	1
http://www.twylah.com	1

Punctuated Equilibrium, Celestial Navigation, and APIs

by Sam Ramji

Accessibility

Categories

Upload Details

Usage Rights

12 Embeds 7,273

Statistics

Punctuated Equilibrium, Celestial Navigation, and APIs Presentation Transcript