Punctuated Equilibrium, Celestial Navigation,and API Strategy
Upcoming SlideShare
Loading in...5
×
 

Punctuated Equilibrium, Celestial Navigation, and API Strategy

on

  • 792 views

Web 2.0 2011 Talk on API strategy by Sam Ramji of Apigee, Dan Jacobson and Michael Hart of Netflix

Web 2.0 2011 Talk on API strategy by Sam Ramji of Apigee, Dan Jacobson and Michael Hart of Netflix

Statistics

Views

Total Views
792
Views on SlideShare
792
Embed Views
0

Actions

Likes
0
Downloads
26
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Creative Commons Attribution-Share Alike 3.0 United States License
  • http://www.britannica.com/EBchecked/topic-art/197367/74641/Fourteen-species-of-Galapagos-finches-that-evolved-from-a-common
  • © Giuliano Gerra and Silvio Sommazzi - http://www.justbirds.it/
  • © Giuliano Gerra and Silvio Sommazzi - http://www.justbirds.it/
  • Camarhynchus Pauper© Nancy Bell
  • Camarhynchus Pauper© Nancy Bell
  • Geospiza Conirostris© Nancy Bell
  • Certhidea Olivacea© Giuliano Gerra and Silvio Sommazzi - http://www.justbirds.it/
  • http://upload.wikimedia.org/wikipedia/commons/thumb/6/69/Punctuated-equilibrium.svg/1000px-Punctuated-equilibrium.svg.pnghttp://www.talkorigins.org/faqs/punc-eq.htmlThere are two models for the flow of evolution, each with its adherents. Phyletic gradualism says that things change slowly over time, day by day, until one day you realize there has been enough change that something new has arisen.The opposing model is Punctuated Equilibrium. This model says that species tend to remain in a homeostatic mode of stable equilibrium – but that every so often some significant new stress affects the environment, prompting rapid, dramatic changes that settle down into equilibrium again.For our purposes, punctuated equilibrium serves as a powerful model for viewing the changes in technology, business models, and data over the last few decades and shows us that the rapid, dramatic changes part of the cycle is coming again.
  • http://www.csuchico.edu/~curbanowicz/DarwinSP2002Phil108.htm"Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows. There is a grandeur in this view of life, with its several powers, having been originally breathed by the Creator into a few forms or into one; and that, whilst this planet has gone cycling on according to the fixed law of gravity, from so simple a beginning endless forms most beautiful and most wonderful have been, and are being, evolved
  • Cherokee-class 10-gun brig-sloop of the Royal Navy downgraded to a 6-gun barquehttp://www.theora.com/msdb.php/OriginOfSpecies/orderBy/subject/msdbCat/subject/12%20-%20Geographical%20Distribution%20--%20continuedExploration - like evolutionary change - only looks smooth in hindsight. Not even the voyage of Darwin’s HMS Beagle was smoothly planned. Looking back he seems to have been predestined to arrive at the Galapagos islands, but this is not the case.
  • http://darwin-online.org.uk/people/chancellor.htmlAt some points the trip got pretty rough.
  • http://thebeagleproject.blogspot.com/2008_07_01_archive.htmlAnd the voyage itself was anything but linear.
  • http://digi.ub.uni-heidelberg.de/diglit/cpgraec398/0113
  • http://www.archive.org/details/cu31924031441847
  • http://en.wikipedia.org/wiki/File:Hannon_map-fr.svg
  • http://www.stedmundsbury.gov.uk/sebc/visit/precisionengineering.cfm
  • http://digi.ub.uni-heidelberg.de/diglit/cpgraec398/0113
  • http://www.britannica.com/EBchecked/topic-art/197367/74641/Fourteen-species-of-Galapagos-finches-that-evolved-from-a-common
  • http://upload.wikimedia.org/wikipedia/commons/thumb/6/69/Punctuated-equilibrium.svg/1000px-Punctuated-equilibrium.svg.pnghttp://www.talkorigins.org/faqs/punc-eq.html
  • http://upload.wikimedia.org/wikipedia/commons/thumb/6/69/Punctuated-equilibrium.svg/1000px-Punctuated-equilibrium.svg.pnghttp://www.talkorigins.org/faqs/punc-eq.html
  • http://www.csuchico.edu/~curbanowicz/DarwinSP2002Phil108.htm"Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows. There is a grandeur in this view of life, with its several powers, having been originally breathed by the Creator into a few forms or into one; and that, whilst this planet has gone cycling on according to the fixed law of gravity, from so simple a beginning endless forms most beautiful and most wonderful have been, and are being, evolved
  • Credit to Dick Hardt who formulated this in December 2010.
  • http://www.csuchico.edu/~curbanowicz/DarwinSP2002Phil108.htm"Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows. There is a grandeur in this view of life, with its several powers, having been originally breathed by the Creator into a few forms or into one; and that, whilst this planet has gone cycling on according to the fixed law of gravity, from so simple a beginning endless forms most beautiful and most wonderful have been, and are being, evolved
  • Also in 2010, Twitter announced at Chrip Conference that 75% of its total traffic comes from sources other than Twitter.com. This is part of a growing trend where new, emerging companies gain relevance on mobile and connected platforms other than the web.
  • The Netflix API is a good example of how metrics can influence the design of the API.As Mike talked about earlier, the Netflix API was originally launched to let 1000 flowers bloom.Over time, we started adding our own devices to run off the API.
  • Also in 2010, Twitter announced at Chrip Conference that 75% of its total traffic comes from sources other than Twitter.com. This is part of a growing trend where new, emerging companies gain relevance on mobile and connected platforms other than the web.
  • The Netflix API is a good example of how metrics can influence the design of the API.As Mike talked about earlier, the Netflix API was originally launched to let 1000 flowers bloom.Over time, we started adding our own devices to run off the API.
  • As a result, we are in the process now of redesigning the API with an emphasis towards supporting our devices, while trickling down API features to the 1000 flowers.
  • Another key metric going into our redesign is total requests per month. The growth of the API skyrocketed around October/November. Scale is one of our biggest issues moving forward and continuing on this trajectory will create more challenges.
  • REST is dominating APIs – expanding share of APIs even as the number of APIs is exploding (from just over 1000 in 2008 to over 3000 in 2010).
  • Netflix: Groups of entire movies will be stored together
  • Netflix: The same part of all movies will be stored together – such as frequently-used data about the movies such as actors, director, synopsis, first few minutes of video, etc.
  • Netflix: The same part of groups of movies will be stored together
  • And don’t just design the API based on your current audience. Dream big and design the system for the audience that you want. That doesn’t mean you need to implement the system for the dream audience. It just means your designs should allow for the system to scale to that audience.
  • Similar to the rapid changes we've seen in API-enabled business models and related architectures, we've witnessed four major shifts in how data is shared.Back in the mainframe days that Daniel spoke of, flat file data storage ruled and data sharing was fairly non-existent.  Allowing multiple applications to access the same file was impractical.Databases with concurrency controls through locks made sharing more possible, as you could now more easily let multiple apps manipulate the same data.  However, having applications work directly against data in the format with which it is stored introduced a very tight dependency that could be hard to manage.Middleware introduced another layer that allowed applications to share data via messages to applications, insulating the apps from the actual storage format of the data, effectively making those apps immune to changes in data storage format and schema.Web services improved data sharing even further.  The adoption of the internet-friendly HTTP protocol for exposing data now lets sharing occur beyond the boundaries of your business, while also lowering the technical expertise and software required for consumers of the data.  
  • So technology has evolved to enable unprecedented levels of data sharing.  How can a business successfully navigate the opportunities and threats of this new world?[Honestly assess the cost, benefits and risks of sharing your data]Sharing your data via an API entails a different set of considerations than other forms of APIs.  Why?You lose significant control once data passes out of your API- Unlike non-data APIs, you can't technically restrict access to data once it's been retrieved, although you can restrict access to data in the future- At that point left mostly with legal recourse as defined by any restrictions in your TOU, which is a very blunt enforcement instrument- Enforcement of TOU can be expensive, in terms of your time, money, and reputation with developers, esp. if you seek legal recourse (even if you're right)That being said, it may be time to get over your control issues- Be honest about the value you might get from sharing your data as well as the costs and risks of sharing it
  • Case study: opening up Netflix movie metadata- Value of sharing: others build great movie discovery experiences -> enables Netflix as a fulfillment option- Costs:      - on par with anyother API for allbut    - Risk: competitor uses the data to bootstrap their business     - data islargely public domain or otherwisereproducible with a small to moderate degree of effort     - with controls: can easilycut off a developer'saccess to new data, whichisfairlycritical     - TOU               - enforcelimits and dev account revocation               - nuclear option: legalrecourseCounter example: Hoovers, which has contact information for executives of top companies, their data is their asset
  • [Optimize your data delivery strategies]1. Consider allowing the download of larger datasets, providing an efficient way to get updates, and allowing liberal retention policies to improve availability, scalability and query-ability- Only allowing bite-sized data requests will cause developers to make many more queries which will affect your scalability- Retention restrictions, such as a one-day limit on data retention, will encourage developers to come back often to refresh their data- even if the data hasn't changed- Both result in higher scalability requirements for your API, which costs money- More frequent requests means an increase likelihood you'll break an app dependent on your data, assuming your service is anything less than 100% available- Push or publish incremental updates to consumers so they don't have to download the world regularly2. Consider limited access to richer queries to determine what new APIs to provide
  • Basics:[Netflix service screen shots? Create two slides if so]- Lists (a user's queue, recommendations, watching histories)- Text search on entity names (Movie titles, actor names, etc.)- Not super-flexible, but how customers use the business.  Limits innovation space.[Netflix movie title API output?]Inevitably, esp. for larger and more interesting data sets, developers will want to slice-and-dice the data- creating queries combining data fields to get very specific subsets of large data sets- minimizing data returned in results to optimize performance- writing a query engine is probably not your core competency, but there are a few options available, albeit with some limitations
  • Two emerging web-based query engines can provide rich query semantics for your data without requiring you to make large data sets available for downloadAPI limits can be tough with open query access.  Consider limiting access (OData) or offloading the work (Freebase).Caveat structor: We haven't seen anyone solve this problem at massive scale, however.Once you use platforms like OData or Freebase and find successful and/or common access patterns to your data, make it a new data API.
  • Open standard, backed by MS under their Open Specification PromiseThe describe it as “a Web protocol for querying and updating data that provides a way to unlock your data and free it from silos that exist in applications today”They provide implementation help for enabling OData on your API servers. .NET producer libraries most mature, other libraries (e.g. Java) starting to emergeIn simpler terms, it enables rich query semantics for your data APIHighest rated titles from the 80's and their awards:http://odata.netflix.com/Catalog/Titles?$filter=ReleaseYear le 1989 and ReleaseYear ge 1980 and AverageRating gt 4&$expand=Awards [json]
  • Wikipedia of structured dataYou push your data into their serviceVery robust query language and APICan query across your data as well as other sources of data with common entities (e.g. an actor, a screenplay, etc.)  - Example: Actors that have co-starred in a movie with Kevin Bacon that have also been in a movie with a screenplay adapted from a Steven King novelLicensing of hosted data is creative commonsFreebase is in control of the data, their rules, limits, availability, etc.
  • API limits can be tough with open query access.  Consider limiting access (OData) or offloading the work (Freebase).Caveat structor: We haven't seen anyone solve this problem at massive scale, however.Once you use platforms like OData or Freebase and find successful and/or common access patterns to your data, make it a new data API.
  • While most of our API discussion today has focused on how to open up your business to others, we'd be remiss if we didn't call attention to the value that can be attained by pulling data from other API providers into your business.  Think about whether it would be valuable for your business to understand:
  • 1. Would it be useful for your business to know which of your customers are friends? Facebook friends[clipart of friends]What does that logo mean to you today?  Maybe it's time to think about it differently.
  • 2. What businesses do your customers frequent?  Foursquare and Facebook check-ins[clipart:high scale shop vs. low-rent shop]
  • 3. What is the gender or age of your customers?  Facebook core profile [clipart: old man and young girl]The next wave of data competition is not only about what data you can provide, but what outside data can you integration into your business?
  • http://upload.wikimedia.org/wikipedia/commons/thumb/6/69/Punctuated-equilibrium.svg/1000px-Punctuated-equilibrium.svg.pnghttp://www.talkorigins.org/faqs/punc-eq.html
  • http://digi.ub.uni-heidelberg.de/diglit/cpgraec398/0113
  • OpenTable, MovieTickets, TaxiMagic, Citysearch, Yelp, Yahoo Local, Gayot, Rotten Tomatoes, NYTimes.com, WeatherBug, AllMenus, StubHub, LiveKick, Maponics, Nuance and TrueKnowledge - http://blogs.zdnet.com/semantic-web/?p=333&tag=col1;post-333
  • Get everyone on the same page – you need business metrics, a picture of what success looks like, and executive buy-in and report-outs
  • http://www.avc.com/a_vc/2009/10/business-model-jujutsu.html

Punctuated Equilibrium, Celestial Navigation,and API Strategy Punctuated Equilibrium, Celestial Navigation, and API Strategy Presentation Transcript