Darwin's theory of punctuated equilibrium proposes that evolution occurs in bursts of rapid change separated by long periods of stability, rather than through gradual change. The document discusses how this applies to business models and APIs, which also experience periods of rapid, revolutionary change punctuated by longer periods of equilibrium. It provides examples of how business models, architectures, and data sharing have evolved in a punctuated manner over time for companies like Netflix.
27. G. Scandens C. Pallidus G. Magnirostris G. Conirostris C. Heliobates G. Fortis G. Fulginosa C. Psittacula C. Pauper C. Olivacea C. Parvulus Certhidea Camarhynchus Geospiza
28. “Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows.” Charles Darwin On Origin of Species
29. So while it may look slow and gradual in hindsight
100. App Developer Device App Big Box Retailer Mobile App National Chain Web Retail Department Store Web Catalog Specialty Store Indirect Sales Direct Sales
102. Hardt’s Theorem: The Internet Power Law The first wave of the Internet demonstrated the economic impact of web-based business models. 99:1 95:5 The API-driven Internet is demonstrating the next concentration of power and is reflecting a “99:1” distribution (examples: Twitter, Facebook) due to the high switching costs and effective lock-in through software. The HTML-driven Internet showed new business models that focused revenue in the 3rd standard deviation (examples: Amazon, EBay). Reality reflected a “95:5” rule where 5% of companies dominated the transactions and profits. 80:20 Previous eras of business showed a normal distribution for revenue, with most firms getting most of the revenue. In the later half of the 20th century, business model innovations focused revenue in the 2nd standard deviation above the mean. The “80:20” rule became conventional wisdom. The next wave of the Internet is demonstrating the economic impact of API-based business models.
103. But you need to tackle it in a way that fits your business
176. Controller Model View iPhone Wrapper Recommendations R12n Service Xbox Wrapper UNIFIED LIST/TITLE API Wii Wrapper User Info User Service PS3 Wrapper Similar Movies Similar Movie Service SHARED API INTERFACES Roku Wrapper Movie Metadata DataNormalization & Resiliency iPad Wrapper USER API Viewing History Apple TV Wrapper Ratings PC / Mac Wrapper TiVo Wrapper API Repository Layer Wrapper Layer Web Service Layer Shared Layer Source Data Layer App Layer API Layer
177. Stable Flexible Agile iPhone Wrapper Recommendations R12n Service Xbox Wrapper UNIFIED LIST/TITLE API Wii Wrapper User Info User Service PS3 Wrapper Similar Movies Similar Movie Service SHARED API INTERFACES Roku Wrapper Movie Metadata DataNormalization & Resiliency iPad Wrapper USER API Viewing History Apple TV Wrapper Ratings PC / Mac Wrapper TiVo Wrapper API Repository Layer Wrapper Layer Web Service Layer Shared Layer Source Data Layer App Layer API Layer
179. Instance Architecture Based on Specialization List Creation Dependency Service ELASTIC INSTANCE LAYER API METADATA CACHING LAYER Meta Data Dependency Service METADATA SERVICE
180. Instance Architecture Based on Specialization Handles Request/Response List Creation Dependency Service ELASTIC INSTANCE LAYER Caches Dependency Data API METADATA CACHING LAYER Meta Data Dependency Service Populates and Manages Cache METADATA SERVICE
181. Map out your usage patterns and cache your data accordingly
205. Extend your API by extending data types Addition is not version-worthy Better to be incomplete than inaccurate Withhold implementation if you are unsure
215. Request-based metrics such as what endpoints were called and what parameters were passed can show you what aspects of your API are popular. Response-based metrics such as what was delivered, how quickly, and whether the response was valid can show you what aspects of your API need work.
216. System trace metrics that track what underlying systems were called and how they responded can show you where you need to evolve your internal architecture. Business performance metrics such as how much revenue or how much customer engagement is occurring through the API show you how close you are to delivering on the business plan.
230. Netflix found that the cost was the same as any API, the risks could be managed through rate limits and access control, but that now others could build great movie discovery experiences that led to increased Netflix viewing hours.
231. Sharing data as a service means making a few course corrections on your API
232. enable larger downloads for fewer queries more liberal retention policies means less API traffic, higher performance, and less cost push incremental updates limited access to richer queries
238. Last time the right solution was a query language Perhaps it’s the right solution now
239. Instead of a static REST call, we could pose a query like “what are the highest rated movies from the 1980s?” http://odata.netflix.com/Catalog/Titles?$filter=ReleaseYear le 1989and ReleaseYearge1980and AverageRatinggt4 &$expand=Awards
240. Caveat structor does apply This is new ground and we haven’t seen anyone do this at massive scale
241. Second, people want just the right parts of your data for their particular need But they need the right parts of other people’s data to have a complete context
244. Last time the right solution was middleware for distributed queries Perhaps it’s the right solution now
245. But the rules have changed a bit since the data is laying all over the Internet
246. [{ "id": null, "name": null, "type": "/dining/restaurant", "/business/business_location/address": [{ "street_address": [], "citytown": { "id": "/en/toronto" } }], "cuisine": [{ "/dining/cuisine/region_of_origin": [{ "!/film/film/featured_film_locations": [{ "id": "/en/the_italian_job" }] }] Instead of a single domain query, we could ask for a list of “Toronto restaurants with cuisine from a filming location of ‘The Italian Job’”
250. What could you offer your customers if you knew which of them were friends with each other?
251.
252. What does that logo mean to you today? Perhaps it’s time to think differently. Facebook is a massive data API that lets you correlate your customers with their true context
253. You could move beyond a flat-earth view where all your customers are their own islands of data And each of them were connected in ways that makes your business more valuable to them
254. What could you offer your customers if you knew where they like to go?
255.
256. The more of your customer’s context that you can understand The more time you can save them And that makes your business more valuable to them
257. The good news is that there are already data APIs to get this context Now you need to focus on sharing in
http://upload.wikimedia.org/wikipedia/commons/thumb/6/69/Punctuated-equilibrium.svg/1000px-Punctuated-equilibrium.svg.pnghttp://www.talkorigins.org/faqs/punc-eq.htmlThere are two models for the flow of evolution, each with its adherents. Phyletic gradualism says that things change slowly over time, day by day, until one day you realize there has been enough change that something new has arisen.The opposing model is Punctuated Equilibrium. This model says that species tend to remain in a homeostatic mode of stable equilibrium – but that every so often some significant new stress affects the environment, prompting rapid, dramatic changes that settle down into equilibrium again.For our purposes, punctuated equilibrium serves as a powerful model for viewing the changes in technology, business models, and data over the last few decades and shows us that the rapid, dramatic changes part of the cycle is coming again.
Adapted from http://www.csuchico.edu/~curbanowicz/DarwinSP2002Phil108.htm
Cherokee-class 10-gun brig-sloop of the Royal Navy downgraded to a 6-gun barquehttp://www.theora.com/msdb.php/OriginOfSpecies/orderBy/subject/msdbCat/subject/12%20-%20Geographical%20Distribution%20--%20continuedExploration - like evolutionary change - only looks smooth in hindsight. Not even the voyage of Darwin’s HMS Beagle was smoothly planned. Looking back he seems to have been predestined to arrive at the Galapagos islands, but this is not the case.
http://darwin-online.org.uk/people/chancellor.htmlAt some points the trip got pretty rough.
http://thebeagleproject.blogspot.com/2008_07_01_archive.htmlAnd the voyage itself was anything but linear.
Credit to Dick Hardt who formulated this in December 2010.
In 2010, Twitter announced at Chirp Conference that 75% of its total traffic comes from sources other than Twitter.com. This is part of a growing trend where new, emerging companies gain relevance on mobile and connected platforms other than the web.
Over time, we started adding our own devices to run off the API. They drove far more traffic than we saw from outside developers.
As a result, we are in the process now of redesigning the API with an emphasis towards supporting our devices, while trickling down API features to the 1000 flowers.
Another key metric going into our redesign is total requests per month. The growth of the API skyrocketed around October/November. Scale is one of our biggest issues moving forward and continuing on this trajectory will create more challenges.
REST is dominating APIs – expanding share of APIs even as the number of APIs is exploding (from just over 1000 in 2008 to over 3000 in 2010).
Netflix: Groups of entire movies will be stored together
Netflix: The same part of all movies will be stored together – such as frequently-used data about the movies such as actors, director, synopsis, first few minutes of video, etc.
Netflix: The same part of groups of movies will be stored together according to geography
Similar to the rapid changes we've seen in API-enabled business models and related architectures, we've witnessed four major shifts in how data is shared.Back in the mainframe days that Daniel spoke of, flat file data storage ruled and data sharing was fairly non-existent. Allowing multiple applications to access the same file was impractical.Databases with concurrency controls through locks made sharing more possible, as you could now more easily let multiple apps manipulate the same data. However, having applications work directly against data in the format with which it is stored introduced a very tight dependency that could be hard to manage.Middleware introduced another layer that allowed applications to share data via messages to applications, insulating the apps from the actual storage format of the data, effectively making those apps immune to changes in data storage format and schema.Web services improved data sharing even further. The adoption of the internet-friendly HTTP protocol for exposing data now lets sharing occur beyond the boundaries of your business, while also lowering the technical expertise and software required for consumers of the data.
Netflix: Groups of entire movies will be stored together
[Optimize your data delivery strategies]1. Consider allowing the download of larger datasets, providing an efficient way to get updates, and allowing liberal retention policies to improve availability, scalability and query-ability- Only allowing bite-sized data requests will cause developers to make many more queries which will affect your scalability- Retention restrictions, such as a one-day limit on data retention, will encourage developers to come back often to refresh their data- even if the data hasn't changed- Both result in higher scalability requirements for your API, which costs money- More frequent requests means an increase likelihood you'll break an app dependent on your data, assuming your service is anything less than 100% available- Push or publish incremental updates to consumers so they don't have to download the world regularly2. Consider limited access to richer queries to determine what new APIs to provide
Caveat structor – developer beware
Similar to the rapid changes we've seen in API-enabled business models and related architectures, we've witnessed four major shifts in how data is shared.Back in the mainframe days that Daniel spoke of, flat file data storage ruled and data sharing was fairly non-existent. Allowing multiple applications to access the same file was impractical.Databases with concurrency controls through locks made sharing more possible, as you could now more easily let multiple apps manipulate the same data. However, having applications work directly against data in the format with which it is stored introduced a very tight dependency that could be hard to manage.Middleware introduced another layer that allowed applications to share data via messages to applications, insulating the apps from the actual storage format of the data, effectively making those apps immune to changes in data storage format and schema.Web services improved data sharing even further. The adoption of the internet-friendly HTTP protocol for exposing data now lets sharing occur beyond the boundaries of your business, while also lowering the technical expertise and software required for consumers of the data.