Darwin's theory of punctuated equilibrium proposes that evolution occurs in bursts of rapid change separated by long periods of stability, rather than through gradual change. The document discusses how this applies to business models and APIs, which also experience periods of rapid, revolutionary change punctuated by longer periods of equilibrium. It provides examples of how business models, architectures, and data sharing have evolved in a punctuated manner over time for companies like Netflix.

1. Punctuated Equilibrium,Celestial Navigation,and APIs Competing through dynamic adaptation Sam Ramji, Apigee @sramji Dan Jacobson, Netflix @daniel_jacobson Michael Hart, Netflix @michaelhart

2. PUNCTUATED EQUILIBRIUM

3. Darwin formulated his theory of evolution about 150 years ago

4. Based on observations he made in the Galapagos Islands 15 years earlier

5. A wild diversity of creatures existed in a new environment

7. Starting from an ancestor which looked like this

8. Geospiza Fulginosa

9. Finches evolved that looked like this

10. Geospiza Fortis

11. and this

12. Camarhynchus Pallidus

13. and this

14. Camarhynchus Pauper

15. and this

16. Geospiza Conirostris

17. and this

18. Certhidea Olivacea

19. and this

20. Geospiza Scandens

21. For many years the belief was that this change happened slowly and gradually.

22. In 1972, Stephen Jay Gould and Niles Eldredge proposed a new idea

23. that evolution is not slow and gradual

24. but sudden and violent.

26. Applying this view to the observations of finches

27. G. Scandens C. Pallidus G. Magnirostris G. Conirostris C. Heliobates G. Fortis G. Fulginosa C. Psittacula C. Pauper C. Olivacea C. Parvulus Certhidea Camarhynchus Geospiza

28. “Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows.” Charles Darwin On Origin of Species

29. So while it may look slow and gradual in hindsight

30. Evolution is experienced in punctuated bursts.

31. If you’re living in a punctuated burst of evolution it feels like a revolution

32. CELESTIAL NAVIGATION

33. Exploration

35. like evolutionary change

36. only looks smooth in hindsight

38. Living through it is usually chaotic

39. Karen James The Beagle Project Blog

40. To navigate, you need a map and instruments

42. Maps exist for transferring knowledge

43. and they too have evolved over time.

44. They started as oral traditionand were written down in a form called a periplus

45. Periplus of Hanno Courtesy of Heidelberg University

46. Periplus of Hanno Courtesy of Cornell University

47. Far less efficient knowledge transfer than a modern map of the same journey

48. Map of Hanno’s Journey Courtesy of Bourrichon/Wikipedia

49. Exploration was dramatically held back for want of a map

50. In the two thousand years between Hanno’s journey on a Phoenician trireme

52. And the Mediterranean caravel of the 15th century

54. Maps had only evolved to be graphical descriptions of coastlines

56. That was a map published a few years before Columbus crossed the Atlantic to find India

57. After his crossing, his expedition shared their knowledge in a new map

59. Still far from perfectbut much improved.

60. The biggest challenge in this kind of exploration was determining their location on the Earth

61. Instruments for measuring latitude had beenused and improved for centuries

63. Longitude was the hard problem.

64. You needed to know not just the angle of the sun and stars

66. you also needed to know the precise time.

68. Regardless of your sailing technology without the proper measurement you were lost

69. We are not promising a perfect map of the new world

70. But it should be more like this

72. than this

73. Periplus of Hanno Courtesy of Heidelberg University

74. and we will show you what we know how to measure.

75. APIS

76. There are more niches today than we’ve seen before, so we need to borrow from nature

78. If we start with an API we can explore all the niches around our business

79. Visualization by Apigee

80. The leaders of today’s Internet

82. clearly understand this mechanism.

83. They understand that the distribution model for value has changed in the Internet era.

84. Packaged Goods Consumer Retail Store Producer Internet Services App Developer Consumer Provider

85. Developers took their APIs and explored the niches for them

87. The providers and the developers both benefited from this adaptation

90. Suddenly this seems obvious to everyone.

91. Data from Programmable Web

92. And developers are racing to pack the niches.

93. Data from Wikipedia

94. This is a sudden, material shift in competition.

96. It only looks gradual if you’re losing.

98. BUSINESS MODELS From 1st Party to Partner to Platform

99. We’ve seen punctuated equilibrium in business models over the last hundred years

100. App Developer Device App Big Box Retailer Mobile App National Chain Web Retail Department Store Web Catalog Specialty Store Indirect Sales Direct Sales

101. What’s the environmental stress driving the current rapid change?

102. Hardt’s Theorem: The Internet Power Law The first wave of the Internet demonstrated the economic impact of web-based business models. 99:1 95:5 The API-driven Internet is demonstrating the next concentration of power and is reflecting a “99:1” distribution (examples: Twitter, Facebook) due to the high switching costs and effective lock-in through software. The HTML-driven Internet showed new business models that focused revenue in the 3rd standard deviation (examples: Amazon, EBay). Reality reflected a “95:5” rule where 5% of companies dominated the transactions and profits. 80:20 Previous eras of business showed a normal distribution for revenue, with most firms getting most of the revenue. In the later half of the 20th century, business model innovations focused revenue in the 2nd standard deviation above the mean. The “80:20” rule became conventional wisdom. The next wave of the Internet is demonstrating the economic impact of API-based business models.

103. But you need to tackle it in a way that fits your business

104. 1st Party Open Open Open Platform Partner

105. These are complementary and distinct. Open is different for each one.

106. 1st Party Apps Partner Apps Platform Ecosystem

107. 1st party 1st party is about offering direct access to your core business via apps that you make or contract out

108. 1st party Here open means all the business is accessible to internal developers and contract specialists

109. partner Partner is about enabling directed development of apps that extend your business model towards your business partners

110. partner Here open means existing partners have access to your business via APIs and can innovate asynchronously

111. platform Platform is about enabling unknown developers to build brand new apps and businesses that will surprise and inform you

112. platform Here open means enabling business models and allowing developers to support each other at massive scale

113. open Open is attractive

115. open Open is Biz Dev 2.0

116. 1st Party Open Open Open Platform Partner

117. open Open lets you navigate across the possible business models when your first model doesn’t work as planned

118. 1st Party Open Open Open Platform Partner

119. To get your API strategy properly grounded

120. “ John Musser Programmable Web

121. But how?

122. Let’s break it down

123. Establish Target Segments Engage Developer Channel Set Industry Goal

124. target segments An API should extend your core business into a new part of the market

125. target segments Your core business already has key performance indicators

126. target segments So apply your KPIs to the new market segment you’re targeting with your APIs

127. target segments What is the market impact you need to create in order to succeed as a business?

128. target segments What does the targetsegment need that it is not getting from you today?

129. target segments The answer will be the foundation of your API strategy.

130. developer channel In most cases the channel for your API will be developers, but what do they need?

131. developer channel A profit motive.

132. developer channel Here are the leading profit models for developers today

133. developer channel In-app purchases Affiliate royalty Your advertising spend Market awareness of their offering App sales

134. industry goal If you don’t know where you’re going, you definitely won’t get there

135. industry goal Partnerships and platform businesses are very different things.

136. industry goal Partnerships are formed to serve a known set of entities

137. industry goal A partner API should be traceable to each partner’s relationship

138. industry goal And support end-to-end business processes

139. industry goal A platform exists to create massive and unpredictable opportunities

140. industry goal All your technology, support, and community decisions will be about surviving the scale of adoption

141. That’s the strategy dimension.

142. The execution dimension is what you already know.

143. Planning. Management. Organization.

144. Putting all this in context gives us a map for our API strategy

146. The instruments will be your KPIs and your core API metrics: performance and adoption

147. ARCHITECTURE From Mainframe to Mobile

148. Connected Devices Smartphone Web App N-tier Personal Computer DCOM CORBA Website Minicomputer Client/Server Mainframe Integrated Distributed Computing

149. “ The Web is Dead. Long Live the Internet. Chris Anderson Wired Magazine

150. The Web is Dead. Long Live APIs!

151. Twitter traffic distribution shows what he means

152. Twitter Traffic in 2010

153. Netflix traffic distribution is nearly the same.

154. The majority of Netflix traffic comes from API-drivenconnected devices.

155. Like Columbus, Netflix started with a map of the coastline

157. Build an open API as a platform and let a thousand flowers bloom

159. But they had left the coastline far behind

160. And the instruments indicated that there were fewer flowers than expected

161. Netflix API Requests by Segment

162. But partners started building apps for connected devices and the business took off

163. Wii iPad PS3 iPhone Apple TV LG TVs Samsung TVs Roku XBox

164. Architecture should reflect the business model

165. So Netflix has drawn the following map

166. Wii Roku PS3 iPhone iPad Google TV Apple TV XBox LG TVs Samsung TVs

167. Instruments show that API traffic has grown tremendously in a short time

168. Growth of Netflix API

169. 20,000,000,000 API requests per month. Is that a cause for celebration? Or for concern?

170. When you’re navigating uncharted waters, speed is not your friend. Perhaps it’s time to slow down and avoid risking unknown reefs.

171. Navigating this growth challenge for Netflix means that the next API revision will focus on reducing overall traffic.

172. Part of this redesign is reviewing conventions

173. Punctuated Equilibrium: REST Data sourced from ProgrammableWeb

174. REST seems obvious but assess what makes sense for your business.

175. Tiered architecture helps you navigate different problems with agility

176. Controller Model View iPhone Wrapper Recommendations R12n Service Xbox Wrapper UNIFIED LIST/TITLE API Wii Wrapper User Info User Service PS3 Wrapper Similar Movies Similar Movie Service SHARED API INTERFACES Roku Wrapper Movie Metadata DataNormalization & Resiliency iPad Wrapper USER API Viewing History Apple TV Wrapper Ratings PC / Mac Wrapper TiVo Wrapper API Repository Layer Wrapper Layer Web Service Layer Shared Layer Source Data Layer App Layer API Layer

177. Stable Flexible Agile iPhone Wrapper Recommendations R12n Service Xbox Wrapper UNIFIED LIST/TITLE API Wii Wrapper User Info User Service PS3 Wrapper Similar Movies Similar Movie Service SHARED API INTERFACES Roku Wrapper Movie Metadata DataNormalization & Resiliency iPad Wrapper USER API Viewing History Apple TV Wrapper Ratings PC / Mac Wrapper TiVo Wrapper API Repository Layer Wrapper Layer Web Service Layer Shared Layer Source Data Layer App Layer API Layer

178. Server architecture should support both crests and troughs of the waves of demand

179. Instance Architecture Based on Specialization List Creation Dependency Service ELASTIC INSTANCE LAYER API METADATA CACHING LAYER Meta Data Dependency Service METADATA SERVICE

180. Instance Architecture Based on Specialization Handles Request/Response List Creation Dependency Service ELASTIC INSTANCE LAYER Caches Dependency Data API METADATA CACHING LAYER Meta Data Dependency Service Populates and Manages Cache METADATA SERVICE

181. Map out your usage patterns and cache your data accordingly

182. Vertical Caching

183. Vertical Caching: Netflix Full Movie Data

184. Horizontal Caching

185. Horizontal Caching: Netflix Basic Data

186. Combining horizontal and vertical caching may be the best approach when building for multiple geographies

187. Two-Dimensional Caching

188. Design for where you’re going

190. Not for where you are

192. You may be starting here

193. Growth of Netflix API

194. But you must design for here

195. Growth of Netflix API

196. You don’t need to implementfor massive scale

197. But you must design for it or you will follow your successful ocean crossing with a massive shipwreck.

198. Navigation also means constantly adjusting your course to ensure you arrive at your final goal

199. Sometimes adjusting course on an API means you must change your version

200. Versioning means supporting multiple applications, all of which basically do the same thing

201. Versioning 1.0 1.5 2.0 3.0? 4.0? 5.0? Today

202. If possible, go versionless

203. Version-less API 1.0 1.5 2.0 Here there be dragons versionless Today

204. Rules for going versionless in your production APIs

205. Extend your API by extending data types Addition is not version-worthy Better to be incomplete than inaccurate Withhold implementation if you are unsure

206. With APIs emerging, we need better tools help us navigate

208. The service level agreement is your sextant

209. Set reasonable service level agreements

210. What is reasonable will vary by API and use case but you must communicate them to your users.

211. Measure average latencies, error rates and types, and respond when SLAs are broken

213. Having visibility into performance means you can tack immediatelyrather than after your users threaten to mutiny.

214. Longer-term navigation requires higher-level metrics

215. Request-based metrics such as what endpoints were called and what parameters were passed can show you what aspects of your API are popular. Response-based metrics such as what was delivered, how quickly, and whether the response was valid can show you what aspects of your API need work.

216. System trace metrics that track what underlying systems were called and how they responded can show you where you need to evolve your internal architecture. Business performance metrics such as how much revenue or how much customer engagement is occurring through the API show you how close you are to delivering on the business plan.

217. DATA From Silos to Social

218. Domain-specific Data APIs Private Cloud DBs Caching DBs Data API Data Warehousing Mainframe RDBMS Flat file Shared Silos Data

219. Punctuated equilibrium in data sharing

220. App Org Cross-org Cross-business Mainframe Databases Middleware APIs

221. So we are evolving to cross-business sharing

222. There are a few challengesin making this journey

223. Sharing your data via an API entails a different set of considerations than APIs which expose your services

224. loss of control only recourse is legal enforcement is expensive

225. That said, it may be time to get over your control issues

226. Be honest about the value you could get from sharing your data outside your corporation

227. instead of just the costs and risks of sharing it.

228. In opening up its movie data warehouse

230. Netflix found that the cost was the same as any API, the risks could be managed through rate limits and access control, but that now others could build great movie discovery experiences that led to increased Netflix viewing hours.

231. Sharing data as a service means making a few course corrections on your API

232. enable larger downloads for fewer queries more liberal retention policies means less API traffic, higher performance, and less cost push incremental updates limited access to richer queries

233. Looking forward, how are we going to work through this wave of shared data?

234. Two different dimensions are apparent

235. First, no one wants all of your data, just some of it. Just the right parts of it for their particular need.

236. Where have we seen this before?

237. App Org Cross-org Cross-business Mainframe Databases Middleware APIs

238. Last time the right solution was a query language Perhaps it’s the right solution now

239. Instead of a static REST call, we could pose a query like “what are the highest rated movies from the 1980s?” http://odata.netflix.com/Catalog/Titles?$filter=ReleaseYear le 1989and ReleaseYearge1980and AverageRatinggt4 &$expand=Awards

240. Caveat structor does apply This is new ground and we haven’t seen anyone do this at massive scale

241. Second, people want just the right parts of your data for their particular need But they need the right parts of other people’s data to have a complete context

242. Where have we seen this before?

243. App Org Cross-org Cross-business Mainframe Databases Middleware APIs

244. Last time the right solution was middleware for distributed queries Perhaps it’s the right solution now

245. But the rules have changed a bit since the data is laying all over the Internet

246. [{ "id": null, "name": null, "type": "/dining/restaurant", "/business/business_location/address": [{ "street_address": [], "citytown": { "id": "/en/toronto" } }], "cuisine": [{ "/dining/cuisine/region_of_origin": [{ "!/film/film/featured_film_locations": [{ "id": "/en/the_italian_job" }] }] Instead of a single domain query, we could ask for a list of “Toronto restaurants with cuisine from a filming location of ‘The Italian Job’”

247. Caveat structor still applies. This is very new terrain indeed.

248. The bigger question than what should you be sharing out Is what should you be sharing in

249. What data APIs should your business be using?

250. What could you offer your customers if you knew which of them were friends with each other?

252. What does that logo mean to you today? Perhaps it’s time to think differently. Facebook is a massive data API that lets you correlate your customers with their true context

253. You could move beyond a flat-earth view where all your customers are their own islands of data And each of them were connected in ways that makes your business more valuable to them

254. What could you offer your customers if you knew where they like to go?

256. The more of your customer’s context that you can understand The more time you can save them And that makes your business more valuable to them

257. The good news is that there are already data APIs to get this context Now you need to focus on sharing in

258. InCLOSING

259. We are going through a period of rapid change in business models, architecture, and data

261. Navigating based strictly on the stories of others

262. Periplus of Hanno Courtesy of Heidelberg University

263. Will not give you the clear map that you need

265. Develop instruments around your API that help you understand where you’re going

267. so that you can correct your course and beat your competition in the race for the future.

268. THANK YOU Questions and ideas to: @michaelhart @daniel_jacobson @sramji