Web Hooks Google Tech Talk


Published on

I gave a modified version of my talk at Google.

Published in: Technology
1 Comment
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Web Hooks Google Tech Talk

  1. 1. WEB HOOKS and the Programmable World of Tomorrow Jeff Lindsay three years after coining web hooks, people are starting to get excited about them. i’m going to share what they are, why they’re significant, and what’s going on in the growing web hooks ecosystem
  2. 2. WEB HOOKS and the Programmable World of Tomorrow Jeff Lindsay i’ve given a version of this talk before, but this time i’d like to try and focus on this “programmable” idea that helped inspire web hooks.
  3. 3. WARNING: Emergent dynamics described ahead. was a guy really excited hearing about web hooks, but was disappointed to see what they are. i’m describing a game changer based on a mechanism that takes one line of code to describe. seems so simple, people assume you can only do simple things. quite the contrary...
  4. 4. go. simple mechanisms can create rich dynamics. easily teach somebody the rules, doesn’t mean they see the implications of those rules. i want to share the implications of this simple mechanism.
  5. 5. in 1997, jon udell talks about websites as data sources that can be reused and remixed today that idea isn’t very novel
  6. 6. “a new programming paradigm that takes the whole Internet as its platform” he envisions the Internet as a programming paradigm
  7. 7. The Programmable Web “The Web as Platform” it starts to manifest as the programmable web and talk of “the web as a platform”
  8. 8. The Programmable Web APIs and Mashups john musser here starts to track apis and mashups in 2005. there are now over 1k apis and increasingly more. apis and mashups became the foundation of the programmable web...
  9. 9. Pro g ramm at ic The Programmable Web APIs and Mashups but are they? after thinking about web hooks and what they can give us, i realized apis and mashups don’t make a programmable web. they make a programmatic web
  10. 10. Pro g ramm at ic The Programmable Web Twitter API lets you use Twitter programmatically. It does not let you program Twitter to do more. they let you use web apps programmatically. they don’t let you program them to do more, whereas web hooks can. so i like to argue that web hooks will bring about the *real* programmable web.
  11. 11. Web Service interwebz User so let’s get specific. here’s an example scenario showing the use of web hooks.
  12. 12. Web Service Hi, I’m Twickr, a new web service. interwebz User
  13. 13. “Twickr” Cool. Sign me up. interwebz User
  14. 14. “Twickr” Okay, you’re all set. Now listen... interwebz User
  15. 15. “Twickr” I might find myself with some data you might be interested in. interwebz User
  16. 16. “Twickr” Alight, hold on a sec... interwebz User
  17. 17. “Twickr” interwebz User hack hackety hack
  18. 18. “Twickr” Could you just post it here when you have any? http://tinyurl.com/6pln4u interwebz User
  19. 19. “Twickr” ...sure, why not. Hmm, what is that URL? interwebz User
  20. 20. “Twickr” It’s a tiny handler I put up on App Engine. interwebz User
  21. 21. “Twickr” What does it do? interwebz User
  22. 22. “Twickr” It doesn’t matter to you. Whatever I want... interwebz User web hooks are just user-defined http callbacks
  23. 23. Web Service EXTENSIBLE interwebz EMPOWERED User but do you see what they did? they made the web service extensible and empowered the user.
  24. 24. a friend of mine got me to explain web hooks to her. not a programmer, but has iphone. she compared it to jailbreaking the iphone: letting users do what they want, customize, add apps, etc having web hooks is like jailbreaking your web apps. opens functional extensibility.
  25. 25. a pseudo code example of what the heart of implementing web hooks looks like. SIMPLE: make an HTTP request to a user-specified URL on major events
  26. 26. paypal represents on the oldest web hook implementors
  27. 27. their instant payment notification is a web hook
  28. 28. “When a customer pays you, PayPal posts a notification to your server at a URL you specify.” its framed as a notification, but that doesn’t properly imply the usage its intended for: integration.
  29. 29. started thinking about this in 2006. everything flashed before my eyes and was very confused why it wasn’t used more. felt like i was taking crazy pills. today i know why...
  30. 30. compare to REST.
  31. 31. both have been around longer. rest is simpler.. in fact, it’s almost described as “using HTTP properly” but not until it got a name could it be used in discourse to make it popular
  32. 32. REST Hooks rest apis and web hooks are two sides of the same coin they complement each other in ways i’ll get to later but i just want to give this pattern a name, and start associating some ideas with it
  33. 33. Push Pipes Plugins talk is split into three sections ways to look at the use of web hooks icons will hopefully make more sense as i talk about them
  34. 34. Push let’s get started with push people are starting to talk about push and pubsub on the web... although its not the first time
  35. 35. 1998 (predating rss) microsoft submitted an internet draft to extend http to provide a basic pubsub framework called GENA
  36. 36. it had your standard pubsub terms
  37. 37. and added 4 new methods to http.
  38. 38. it expired a year later in 1999 and the group behind it seemed to disappear. however...
  39. 39. Sneaky! the methods and protocol ended up in the webdav implemented for exchange
  40. 40. but then in 2000, rss 0.92 was released. five years later it gets an icon and widespread adoption. it started with blog feeds, then comment feeds...
  41. 41. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds even feeds of feeds
  42. 42. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds even feeds of feeds
  43. 43. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds even feeds of feeds
  44. 44. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds even feeds of feeds
  45. 45. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds even feeds of feeds
  46. 46. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds even feeds of feeds
  47. 47. then soon exploded into twitter feeds, photo feeds, activity feeds, event feeds, bookmark feeds even feeds of feeds
  48. 48. it makes you think of feeds like in the telecom world
  49. 49. data coming directly to you
  50. 50. but we know that’s not how it works
  51. 51. ? instead we have to go request the data
  52. 52. then it gives it to us. and we do this over and over. are we there yet? are we there yet?
  53. 53. ? feeds made sense in a world where feed readers ran on desktops that couldn’t be pushed to over http
  54. 54. ? of course, now we have other web applications consuming feeds and it doesn’t make sense. even most of our feed readers have become web applications
  55. 55. SUP is a recent incremental solution to increase the eficiency of consuming large amounts of feeds. however, its based on polling and is essentially yet another feed, so i couldn’t help...
  57. 57. evan and kellan gave a great talk a while back about this push issue, and their proposed solution: xmpp i have a condensed version of this talk. the slides speak for themselves
  58. 58. (aka XMPP) they have a great point. polling sucks, and xmpp is a pretty good solution for data streams
  59. 59. but it’s kind of heavy weight. it does a lot and makes a decently complex little system. luckily it’s not *that* hard to use with today’s library support
  60. 60. evan and kellan did point out there are extremes when it comes to data streams. most data streams will probably fall somewhere in between, but i do think xmpp is suited for the fast and furious end
  61. 61. joshua schachter of delicious responded to this talk in a blog post. he basically suggests web hooks as an alternative
  62. 62. “One solution that occurred to me at the time was to build a simple callback system over HTTP. This would fall comfortably between full polling and full persistent publish/subscribe.” he says they fall comfortably between polling and xmpp. i agree, and think they can cover most use-cases
  63. 63. others seem to agree.... this is a standard for discovering and subscribing to content changes. they let you use web hooks OR xmpp, which is nice. good idea, but (like GENA) standard specs alone don’t get very far
  64. 64. gnip is a service that some may have heard of (but don’t understand)
  65. 65. Polling their main goal is to stop polling as a service in the middle.
  66. 66. Source Destination Protocol + Mechanism Protocol + Mechanism Format Format Web Hooks Web Hooks RSS RSS XMPP XMPP Atom Atom XML XML REST REST Publisher Consumer they’re basically an adaptor for data streams, letting you pick your own protocol and mechanism, no matter what the feed publisher is providing.
  67. 67. Source Destination Protocol + Mechanism Protocol + Mechanism Format Format Web Hooks Web Hooks RSS RSS XMPP XMPP Atom Atom XML XML REST REST Publisher Consumer however, they recently dropped support to consume via xmpp, making web hooks their primary push mechanism for consuming data streams.
  68. 68. Source Protocol + Mechanism Protocol + Mechanism Format Format Web Hooks Web Hooks RSS RSS XMPP XMPP Atom Atom XML XML REST REST Publisher Consumer as an example, if i wanted to get digg updates via web hooks, gnip will poll digg for me and invoke my callback with the content as new updates come in.
  69. 69. Source Protocol + Mechanism Protocol + Mechanism Format Format Web Hooks Web Hooks RSS RSS XMPP XMPP Atom Atom XML XML REST REST Publisher Consumer alternatively, i could even poll gnip for digg updates. although seemingly redundant, it helps ease the load on digg and allows gnip to provide filtering functionality.
  70. 70. Protocol + Mechanism Protocol + Mechanism Format Format Web Hooks Web Hooks RSS RSS XMPP XMPP Atom Atom XML XML REST REST Publisher Consumer but another example might be if twitter provides an xmpp stream, but i already have a polling setup... gnip would let me poll instead of integrate xmpp.
  71. 71. microformats in promoting web hooks, i sort of look up to the way microformats work in values and process. very ground up, grassroots... take existing popular use-patterns and make it a convention. microformats can be viewed as an alternative xml+rdf
  72. 72. xml+rdf vs microformats “Here's a new language we want you to learn, and now you need to output these additional files on your server. It's a hassle. (Microformats) lower the barrier to entry.” tantek is a big microformats evangelist. he says.... so i told him about web hooks. “what are they?” “push over http” “how are they di than xmpp?” “they’re a lightweight alternative” lower the barrier to entry...
  73. 73. xmpp vs web hooks “Good. XMPP needs a competitor.” he says... this was encouraging. i mean, when tantek talks, you listen... if for no other reason than
  74. 74. he gets the chicks
  75. 75. unfortunately, xmpp has many features and proposals to do pubsub that web hooks doesn’t.... after all, it’s just a creative use of http requests, not a messaging protocol. so in order to do pubsub with web hooks, you need more to be implemented. well, brad fitzpatrick and brett slatkin are working on that.
  76. 76. Push is good. so the moral here is that push is good and there’s finally a demand for it. we have some good solutions waiting adoption...
  77. 77. XMPP is ideal when needed, but Web Hooks generally do the job. as far as xmpp vs web hooks, i think they both have their place. web hooks are easier, so you might as well default to web hooks unless you really need xmpp.
  78. 78. But push is not the point. however, as i framed this talk, it’s really not about push. it’s a nice way to get the social media kids into bed with web hooks, but hooks are really about more than pubsub and notifications. otherwise i would have called them web notifications or something...
  79. 79. Pipes instead of push, i was more drawn by the pipes metaphor. i wanted to integrate and orchestrate the web applications i use, conceptually similar to unix pipelining.
  80. 80. so i thought about it. pipes were a really amazing feature that let you get more out of your commands by letting you chain them together. it sort of let you program without programming, combining commands in ways that weren’t necessarily intended by the author.
  81. 81. Input Output Program all from a bit of infrastructure involving input and output
  82. 82. STDIN STDOUT Program STDERR stdin, stdout were available to reroute wherever the user wanted most common use was chaining commands together: piping
  83. 83. xargs wget echo mail grep wc cat so you had all these simple little programs, that might not even be useful alone
  84. 84. cat xargs wget echo mail grep wc string them together...
  85. 85. cat grep xargs wget echo mail wc
  86. 86. cat grep mail xargs wget echo wc and you have something more useful than the sum of the parts
  87. 87. Write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface. this helped put forth the unix philosophy and encouraged building these small composable commands
  88. 88. STDIN Program but it doesn’t work without the output. it just breaks.
  89. 89. API Web App unfortunately that’s how the web is today. we can talk to web apps, but they really can’t talk to us. or anything else really.
  90. 90. API Hooks Web App it’s not that they can’t, they just don’t. we just need to start putting hooks in so they can. those roles are best played by mechanisms that use the protocol the web is built on: http
  91. 91. backhoe
  92. 92. + front-loader + excavator
  93. 93. cat grep mail Basecamp so we want to combine web applications like we can CLI programs. get more than the sum of the parts. web hooks open up this possibility, but need like APIs, need to be implemented
  94. 94. Basecamp imagine basecamp with a bunch of hooks for events
  95. 95. Basecamp Project finished Todo completed Milestone created Contact added File uploaded
  96. 96. Basecamp My handler http://example.com/handler users can write handlers that are just web scripts. they have a url, and thats what you give basecamp
  97. 97. Basecamp My handler http://example.com/handler it’s code. it can do anything from there. integrate with other services, make a phone call, order pizza, whatever
  98. 98. Todos Basecamp for example, all these apps share data about todos. they each have respective specialized talents, but all work with todos. by putting hooks on todo CRUD, you can use their apis to keep them synced pretty well. magically. real-time.
  99. 99. Service integration web hooks enable service integration
  100. 100. Composability adding composability to web apps
  101. 101. Code as glue based on the idea that web urls can run code. and code can do anything.
  102. 102. when i first thought about this, cheap PHP hosting was all over, but it could be even simpler. there are paste bins like this one. made to share formatted code with people over IRC or email
  103. 103. put in some code you want to share or ask a question about
  104. 104. http://pastie.org/84826 hit save and you get a URL you can share when you ask “where’s the data stored?”
  105. 105. i’m thinking of the same thing, but that you write handlers with
  106. 106. http://pastie.org/run/24576 and saving doesn’t just give a url to view, but to run. pass that url into an app with web hooks
  107. 107. Basecamp Project finished http://pastie.org/run/24576 for example, basecamp. now when you finish a project, everybody meets for shots in the break room.
  108. 108. fortunately, we have this. it’s called appjet by ex googlers.
  109. 109. just hit a button, write code, hit save, share the url. it’s javascript
  110. 110. obviously app engine, although it’s a little more involved than appjet for quick handlers. but it is an option for python. and there are ruby/rails hosts like heroku
  111. 111. one thing i’ve been working on is an extension to play with ways end-users might interact with web hooks
  112. 112. Hey, there’s an event hook here! by detecting some markup in a page, it discovers hooks. like say for new photos from contacts. you want to do something when that happens, click it
  113. 113. Save and write some code. hit save, it posts to AppJet (or wherever), registers the handler (assuming a standard protocol), and done. all inline. go back and change the code.
  114. 114. Real world examples but these are all mockups and what-ifs... there is a world of web hooks already evolving...
  115. 115. i started by exposing svn hooks as web hooks in devjavu
  116. 116. i talked about web hooks enough using pbwiki as an example, their mysterious cto decided to implement them
  117. 117. and apparently really liked what he found
  118. 118. went all out on hooked events. not sure if it’s made it to production, but really cool
  119. 119. “Building projects with web hooks in mind lets me keep the core Lighthouse source focused, while external services live in their own libraries.” --Rick Olson the idea silently spread to rails guys. rick olson used them in lighthouse
  120. 120. “We implemented web hooks a while ago and people have been building all sorts of unexpected stuff on top of it.” --Tobias Lütke tobias used them in shopify. i’m told he’s revamping their api to have more hooks. they were one of the earliest adopters and recently had their 1 yr anniversary using web hooks
  121. 121. google code recently caught up other code hosts by providing a post commit hook. it was very well done, as you’d expect from google and in particular shows that authentication can be done with hmac signatures
  122. 122. github was one of the first majorly popular site to use and promote web hooks.
  123. 123. they’ve been doing really well with their post-receive hook. users have used it to integrate with mailing lists, chat, other project management tools, continuous integration, etc
  124. 124. they were so successful with the adhoc integration, they formalized it. but in the best way! using their existing web hook infrastructure. they just have modules running in a separate but local web service.
  125. 125. in fact, that lets them open source it. letting people fork, write new handlers, and push back. this is probably going to be the standard model of service integration.
  126. 126. and a great example of services integrated with github, besides lighthouse, is runcoderun. they run your regression tests for you. continuous integration in the sky. love it. they sign you up automatically if you put their hook in github.
  127. 127. and of course i mentioned paypal. but i should mention, web hooks make so much sense for paypal... they’re not so much about pushing content, but INTEGRATING. that’s what web hooks are great for, even though they can be used for content push.
  128. 128. jott is another example of a web hook implementer that doesn’t know it. they parse voice over the phone and do stu with it, like post to twitter, etc
  129. 129. they do it with “Links”... which are just hooks. they post to a script that does something with the parsed text. really cool for todos.
  130. 130. Jott reads back the The message is Message is sent via User receives User jotts to response and sends converted HTTP Post to a information back a Jott Link it via SMS SMTP into text web page here’s their diagram. totally web hooks.
  131. 131. it become obvious making adaptor services would not only be useful in specific cases, but in general, it makes sense to have services that will turn other protocols into inputs to the web hook ecosystem. the first was email... i built mailhook...
  132. 132. GAE community made one because GAE doesn’t have a way to accept email (but will soon). web hooks were the obvious solution.
  133. 133. rick olson has an open source non-hosted ruby version that will do xmpp. he uses it for lighthouse.
  134. 134. but smtp2web is interesting because it was made because of the limitations of GAE...
  135. 135. in fact a lot of people made these kinds of “micro webservices” to do simple things GAE didn’t do. it was the first glimpse at small, focused services that are like the equivalent of grep, cat, wordcount, etc in the command line piping ecosystem.
  136. 136. then there’s martyn and andy. two guys in the uk that love web hooks. they built this thing called spaghetti junction at a hackday. it involved into...
  137. 137. switchub. i REALLY love this. i knew this sort of thing would emerge, but i didn’t think it would happen until web hooks were more popular. kind of like the pastinbin code runner, they let you create hook inputs with urls to put in apps that you can route to various output handlers: email, irc, etc
  138. 138. my example switchboard. this kind of feels like gnip, only more focused and more about web hooks. so i like it lots.
  139. 139. opening handlers up like github. anybody can write handlers soon. working with them a little to make it real awesome.
  140. 140. switchub would benefit from builtin inputs from various other protocols like email, rss, etc... but instead of having them builtin, they can work with other services to support those kinds of inputs. for example, rssfwd could easily be modified to provide a web hook for rss
  141. 141. switchub is a lot like how i visualized a way for regular users to orchestrate web apps. was inspired by reason: virtual rack mounts
  142. 142. flip it around and wire them together however you like. totally cool.
  143. 143. Plugins there’s one more thing i want to talk about.
  144. 144. web was a bunch of pages linked together.
  145. 145. but people kept wanting to access other things through the web (like me today)
  146. 146. so they developed cgi
  147. 147. tangent: this is a neat find. was on reddit. andreessen proposing IMG tag. people fought it, said it needed to be more generalized. he just put it on mosaic and that was that
  148. 148. so these pages... today... because of cgi
  149. 149. are actually the result
  150. 150. of these higher order nodes... the code... the apps. this is most of the web today. neat how that evolved
  151. 151. so you have these applications
  152. 152. and the dierence you might think is backend and frontend. but there is a subtly dierent way to think about it
  153. 153. Application Pages logic and content. the read-write web gave us user contributed content... i want user contributed (democratized) logic.
  154. 154. Function Content logic and content. the read-write web gave us user contributed content... i want user contributed (democratized) logic.
  155. 155. Integrate Function Content not just integrating and composing apps
  156. 156. Extend Integrate Function Content but extending apps...
  157. 157. almost called this section Platforms. platforms are really cool. we all love them. i LOVE them, so fb platform was really cool. asked a friend how it worked. he said “web hooks”
  158. 158. sure enough, this looks like web hooks to me. as long as it’s http, calling out... but then using the results in their app? thats dierent...
  159. 159. in fact a few people have used web hooks for plugins. dabble is a great example.
  160. 160. they do online databased for people that use excel as a database.
  161. 161. their plugin API is great. it uses web hooks!
  162. 162. “Dabble plugins allow Dabble applications to create new, derived fields by calling out to external HTTP-accessible applications. This solves the problem of safely enabling extension of a centrally-located hosted application, in that, while you’re writing code to extend and enhance the behavior of a Dabble application, your code never actually runs inside Dabble.” [General] Name = Amazon Sales Rank [Sales Rank by ISBN] URL = http://chadfowler.com/dabble/amazon_sales_rank.cgi Input = Text Output = Number only they have an extra layer for meta data. but that’s a cool pattern.
  163. 163. “If you’ve used a UNIX-based operating system, you’re probably familiar with the notion of pipes. The output of one program is piped into the input of another, creating a filter chain. This is conceptually the same as the way Dabble’s plugin IO works. Nice and simple.” of course, they compare it to pipes. the simplicity. the natural fit of it.
  164. 164. of course, i think they should have web hooks for all their standard CRUD events... this way their database apps can integrate (like PayPal) with the rest of your workflow
  165. 165. in fact, all these “app platforms” like coghead and salesforce should have web hooks. that would make them more useful, less silo’d o into just processing data in their world
  166. 166. IMified uses web hooks. sells the tech too: “allows anyone with basic web programming skills to quickly and easily create ...”
  167. 167. General Systems Theory close by dropping some GST on you
  168. 168. General Systems Theory central tenet is value is not in the elements or parts of a system
  169. 169. General Systems Theory the real value is in the interactions, how they work together. this creates the emergent phenomenon of a system, and defines its behavior
  170. 170. Degrees of freedom increase the ways to work together, the means of combination, increases the DoF
  171. 171. Increase possibility space uncover new opportunities in business, tools, and empowerment
  172. 172. “a new programming paradigm that takes the whole Internet as its platform” this vision for a programming paradigm that IS the internet/web is very compelling... but we’re not there yet.
  173. 173. The Programmable Web APIs and Hooks the real programmable web will require
  174. 174. http://flickr.com/photos/nikonfans/439432049/ http://flickr.com/photos/omegaman/477866054/ http://flickr.com/photos/aquan/2780542698/ http://flickr.com/photos/lucky_13/191038503/ http://flickr.com/photos/narwhal85/1290912011/ http://flickr.com/photos/thomashawk/521532598/ http://flickr.com/photos/bendodson/2601002535/ http://flickr.com/photos/ntr23/218966763/ http://flickr.com/photos/shrimphead/2259353414/ http://flickr.com/photos/raphaelmuli/2913907316/ http://flickr.com/photos/x180/467764284/ http://flickr.com/photos/x180/1704475856/ Thanks! webhooks.org
  175. 175. “What is the equivalent of the pipe in the age of the web?” in 2000, just as rss 0.92 was being released, tim oreilly asks... since we had nothing better, we assumed the answer was feeds.
  176. 176. this eventually gave us yahoo pipes. which just didn’t seem to change the game like you would think... perhaps feeds just aren’t the answer. or maybe they are and the problem is using the pipes analogy. it’s definitely a stretch...
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.