WEB HOOKS and the Programmable
World of Tomorrow Jeff Lindsay three years after coining web hooks, people are starting to get excited about them. i’m going to share what they are, why they’re signiﬁcant, and what’s going on in the growing web hooks ecosystem
WEB HOOKS and the Programmable
World of Tomorrow Jeff Lindsay i’ve given a version of this talk before, but this time i’d like to try and focus on this “programmable” idea that helped inspire web hooks.
WARNING: Emergent dynamics described ahead.
was a guy really excited hearing about web hooks, but was disappointed to see what they are. i’m describing a game changer based on a mechanism that takes one line of code to describe. seems so simple, people assume you can only do simple things. quite the contrary...
go. simple mechanisms can create
rich dynamics. easily teach somebody the rules, doesn’t mean they see the implications of those rules. i want to share the implications of this simple mechanism.
The Programmable Web APIs and
Mashups john musser here starts to track apis and mashups in 2005. there are now over 1k apis and increasingly more. apis and mashups became the foundation of the programmable web...
Pro g ramm at ic
The Programmable Web APIs and Mashups but are they? after thinking about web hooks and what they can give us, i realized apis and mashups don’t make a programmable web. they make a programmatic web
Pro g ramm at ic
The Programmable Web Twitter API lets you use Twitter programmatically. It does not let you program Twitter to do more. they let you use web apps programmatically. they don’t let you program them to do more, whereas web hooks can. so i like to argue that web hooks will bring about the *real* programmable web.
a friend of mine got
me to explain web hooks to her. not a programmer, but has iphone. she compared it to jailbreaking the iphone: letting users do what they want, customize, add apps, etc having web hooks is like jailbreaking your web apps. opens functional extensibility.
a pseudo code example of
what the heart of implementing web hooks looks like. SIMPLE: make an HTTP request to a user-speciﬁed URL on major events
both have been around longer.
rest is simpler.. in fact, it’s almost described as “using HTTP properly” but not until it got a name could it be used in discourse to make it popular
REST Hooks rest apis and
web hooks are two sides of the same coin they complement each other in ways i’ll get to later but i just want to give this pattern a name, and start associating some ideas with it
Push Pipes Plugins talk is
split into three sections ways to look at the use of web hooks icons will hopefully make more sense as i talk about them
“One solution that occurred to
me at the time was to build a simple callback system over HTTP. This would fall comfortably between full polling and full persistent publish/subscribe.” he says they fall comfortably between polling and xmpp. i agree, and think they can cover most use-cases
others seem to agree.... this
is a standard for discovering and subscribing to content changes. they let you use web hooks OR xmpp, which is nice. good idea, but (like GENA) standard specs alone don’t get very far
Source Destination Protocol + Mechanism
Protocol + Mechanism Format Format Web Hooks Web Hooks RSS RSS XMPP XMPP Atom Atom XML XML REST REST Publisher Consumer they’re basically an adaptor for data streams, letting you pick your own protocol and mechanism, no matter what the feed publisher is providing.
Source Destination Protocol + Mechanism
Protocol + Mechanism Format Format Web Hooks Web Hooks RSS RSS XMPP XMPP Atom Atom XML XML REST REST Publisher Consumer however, they recently dropped support to consume via xmpp, making web hooks their primary push mechanism for consuming data streams.
Source Protocol + Mechanism Protocol
+ Mechanism Format Format Web Hooks Web Hooks RSS RSS XMPP XMPP Atom Atom XML XML REST REST Publisher Consumer as an example, if i wanted to get digg updates via web hooks, gnip will poll digg for me and invoke my callback with the content as new updates come in.
Source Protocol + Mechanism Protocol
+ Mechanism Format Format Web Hooks Web Hooks RSS RSS XMPP XMPP Atom Atom XML XML REST REST Publisher Consumer alternatively, i could even poll gnip for digg updates. although seemingly redundant, it helps ease the load on digg and allows gnip to provide ﬁltering functionality.
Protocol + Mechanism Protocol +
Mechanism Format Format Web Hooks Web Hooks RSS RSS XMPP XMPP Atom Atom XML XML REST REST Publisher Consumer but another example might be if twitter provides an xmpp stream, but i already have a polling setup... gnip would let me poll instead of integrate xmpp.
microformats in promoting web hooks,
i sort of look up to the way microformats work in values and process. very ground up, grassroots... take existing popular use-patterns and make it a convention. microformats can be viewed as an alternative xml+rdf
xml+rdf vs microformats “Here's a
new language we want you to learn, and now you need to output these additional ﬁles on your server. It's a hassle. (Microformats) lower the barrier to entry.” tantek is a big microformats evangelist. he says.... so i told him about web hooks. “what are they?” “push over http” “how are they di than xmpp?” “they’re a lightweight alternative” lower the barrier to entry...
xmpp vs web hooks “Good.
XMPP needs a competitor.” he says... this was encouraging. i mean, when tantek talks, you listen... if for no other reason than
unfortunately, xmpp has many features
and proposals to do pubsub that web hooks doesn’t.... after all, it’s just a creative use of http requests, not a messaging protocol. so in order to do pubsub with web hooks, you need more to be implemented. well, brad ﬁtzpatrick and brett slatkin are working on that.
Push is good. so the
moral here is that push is good and there’s ﬁnally a demand for it. we have some good solutions waiting adoption...
XMPP is ideal when needed,
but Web Hooks generally do the job. as far as xmpp vs web hooks, i think they both have their place. web hooks are easier, so you might as well default to web hooks unless you really need xmpp.
But push is not the
point. however, as i framed this talk, it’s really not about push. it’s a nice way to get the social media kids into bed with web hooks, but hooks are really about more than pubsub and notiﬁcations. otherwise i would have called them web notiﬁcations or something...
Pipes instead of push, i
was more drawn by the pipes metaphor. i wanted to integrate and orchestrate the web applications i use, conceptually similar to unix pipelining.
so i thought about it.
pipes were a really amazing feature that let you get more out of your commands by letting you chain them together. it sort of let you program without programming, combining commands in ways that weren’t necessarily intended by the author.
Write programs that do one
thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface. this helped put forth the unix philosophy and encouraged building these small composable commands
API Hooks Web App it’s
not that they can’t, they just don’t. we just need to start putting hooks in so they can. those roles are best played by mechanisms that use the protocol the web is built on: http
cat grep mail Basecamp so
we want to combine web applications like we can CLI programs. get more than the sum of the parts. web hooks open up this possibility, but need like APIs, need to be implemented
Todos Basecamp for example, all
these apps share data about todos. they each have respective specialized talents, but all work with todos. by putting hooks on todo CRUD, you can use their apis to keep them synced pretty well. magically. real-time.
“Building projects with web hooks
in mind lets me keep the core Lighthouse source focused, while external services live in their own libraries.” --Rick Olson the idea silently spread to rails guys. rick olson used them in lighthouse
“We implemented web hooks a
while ago and people have been building all sorts of unexpected stuff on top of it.” --Tobias Lütke tobias used them in shopify. i’m told he’s revamping their api to have more hooks. they were one of the earliest adopters and recently had their 1 yr anniversary using web hooks
google code recently caught up
other code hosts by providing a post commit hook. it was very well done, as you’d expect from google and in particular shows that authentication can be done with hmac signatures
they’ve been doing really well
with their post-receive hook. users have used it to integrate with mailing lists, chat, other project management tools, continuous integration, etc
they were so successful with
the adhoc integration, they formalized it. but in the best way! using their existing web hook infrastructure. they just have modules running in a separate but local web service.
in fact, that lets them
open source it. letting people fork, write new handlers, and push back. this is probably going to be the standard model of service integration.
and a great example of
services integrated with github, besides lighthouse, is runcoderun. they run your regression tests for you. continuous integration in the sky. love it. they sign you up automatically if you put their hook in github.
and of course i mentioned
paypal. but i should mention, web hooks make so much sense for paypal... they’re not so much about pushing content, but INTEGRATING. that’s what web hooks are great for, even though they can be used for content push.
jott is another example of
a web hook implementer that doesn’t know it. they parse voice over the phone and do stu with it, like post to twitter, etc
they do it with “Links”...
which are just hooks. they post to a script that does something with the parsed text. really cool for todos.
Jott reads back the The
message is Message is sent via User receives User jotts to response and sends converted HTTP Post to a information back a Jott Link it via SMS SMTP into text web page here’s their diagram. totally web hooks.
it become obvious making adaptor
services would not only be useful in speciﬁc cases, but in general, it makes sense to have services that will turn other protocols into inputs to the web hook ecosystem. the ﬁrst was email... i built mailhook...
in fact a lot of
people made these kinds of “micro webservices” to do simple things GAE didn’t do. it was the ﬁrst glimpse at small, focused services that are like the equivalent of grep, cat, wordcount, etc in the command line piping ecosystem.
then there’s martyn and andy.
two guys in the uk that love web hooks. they built this thing called spaghetti junction at a hackday. it involved into...
switchub. i REALLY love this.
i knew this sort of thing would emerge, but i didn’t think it would happen until web hooks were more popular. kind of like the pastinbin code runner, they let you create hook inputs with urls to put in apps that you can route to various output handlers: email, irc, etc
switchub would beneﬁt from builtin
inputs from various other protocols like email, rss, etc... but instead of having them builtin, they can work with other services to support those kinds of inputs. for example, rssfwd could easily be modiﬁed to provide a web hook for rss
switchub is a lot like
how i visualized a way for regular users to orchestrate web apps. was inspired by reason: virtual rack mounts
“Dabble plugins allow Dabble applications
to create new, derived ﬁelds by calling out to external HTTP-accessible applications. This solves the problem of safely enabling extension of a centrally-located hosted application, in that, while you’re writing code to extend and enhance the behavior of a Dabble application, your code never actually runs inside Dabble.” [General] Name = Amazon Sales Rank [Sales Rank by ISBN] URL = http://chadfowler.com/dabble/amazon_sales_rank.cgi Input = Text Output = Number only they have an extra layer for meta data. but that’s a cool pattern.
“If you’ve used a UNIX-based
operating system, you’re probably familiar with the notion of pipes. The output of one program is piped into the input of another, creating a ﬁlter chain. This is conceptually the same as the way Dabble’s plugin IO works. Nice and simple.” of course, they compare it to pipes. the simplicity. the natural ﬁt of it.
of course, i think they
should have web hooks for all their standard CRUD events... this way their database apps can integrate (like PayPal) with the rest of your workﬂow
in fact, all these “app
platforms” like coghead and salesforce should have web hooks. that would make them more useful, less silo’d o into just processing data in their world
“What is the equivalent of
the pipe in the age of the web?” in 2000, just as rss 0.92 was being released, tim oreilly asks... since we had nothing better, we assumed the answer was feeds.
this eventually gave us yahoo
pipes. which just didn’t seem to change the game like you would think... perhaps feeds just aren’t the answer. or maybe they are and the problem is using the pipes analogy. it’s deﬁnitely a stretch...