The Real Time Web
      (Building It)
Blaine Cook
 Twitter, OAuth, ?
1
Real-Time
 Web?           2
            Problems &
             Solutions        3
                           First Step...
the real time web?
Social Objects

• The things we exchange
• Media: Writing, photos, audio, video, …
• Metadata: Location, relationship data...
problems and
  solutions
What are our goals?

• Real time
• Low cost
• Asynchronous
• Simple
HTTP?

• Works fantastically for web browsers.
• Hard to scale for frequent updates.
• Hard to scale for frequent polling....
HTTP Ping/Push?

• NAT traversal breaks desktop clients.
• HTTP time-outs
• Inconsistent APIs
• Authentication
SMTP?

• No verifiable authentication scheme
• No consistent approach to API design
• Servers not tuned for high volume of
...
Comet?

• GMail
• Successful for web-based clients
• One connection per user
• Requires polling
• Stretching the HTTP meta...
Jabber

• Fulfills all the goals
• Open
• Simple (if youʼre careful)
first steps
architecture

• Not p2p
• Client-to-server
• Clients maintain persistent connections
• Federation looks exactly like email...
itʼs all just xml
• a jabber session is simply two
  streaming XML documents

• the spec defines common elements
• but you ...
jabber addressing

• addresses look like email addresses:
  
   user@domain.com

• but you can omit the username (node):
 ...
jabber federation
• i.e., why itʼs not spammy
• s2s - server to server
• support for verified authentication of
  servers u...
messages
• primary jabber payload. email.
• the simplest message is an addressed
  stanza with a body (the message)

• sub...
presence

• online, offline, chat, away, xa
• the intellectual pivot between optimizing
  for store and forward (http, smtp...
tracking presence


• we can subscribe and unsubscribe
• also allow, deny, or block requests
the roster

• your social connections
• maintains presence subscriptions
• maintains your whitelist
• synonomous with your...
jabber basics
navigating jabber

• Nearly 200 specs defining all sorts of
  behaviour. Ignore them.

• Unless you need them.
• Core & IM:...
stanzas

• XML Elements
• Shared attributes:
 • to, from, id, type
messages

<message from=quot;romeo@montague.netquot;
             to=quot;juliet@capulet.comquot;>


  <body>Hi!</body>
</...
messages

<message from=quot;romeo@montague.net/orchardquot;
             to=quot;juliet@capulet.comquot;
             id=...
presence

<presence from=quot;romeo@montague.netquot;
         to=quot;juliet@capulet.comquot; />
presence

<presence from=quot;romeo@montague.netquot;
          to=quot;juliet@capulet.comquot;>
  <show>away</show>
  <st...
presence


<presence from=quot;romeo@montague.netquot;
         to=quot;juliet@capulet.comquot;
         type=quot;subscri...
presence


<presence from=quot;juliet@capulet.comquot;
         to=quot;romeo@montague.netquot;
         type=quot;subscri...
presence
<presence from=quot;romeo@montague.netquot;
         to=quot;juliet@capulet.comquot;
         type=quot;unsubscri...
iq

• Information Query
• Enables the roster, discovery, XEPs
• Should almost always be hidden behind
  libraries.
building applications
taking stock

• we can send/receive messages
• add / remove contacts
• track presence
• let's build something!
define bot behaviour

• what does your bot do?
• conversational
• informational
• recorder
define api behaviour

• what does your api look like?
• atom?
• custom xml with namespaces?
• we'll dig in a bit more later.
write the behaviour

• build a class or interface that handles
  messages

• test the class with mock xmpp stanzas
• mock ...
behaviour
class MyHandler
 def on_message(message)
      puts quot;Got Message:quot;
      puts quot;from #{message.from}q...
event handler
client = Jabber::Simple.new('user@ex.com', 'pwd')
handler = MyHandler.new


 client.received_messages do |me...
event loop
client = Jabber::Simple.new('user@ex.com', 'pwd')
handler = MyHandler.new
loop do
  client.received_messages do...
handling presence

  client.status(:away, quot;eatingquot;)

client.presence_updates do |update|
  friend = update[0]
  pr...
handling presence

<presence from=quot;user@ex.comquot;>
  <show>away</show>
  <status>eating</status>
</presence>
rosters


• should ideally be handled by libraries
• if not, at least aim for being able to fetch
  your roster using a li...
rosters

Roster roster = connection.getRoster();
Collection<RosterEntry> entries = roster.getEntries();
for (RosterEntry e...
process management

• Very difficult to run Jabber clients from
  non-persistent connections

• Run a persistent daemon tha...
next steps
PubSub

• A mechanism for Publishing and
  Subscribing to feeds

• Like presence subscriptions, but for
  data
PubSub

• Over-specified
• Don't try to read the spec if you can
  avoid it

• Thankfully, the concept is simple and
  the ...
PubSub Subscribe
<iq type='set' from='francisco@denmark.lit/barracks'
   to='example.com' id='sub1'>
  <pubsub xmlns='http...
PubSub Confirmation
<iq type='result' from='example.com'
    to='francisco@denmark.lit/barracks' id='sub1'>
  <pubsub xmlns...
PubSub Messages
<message from='example.com'
  to='francisco@denmark.lit/barracks' id='foo'>
  <body>blah</body>
  <event x...
PubSub Unsubscribe
<iq type='set' from='francisco@denmark.lit/barracks'
   to='example.com' id='unsub1'>
  <pubsub xmlns='...
PubSub Confirmation

<iq type='result'
    from='example.com'
    to='francisco@denmark.lit/barracks'
    id='unsub1'/>
PEP

• Personal Eventing via PubSub
• You can think of it as exactly the same
  as regular PubSub, except the node
  becom...
PEP
<iq type='set' from='francisco@denmark.lit/barracks'
   to='user@example.com' id='sub1'>
  <pubsub xmlns='http://jabbe...
Federation

• Social Network Federation
• Use PubSub to allow users on remote
  services to subscribe to eachother

• Brea...
best practices
• keeping the api simple
• choosing where to use jabber
• atom over xmpp
scaling techniques
scalability?

• Jabber scales well out of the box for
  relatively small numbers of contacts.

• Stops working at around 3...
components

• In order to work around this, we use the
  component protocol, XEP-0114

• Horrendously bad documentation
• ...
components

• A component allows you to handle
  everything for a JID, or a whole domain

• You can turn off the roster!
•...
components

• Components work just like client-to-
  server bots, but we need to handle
  presence ourselves.

• The easie...
components
client = Jabber::Component.new('example.com')
client.connect(quot;127.0.0.1quot;)
client.auth(quot;secretquot;)...
horizontal scaling

• Many processes across machines
• Need a queuing mechanism
• We use Starling
• ActiveMQ, RabbitMQ, My...
horizontal scaling
client.add_message_callback do |message|
  incoming_message_queue.push message
end


loop do
  message ...
client connections

• If you plan to offer Jabber user
  accounts, you'll need to scale to many
  persistent connections.
...
tools
Client Libraries

• Ruby: xmpp4r & xmpp4r-simple
• Java: Smack
• Python: twisted-words
• Perl: Net::Jabber
• Javascript: J...
Jabber Servers

• ejabberd (recently 2.0)
• openfire
• Jabber XCP
Other Tools


• Debugging: Psi (cross-platform, fully
  featured)

• PubSub: Idavoll
Jabber-enabled
• livejournal
• twitter
• jaiku
• gtalk / gmail
• chesspark
• fire eagle (soon!)
questions?
Building The Real Time Web Presentation
Building The Real Time Web Presentation
Upcoming SlideShare
Loading in...5
×

Building The Real Time Web Presentation

1,368

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,368
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
42
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Building The Real Time Web Presentation

  1. 1. The Real Time Web (Building It)
  2. 2. Blaine Cook Twitter, OAuth, ?
  3. 3. 1 Real-Time Web? 2 Problems & Solutions 3 First Steps Jabber 4 Basics Building 5 Applications 6 Next Steps 7 Best Practices 8 Scaling Techniques 9 Jabber Tools
  4. 4. the real time web?
  5. 5. Social Objects • The things we exchange • Media: Writing, photos, audio, video, … • Metadata: Location, relationship data, personal data
  6. 6. problems and solutions
  7. 7. What are our goals? • Real time • Low cost • Asynchronous • Simple
  8. 8. HTTP? • Works fantastically for web browsers. • Hard to scale for frequent updates. • Hard to scale for frequent polling. • Asks the wrong question: “what happened in the past?”
  9. 9. HTTP Ping/Push? • NAT traversal breaks desktop clients. • HTTP time-outs • Inconsistent APIs • Authentication
  10. 10. SMTP? • No verifiable authentication scheme • No consistent approach to API design • Servers not tuned for high volume of low-latency messages
  11. 11. Comet? • GMail • Successful for web-based clients • One connection per user • Requires polling • Stretching the HTTP metaphor
  12. 12. Jabber • Fulfills all the goals • Open • Simple (if youʼre careful)
  13. 13. first steps
  14. 14. architecture • Not p2p • Client-to-server • Clients maintain persistent connections • Federation looks exactly like email • Servers communicate directly
  15. 15. itʼs all just xml • a jabber session is simply two streaming XML documents • the spec defines common elements • but you can extend it at any time • similar to html, but with a larger vocabulary
  16. 16. jabber addressing • addresses look like email addresses: user@domain.com • but you can omit the username (node): domain.com • or you can include a resource: user@domain.com/mydevice
  17. 17. jabber federation • i.e., why itʼs not spammy • s2s - server to server • support for verified authentication of servers using SSL • dialback authentication • explicit whitelist by default
  18. 18. messages • primary jabber payload. email. • the simplest message is an addressed stanza with a body (the message) • subject, alternate message types are available but client ui is poorly implemented • we can send html and atom, too
  19. 19. presence • online, offline, chat, away, xa • the intellectual pivot between optimizing for store and forward (http, smtp) and streams of data
  20. 20. tracking presence • we can subscribe and unsubscribe • also allow, deny, or block requests
  21. 21. the roster • your social connections • maintains presence subscriptions • maintains your whitelist • synonomous with your buddy list on MSN / AIM / YahooIM
  22. 22. jabber basics
  23. 23. navigating jabber • Nearly 200 specs defining all sorts of behaviour. Ignore them. • Unless you need them. • Core & IM: RFCs 3920 & 3921
  24. 24. stanzas • XML Elements • Shared attributes: • to, from, id, type
  25. 25. messages <message from=quot;romeo@montague.netquot; to=quot;juliet@capulet.comquot;> <body>Hi!</body> </message>
  26. 26. messages <message from=quot;romeo@montague.net/orchardquot; to=quot;juliet@capulet.comquot; id=quot;msg:montague.net,1quot;> <body>Hi!</body> </message>
  27. 27. presence <presence from=quot;romeo@montague.netquot; to=quot;juliet@capulet.comquot; />
  28. 28. presence <presence from=quot;romeo@montague.netquot; to=quot;juliet@capulet.comquot;> <show>away</show> <status>swooning</status> </presence>
  29. 29. presence <presence from=quot;romeo@montague.netquot; to=quot;juliet@capulet.comquot; type=quot;subscribequot; />
  30. 30. presence <presence from=quot;juliet@capulet.comquot; to=quot;romeo@montague.netquot; type=quot;subscribedquot; />
  31. 31. presence <presence from=quot;romeo@montague.netquot; to=quot;juliet@capulet.comquot; type=quot;unsubscribequot; /> <presence from=quot;juliet@capulet.comquot; to=quot;romeo@montague.netquot; type=quot;unsubscribedquot; />
  32. 32. iq • Information Query • Enables the roster, discovery, XEPs • Should almost always be hidden behind libraries.
  33. 33. building applications
  34. 34. taking stock • we can send/receive messages • add / remove contacts • track presence • let's build something!
  35. 35. define bot behaviour • what does your bot do? • conversational • informational • recorder
  36. 36. define api behaviour • what does your api look like? • atom? • custom xml with namespaces? • we'll dig in a bit more later.
  37. 37. write the behaviour • build a class or interface that handles messages • test the class with mock xmpp stanzas • mock out sending functions in your xmpp lib so you don't need an active connection
  38. 38. behaviour class MyHandler def on_message(message) puts quot;Got Message:quot; puts quot;from #{message.from}quot; puts quot;to #{message.to}quot; puts quot;body #{message.body}quot; out = Jabber::Message.new(message.from, quot;got it!quot;) yield out end end
  39. 39. event handler client = Jabber::Simple.new('user@ex.com', 'pwd') handler = MyHandler.new client.received_messages do |message| handler.on_message(message) do |out| client.send(out) end end
  40. 40. event loop client = Jabber::Simple.new('user@ex.com', 'pwd') handler = MyHandler.new loop do client.received_messages do |message| handler.on_message(message) do |out| client.send(out) end end end
  41. 41. handling presence client.status(:away, quot;eatingquot;) client.presence_updates do |update| friend = update[0] presence = update[2] puts quot;#{friend.jid} is #{presence.status}quot; end
  42. 42. handling presence <presence from=quot;user@ex.comquot;> <show>away</show> <status>eating</status> </presence>
  43. 43. rosters • should ideally be handled by libraries • if not, at least aim for being able to fetch your roster using a library call
  44. 44. rosters Roster roster = connection.getRoster(); Collection<RosterEntry> entries = roster.getEntries(); for (RosterEntry entry : entries) { System.out.println(entry); }
  45. 45. process management • Very difficult to run Jabber clients from non-persistent connections • Run a persistent daemon that manages your Jabber connection
  46. 46. next steps
  47. 47. PubSub • A mechanism for Publishing and Subscribing to feeds • Like presence subscriptions, but for data
  48. 48. PubSub • Over-specified • Don't try to read the spec if you can avoid it • Thankfully, the concept is simple and the core implementation is easy
  49. 49. PubSub Subscribe <iq type='set' from='francisco@denmark.lit/barracks' to='example.com' id='sub1'> <pubsub xmlns='http://jabber.org/protocol/pubsub'> <subscribe node='http://example.com/updates' jid='francisco@denmark.lit/barracks'/> </pubsub> </iq>
  50. 50. PubSub Confirmation <iq type='result' from='example.com' to='francisco@denmark.lit/barracks' id='sub1'> <pubsub xmlns='http://jabber.org/protocol/pubsub'> <subscription node='http://example.com/updates' jid='francisco@denmark.lit/barracks' subscription='subscribed'/> </pubsub> </iq>
  51. 51. PubSub Messages <message from='example.com' to='francisco@denmark.lit/barracks' id='foo'> <body>blah</body> <event xmlns='http://jabber.org/protocol/pubsub#event'> <items node='http://twitter.com/xmpp'> <item id='http://twitter.com/blaine/statuses/324236243'> <entry>...</entry> </item> </items> </event> </message>
  52. 52. PubSub Unsubscribe <iq type='set' from='francisco@denmark.lit/barracks' to='example.com' id='unsub1'> <pubsub xmlns='http://jabber.org/protocol/pubsub'> <unsubscribe node='http://example.com/xmpp' jid='francisco@denmark.lit'/> </pubsub> </iq>
  53. 53. PubSub Confirmation <iq type='result' from='example.com' to='francisco@denmark.lit/barracks' id='unsub1'/>
  54. 54. PEP • Personal Eventing via PubSub • You can think of it as exactly the same as regular PubSub, except the node becomes relative to a user (full JID)
  55. 55. PEP <iq type='set' from='francisco@denmark.lit/barracks' to='user@example.com' id='sub1'> <pubsub xmlns='http://jabber.org/protocol/pubsub'> <subscribe node='http://example.com/updates' jid='francisco@denmark.lit/barracks'/> </pubsub> </iq>
  56. 56. Federation • Social Network Federation • Use PubSub to allow users on remote services to subscribe to eachother • Breaking down walled gardens
  57. 57. best practices
  58. 58. • keeping the api simple
  59. 59. • choosing where to use jabber
  60. 60. • atom over xmpp
  61. 61. scaling techniques
  62. 62. scalability? • Jabber scales well out of the box for relatively small numbers of contacts. • Stops working at around 35k contacts, due to roster presence behaviour. • Come online, find out what everyone's presence is.
  63. 63. components • In order to work around this, we use the component protocol, XEP-0114 • Horrendously bad documentation • But thankfully it's simple
  64. 64. components • A component allows you to handle everything for a JID, or a whole domain • You can turn off the roster! • Without roster management, we now assume that out bot is always online.
  65. 65. components • Components work just like client-to- server bots, but we need to handle presence ourselves. • The easiest way is to do the following…
  66. 66. components client = Jabber::Component.new('example.com') client.connect(quot;127.0.0.1quot;) client.auth(quot;secretquot;) client.add_presence_callback do |presence| case presence.type.to_s when nil, 'unavailable': save_presence(presence) when 'probe': send_online(presence.from) when 'subscribe': send_subscribed(presence.from) end end
  67. 67. horizontal scaling • Many processes across machines • Need a queuing mechanism • We use Starling • ActiveMQ, RabbitMQ, MySQL, local HTTP push are also viable options
  68. 68. horizontal scaling client.add_message_callback do |message| incoming_message_queue.push message end loop do message = message_queue.pop client.send message end
  69. 69. client connections • If you plan to offer Jabber user accounts, you'll need to scale to many persistent connections. • Thankfully, most Jabber servers do this part out of the box.
  70. 70. tools
  71. 71. Client Libraries • Ruby: xmpp4r & xmpp4r-simple • Java: Smack • Python: twisted-words • Perl: Net::Jabber • Javascript: JSJaC
  72. 72. Jabber Servers • ejabberd (recently 2.0) • openfire • Jabber XCP
  73. 73. Other Tools • Debugging: Psi (cross-platform, fully featured) • PubSub: Idavoll
  74. 74. Jabber-enabled • livejournal • twitter • jaiku • gtalk / gmail • chesspark • fire eagle (soon!)
  75. 75. questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×