Ocean
XMPP Based
Realtime Communication
Framework Suite
!
加藤 亮 2012/03

Realtime
• Data (micro contents)
• Delivery Method ( Push )
• UI

PUSH Friendly
Contents
• Notification
• Activity Stream
• Message, Chat
• Presence

PUSH
• Long Polling
• XMPP
• HTML5 (SSE, WebSocekt)
• Smart phone (APNS, C2DM)

UI
• Ajaxify - auto view updating

Ocean

Framework Suite
• Front Server
• Cluster
• HTTP Binding
• P2P(Audio/Video chat) 予定

Protocol
•
•
XMPP Based
•
XMPPのサブセットをサポート
XMPPと内部で互換性を持たせた数種類の独自
プロトコル
•
•
External WebSocket Binding
HTTP Binding ( WebSocket/SSE/XHR )

Jabber/XMPP

examples
• google talk
• facebook chat
• PSN
• iMessage
• google wave ( S2S )

Stream Procedure
• Stream Initialization
• TLS Negotiation
• Authentication
• Resource Binding
• Session Establishment
• Roster Request
• Initial Presence

Stanza
• Feature
• Auth
• Message
• Presence
• IQ

ID
• JID (example: lyokato@mixi.jp/resource )
• Full JID (lyokato@mixi.jp/resource)
• Bare JID (lyokato@mixi.jp)

Authentication
• SASL
• DIGEST-MD5
• PLAIN ( under TLS )
• EXTERNAL ( for S2S TLS )
• We need
• OAuth 2.0?
• Cookie/Ticket ( for WebSocket )

Message

Message
• service notification (type=headline)
• chat message (type=chat)
• group chat message (type=groupchat)
• normal ? message (type=message)

Presence

Presence
• presence
• subscription
• Ocean doesn’t support subscription
• extension
• join/leave a room

Presence
• show (chat, away, dnd, xa )
• status (custom message)
• priority

Presence traps
• protocol design
• presence:connection = 1:1
• de facto usage
• presence:user = 1:1

Presence De Facto Usage
• one presence for one user
• we don’t need priority
• ‘chat’ and ‘away’ are enough for ‘show’
param

Resource Binding
• node@domain/resource
• you can choose ‘home’ or ‘work’
• de facto usage
• users and client softwares don’t mind
what the resource is.
• so server generates random string (like a
cookie )

Session Establishment
• Historical Reason
• Do nothing

IQ
• Roster ( Friend List )
• vCard ( Profile )
• bind, session, ping, disco, etc...
• packets should be back to sender

Roster & vCard
• like OpenSocial’s ‘People Data API’

Roster
• there isn’t profile image, which is included
in vCard
• no pagination control by default

vCard traps
• just for profile image
• but base64 encoded binary
• we want URL

Subscription & Roster Push
• Friend Request
• Relationship State Control

C2S & S2S
• Client to Server
• Server to Server
• Ocean doesn’t support S2S

C2S Example
gmail
C2S Stream
User A
C2S Stream
User B

S2S
Example
S2S Stream
chat.facebook.com
gmail.com
User A
User B
User C
User D

S2S Problems
• We have to manage new database that
represents ‘relationships’ between users of
our service and ones of other service.
• We have to manage a white list or spam
control. ( like OpenID )
• Authentication Compatibility Cost
• Scaling Cost
• Error Handling

Implementation

Other Implementations
• I/O Multiplexing
• Node.js
• Apache MINA, naggati
• Jabber/XMPP software
• ejebberd
• DJabberd
• Openfire

Ocean’s XMPP support
• subset
• customized
• fixed features(not pluggable )
• assumed cluster installation

Why subset?
• Ocean is just for Web-based Social
Network Service
• Create/Modify friendships on the web
site.
• S2S costs too much
• Developers have to write only minimum
number of event handlers.

Protocol Customized
• resource binding & roster/vCard responses
support extra-attributes.
• Because of regardless of HTTP
federation, especially URL
• WebSocket binding (experimental )
• try demo application
• Simple-MUC

Not flexible and pluggable?
• It may cause complication. requires detailed
protocol knowledge, framework
knowledge.
• When the servers are distributed, frontserver-plugins require complicated spec.
See ‘Cluster’ section.
• There are de-facto-standard specs.

Why full scratch?
• Proprietary Reason
• We may need non-standard features
• Rapid Updating (community based
development requires many arguments)
• Architectural Dependencies
• No Money

Inside Ocean

Getting Started
• All you need to do is ...
• generate project template
• edit configuration file
• prepare your handler class

Generate Project Template
•
•
./ocean
cd Foobar

Configuration
• vi conf/ocean.yml

Write Handler Class
• inherit Ocean::Handler
• implement all event handler methods

Ocean Components
Daemonizer
Signal Handler
Handler
Listener
Server
Stream
Stream Builder
Stream Manager
Stream
Stream

Inside
Stream
Handler
Server
Stream Manager
Stream Components
Protocol States
IO Components
SASL
Encoder
Events
Socket
Decoder
Client
Active
...

Problem?
• Bad performance
• too many call stack on each event
• each component consume memory

Benefit
• Testability
• You can switch to stub components
• Customizability
• You can switch Encoder/Decoder

Pay attention to
memory usage
• 4byte on a Stream object
• 4x10,000byte on a Server witch has 10,000
connection
• bless Array instead of Hash
• use Config instance as singleton object
• use delegate object for event handling

Handler
Handler
Connection Info
User A
User A
User B
User C
Home
Work
Home
Home
chat
away
chat
chat
Users
Relationships
Databases
Profile Pictures
Files
Your Service

Handler uses
2 or 3 kinds of store
• Persistent Data Store (users, relationships)
• Half Persistent Data Store. Same lifecycle as
XMPP service (connection map)
• Cache system ( if needed )

Handler
Implementation

Handler Example
• modules under Ocean::Standalone::Handler
namespace are good example

Server Events
• on_node_init
• on_node_exit

Stream Events
• on_sasl_auth_request

Stream Events
• on_bind_request

Stream Events
• on_message

Stream Events
• on_presence
• on_initial_presence
• on_unavailable_presence

Presence doesn’t scale
• user_001 login-ed ( send initial-presence )
• get all friends of user_001
• get all available-resources of the friends
• send user_001’s presence to all the
available-resources.
• send presences of the all availableresources to user_001
It’s important to limit
the maximum number of friends.

Stream Events
• on_silent_disconnection

Stream Events
• on_oster_request

Stream Events
• on_vcard_request

Stream Events
• on_too_many_auth_attempt

Start & Stop
• bin/ocean-start
• bin/ocean-start --daemonize
• bin/ocean-stop ( if daemonized )

Cluster
• C10K Problem on Persistent Connection
• Blocking IO multiplexing
• Message delivery/Presence broadcast

Message Delivery
Connection Map
User A
User A
User B
User C
Home
Work
Home
Home
1
2
1
2
Delivery Server
XMPP Server 2
XMPP Server 1
User A
(Home)
User B
(Home)
User A
(Work)
User C
(Home)

Use 2 kinds of gearman
• Background Job Dispatcher
• Inbox ( 1 inbox for each front-server )

Message Delivery
Connection Map
User A
User A
User B
User C
Home
Work
Home
Home
inbox1
(gearmand)
XMPP Server 1
1
2
1
2
Delivery Server
Background Job
Dispatcher
(gearmand)
inbox2
(gearmand)
XMPP Server 2

Message Delivery
Connection Map
User A
User A
User B
User C
Home
Work
Home
Home
inbox1
(gearmand)
XMPP Server 1
1
2
1
2
Delivery Server
Background Job
Dispatcher
(gearmand)
inbox2
(gearmand)
XMPP Server 2

Message Delivery
Connection Map
User A
User A
User B
User C
Home
Work
Home
Home
inbox1
(gearmand)
XMPP Server 1
1
2
1
2
Delivery Server
Background Job
Dispatcher
(gearmand)
inbox2
(gearmand)
XMPP Server 2

Connection Map
• too many ‘read’
• ‘write’ on resource-binding, presencechange
• Implementation
• TokyoTyrant
• HandlerSocket
• must not be volatile

Ocean::Cluster
• Ocean::Cluster::Gearman::Server
• Ocean::Cluster::Gearman::Worker

Cluster support
on front server
• set config
‘handler_class=Ocean::Cluster::Gearman::S
erver::Handler’
• set inbox and background-job-dispatcher

Front server
components
Background Job
Dispatcher
(gearmand)
inbox
(gearmand)
Ocean::Cluster::Gearman::Server::Handler
XMPP Server

Build worker project
!
• same as front server
• build project-template
• edit configuration
• write your handler class

Generate Project Template
•
•
ocean_cluster_gen --project=FooWorker --handler=FooWorker::Handler
cd FooWorker

Configuration
• set information about all your inboxes
• set information about background-jobdispatcher

Worker side (delivery server)
components
Process Manager
Connection Map
Handler
User A
User A
User B
User C
Serializer
1
2
1
2
Deliverer
Subscriber
DB (users, relationships, etc...)
Worker
Background Job
Dispatcher
(gearmand)
Home
Work
Home
Home
inbox0
(gearmand)
inbox1
(gearmand)

Transaction
• 同じユーザーからのイベント順序が壊
れないように調整が必要

Operation

Balancing
• Round-Robin
• DNS SRV Record
•
•
_xmpp_client._tcp.chat.mixi.jp 86400 IN SRV 10 20 5222 chat1.mixi.jp
_xmpp_client._tcp.chat.mixi.jp 86400 IN SRV 10 5 5222 chat2.mixi.jp

Service Status Store
name
chat1
chat2
chat3
connections
100000
1000
100000
Service Status API
and Client Hack
wait
2
1
2
Update Periodically
Status API
chat1.mixi.jp
User A
(Home)
chat2.mixi.jp
chat3.mixi.jp

Service Status Store
name
chat1
chat2
chat3
connections
100000
1000
100000
Service Status API
and Client Hack
wait
2
1
2
Update Periodically
Status API
chat1.mixi.jp
User A
(Home)
chat2.mixi.jp
chat3.mixi.jp

Service Status Store
name
chat1
chat2
chat3
connections
100000
1000
100000
Service Status API
and Client Hack
wait
2
1
2
Update Periodically
Status API
chat1.mixi.jp
User A
(Home)
chat2.mixi.jp
chat3.mixi.jp

Update & Reboot
• Worker side handler includes service-code,
so it’s required updating and rebooting
frequently. Worker’s job is stateless. So, it’s
no problem.
• Front-end server doesn’t include service-
code. So you don’t need to reboot it
frequently, but when you want to do it, you
have to pay attention.

Shutdown
• You can’t shutdown server without client
awareness.
• You have to write re-connection on client
side code
• Problem: algorithm distribution case
• At graceful shutdown, unavailable presences
are handled correctly
• Problem: at non-graceful shutdown

Front-server Broken
• フロントがぶっ壊れたときのために、
ConnectionMapから、壊れたサーバーに
つないでたユーザー全員のunavailable
presenceを処理するバッチを用意する必
要がある

Front-server
maintainance
• algorithm分散のフロントを定期再起動
するときは、代替が先に必要
• group_id = get_group(user_id)
• host = get_host(group_id)
• 代替サーバーを準備し、group/host
マップを書き換えてから、shutdown

Service(Client-side)
Strategy

Device/Platform
• PC Web ( HTTP Binding )
• PC Desktop App(AIR, Titanium, or Native)
• Smartphone
• Tablet
• Browser extension

Service
• 1to1 Messaging
• Room? (group chat)
• Voice/Video chat
• Push Notification

Ocean supports
• WebSocket hybi-draft-10 support
• HTTPBinding(WebSocket/SSE/XHR)

HTTP Binding

Methods
• Polling ( Ajax ) + Post API (+APNS|C2DM)
• Long Polling ( Ajax, Comet ) + Post API
• ServerSentEvent + Post API
• WebSocket
• TCP XMPP

‘Stateless’ problems
• Different way of connection management
from ordinary persistent connection
• single user-session may open many tabs
• user moves in short time. too many reconnection
• how to detect user to leave? (leave or
move)

Connection Map
User A
User A
User B
User C
Home
Work
Home
Home
1
2
1
2
too many ‘write’ and ‘presence’
in short time
server 1
open many tabs or
hop http-link
in short time
Tab
User A
server 2
server 3
mod_proxy(_balancer)
Tab
Tab
User B
when user leaves?

Solution 1
Algorithm Based Distribution
• ユーザー毎の宛先を固定することによっ
て、Connection Mapを利用しなくても、
特定のユーザーにパケットを届けるこ
とは可能になる
• ユーザーがWebサービスを開いてるか
どうかは関係なくなる

Problem
• Connection Mapを利用しないと、正確な
プレゼンスを把握できなくなる

Solution 2.
Stream Aggregation
• 同一cookieを持つ接続をまとめあげる
Sessionオブジェクトを利用する
• 状態管理はSessionオブジェクトで工夫
• Pending TimerでLeaveの検知

Connection Map
User A
User A
User B
User C
Aggregate
connections
as single session
hb1.mixi.jp
Home
Work
Home
Home
1
2
1
2
hb2.mixi.jp
Manager
hb3.mixi.jp
Manager
use timer to detect
users to leave.
Manager
until the expiration,
server should buffer
messages if received
algorithm based distribution: server_node=algorithm(user_id)
Tab
Tab
Tab
Tab
User A
Tab
Tab
User B
User C
(*) number of simultaneous Ajax request directed to same domain is limited,
so we need many nickname for each hb servers. (if we adopt long polling)

Handle Multiple Streams as One Single Session
Handler
Server
Session
Stream Manager
ResourceBinding
SessionEstablishment
Session
InitialPresence
User A
Stream
User A
Stream
User B
Stream
User B
Stream
Handshake
Auth

Tasks
• We need to
• prepare social ad system
• improve UI (moveless)

PubSub Support
• XEP-0061 Publish Subscribe
• We don’t need full-spec support
• don’t allow use to publish/subscribe
• server publish event

Publish Event from Web Service
Delivery Server
XMPP Server 1
User A
(Home)
User B
(Home)
UserEvents
(Notification,
Activity)
Web Service
XMPP Server 2
User A
(Work)
User C
(Home)

Web Service
UserEvents
(Notification,
Activity)
DeliveryCondition: ContentKind, UserPresence
WebHook
PubsubHubbub
C2DM
XMPP (HTTPBinding)
Streams
APNS
Email

Groupchat

Overview
• XMPP has MUC extension spec. but it’s
over spec, so we need arrangement
• Costs too much like presence
• limit numbers of channels and
participants
• Static-type groupchat should keep
synchronicity with ‘group(room?)’ feature
on our service

Video/Voice Chat

Problem?
• Isn’t Identity Association enough?
• AccountManage (android)
• FaceTime (iOS)

VoIP
Signaling
Server
Signaling Channel
Signaling Channel
RTP
Peer A
Peer B
Data Channel(Audio,Video)

VoIP
Signaling
Server
Cant’ get PeerB’s
Transport Address
Cant’ get PeerA’s
Transport Address
RTP
NATs
Peer A
NATs
Peer B

NAT Traversal

Hole Punching
1. Local host connects to an external host
NATs
Local
Host

Hole Punching
1. Local host connects to an external host
2. NAT allocates a transport address to localhost
10.100.100.100:10002
NATs
MAPPING
Local
Host
192.168.0.1:10001

Hole Punching
1. Local host connects to an external host
2. NAT allocates a transport address to localhost
10.100.100.100:10002
NATs
MAPPING
Local
Host
192.168.0.1:10001

Hole Punching
1. Local host connects to an external host
2. NAT allocates a transport address to localhost
3. An external host can send packet to local host
through mapped global address
NATs
Local
Host
But every NAT doesn’t work as expected

STUN
STUN
Server
Server Reflexive Transport Address
NATs
Host Transport Address
STUN
Client

STUN
STUN
Server
Signaling
Server
STUN
Server
Do Hole Punching
And
Obtain Reflexive Address
NATs
Peer A
NATs
Peer B

STUN
STUN
Server
Signaling
Server
STUN
Server
Exchange
Obtained Reflexive Address
Through
Signaling Server
NATs
Peer A
NATs
Peer B

STUN
STUN
Server
Signaling
Server
STUN
Server
Direct Communication Path
NATs
Peer A
NATs
Peer B

TURN
Signaling
Server
relay packet
NATs
Peer A
TURN
Server
NATs
Peer B

TURN
Relayed Transport Address
TURN
Server
NATs
Server Reflexive Transport Address
NATs
Host Transport Address
TURN
Client
Remote Peer

TURN
Relayed Transport Address
TURN
Server
NATs
Remote Peer
Server Reflexive Transport Address
NATs
TURN
Client
Allocation
Client ReflexiveTransport
XXX.XXX.XXX.XXX:10001
XXX.XXX.XXX.XXX:10001
XXX.XXX.XXX.XXX:10002
Relayed Transport
YYY.YYY.YYY.YYY:10001
YYY.YYY.YYY.YYY:10002
YYY.YYY.YYY.YYY:10003

TURN
Relayed Transport Address
TURN
Server
NATs
Remote Peer
A
NATs
Remote Peer
B
NATs
Remote Peer
C
NATs
TURN
Client

TURN
TURN
Server
NATs
TURN
Client
NATs
Remote Peer
Channel Binding
Client ReflexiveTransport
XXX.XXX.XXX.XXX:10001
XXX.XXX.XXX.XXX:10001
XXX.XXX.XXX.XXX:10002
Number
Relayed Transport
0x4001 YYY.YYY.YYY.YYY:10001
0x4002 YYY.YYY.YYY.YYY:10002
0x5001 YYY.YYY.YYY.YYY:10003
TODO: need peer reflexive address?

XMPP Jingle
• Signaling on XMPP

Serverless Messaging

Protocols
• UPnP
• Bonjour(Rendezvous)

Bonjour
• mDNS
• DNS-SD

Find Friends
• search service by DNS PTR records
• search host for indicated service by SRV
records
• search IP address by A record
• (TXT records for capability and status)

TODO
• backward compatibilityの確保(iChat, etc)
• 現在の要件への現実的な対応
(HTTPBinding)
• 将来の準備(WebRTC)