Funambol Server Architecture

Funambol Server Architecture Stefano Nichele

Overview

Funambol server components

Synchronization overview

Data synchronization service architecture

Push overview

Push Connection service

InboxListener and PIMListener

Funambol Server Architecture We will refer to those as services . The DS service is the primary service , without that the Funambol Server cannot run. And we are going to change some names.... Core component of the Funambol Server. It provides synchronization and push engines Allows push mail by checking email accounts Allows pim push by checking the database Allows Client TCP Push DataSynchronization Server Inbox Listener Pim Listener CTP Server DataSynchronization service Pim Listener service Push Connection service Inbox Listener service

Funambol Server Architecture Synchronization Push DataSynchronization service Pim Listener service Push Connection service Inbox Listener service DB Mail Server

Synchronization

Process to make equal two sets of data (repository, database)

One-way : data is coming just from one source (i.e. backup, restore)

Two-way : data is coming from both sources

PDA Backup Repo PDA My laptop

Synchronization

Two different modes:

Full-sync : the sources are synchronized by comparing the whole sets of data (OMA terminology: slow-sync)

Fast-sync : the sources are synchronized by comparing just the changes that occurred since the last sync.

SyncML

Open standard for data synchronization

XML based

Supported by major companies such as Nokia, Motorola....Funambol :-)

Since 2002 defined by Open Mobile Alliance (http://www.openmobilealliance.org)

SyncML – an example <SyncML xmlns='SYNCML:SYNCML1.2'> <SyncHdr> <VerDTD>1.2</VerDTD> <VerProto>SyncML/1.2</VerProto> <SessionID>1</SessionID> <MsgID>3</MsgID> <Target> <LocURI>IMEI:004400061769830</LocURI> </Target> <Source> <LocURI>http://syncserver.com/servlets/SyncML</LocURI> </Source> ... </SyncHdr> <SyncBody> <Status> <CmdID>1</CmdID> <MsgRef>3</MsgRef> <CmdRef>0</CmdRef> <Cmd>SyncHdr</Cmd> <TargetRef>http://syncserver.com/servlets/SyncML</TargetRef> <SourceRef>IMEI:004400061769830</SourceRef> <Data>200</Data> </Status> </SyncBody> </SyncML>

Data Synchronization service

It's the core and the only required service of the Funambol server (primary service)

It governs synchronization and push

Extensible and highly configurable

SyncML 1.1 and 1.2

Data Synchronization service Admin Manager Pipeline Manager Nokia Synclet Razr Synclet Push Engine Push Sender WebServices API Plugins Manager Plugin A Plugin B Framework Device Inventory SMS Service SMS Provider Synchr. Engine Email Sync Source PIM Sync Source Google Sync Source Officer

It allows additional processing and manipulation of the messages exchanged between the server and the client.

The kind of processing that is performed in the pipeline is a message level processing, such as the manipulation of the message elements.

Possible applications are:

Encoding/decoding of item content

Item filtering

Item ordering

Message decoration adding/removing elements on a custom basis

Pipeline Manager Nokia Synclet Razr Synclet

The idea behind the message processing pipeline is to be able to modify both incoming and outgoing messages:

in the former case, we want to be able to manipulate the message before it goes into the sync engine

instead, in the latter, we want to be able to change the message returned by the sync engine before sending it to the client.

Input and output pipeline components are called Synclet.

Pipeline Manager Nokia Synclet Razr Synclet Synchr. Engine Email Sync Source PIM Sync Source Google Sync Source Officer S E R V L E T Input pipeline Output pipeline Pipeline Manager Raw data SyncML SyncML

How are the synclets to perform selected ?

Synclets for any device

Synclets just for some devices: based on the user-agent of the requests

Example of synclet configuration file <?xml version="1.0" encoding="UTF-8"?> <java version="1.5.0_08" class="java.beans.XMLDecoder"> <object class="com.funambol.foundation.synclet.BeanShellSynclet"> <void property="header"> <string> user-agent </string> </void> <void property="pattern"> <string> NokiaE61 </string> </void> <void property="script"> <string>com/funambol/server/engine/pipeline/phones-support/bsh/NokiaE65in.bsh</string> </void> </object> </java> Pipeline Manager Nokia Synclet Razr Synclet

Main responsibilities:

authenticate and authorize the user

identify the sources and the destinations of data to be synchronized

identify what data needs to be updated/added/deleted

determine how updates must be applied

detect conflicts

resolve conflicts

Sync. Engine Email SS PIM SS Google SS Officer

Deep look at the synchronization engine

SyncML Messages Sync Session Handler Sync4jEngine Sync4jStrategy Email Sync Source PIM Sync Source Officer It represents the handler for a SyncML session. It coordinates and handles messages as dictated by the protocol. It coordinates the sync process delegating almost all the sync logic to Sync4jStrategy Given two sources of data, it performs the real synchronization process It checks if the credentials provided by the client are valid and if the user is authorized. A SyncSource is the interface versus the data store Sync. Engine Email SS PIM SS Google SS Officer

How can you interact with the engine process and how can you synchronize your data repository ?

Officer : it's the component that authenticates and authorizes a user. It's important since implementing your own officer you are able to authenticate a user on your user database.

SyncSource : it's the class used to access to the server data repository. You can develop your own PIM syncsource but also your source for MP3 or for whatever you want synchronization.

For more details, see the Developer Guide.

Sync. Engine Email SS PIM SS Google SS Officer

The Data Synchronization service provides the required infrastructure to allow you an easy integration with your SMS Gateway.

SMS Service

it's the entry point for the SMS infrastructure

no specific SMS gateway logic

SMS Provider

it's the real interface towards your preferred SMS gateway (i.e. SubitoSMS or Clickatell)

SMS Infrastructure SMS Message Recipient Sender SMS Service SMS Provider SMS Gateway Usually this is an HTTP request SMS Service SMS Provider

What can I do in the SMS Service ?

decorate the recipient/sender with additional information (like carrier/country)

work as proxy towards more than one SMSProvider selecting the one to use based on the recipient

provide smart functionalities (for example using the delivery detail/status information, a sms can be discharged if the previous one is still pending)

Out of the box we provide a simple SMS Service that forwards requests to an SMS Provider that just logs the messages.

SMS Service SMS Provider

Plug-ins : configurable objects that can be “deployed” in the DS service in order to extend its functionalities. For instance a simple plug-in could send an email to the system administrator to notify him for startup and shutdown events.

By now the plug-ins are notified just for startup and shutdown events.

As for the other configurable components, they can be deployed without stopping the DS service just copying the xml bean file under com/funambol/server/plugin .

Plugins Manager Plugin A Plugin B

It's the core of the administration API. It can be accessed locally or remotely using web services.

It exposes the administration API via web services.

It's the component that handles devices and their capabilities

....the latest DS service components Admin Manager WebServices API Device Inventory

Funambol Server Architecture Push DataSynchronization service Pim Listener service Push Connection service Inbox Listener service DB Mail Server

Push

Push sequence in summary:

1. the Funambol server detects that a device needs to sync

2. the Funambol server sends a push message to the device. This is pkg #0 in the OMA specs

3. the device starts a sync so that its data is updated with the one on the server

Funambol Server Data Repository Data Repository 1 2 3

Push

How can the server send the push message to the device ?

SMS seems the easiest way but SMSs are not free of charge

The client can run a TCP server so the server can open a TCP connection and sends the push message over that

The client can open a connection versus the server and keep it always open, ready for push message from the server

Funambol Server supports out of the box connection-less and connection-oriented push but SMS Push is easily implementable.

STP Push CTP Push SMS Push SMS Push Connection-less Push Connection-oriented Push

Push

Who does what ?

DS service Mail Server InboxListener service PIMListener service PIM DB Mail Server WS API Push Engine Push Connection service The Inbox Listener at regular time checks on the mail server if there are users to push If there are devices to push, the IL and the PIM listener call the DS service admin API The Push Engine tries to send the push message using connection-less push The Push Connection service tries to push the device using an existing connection The Push Engine asks to the Push Connection service to push the device using connection-oriented push The PIM Listener at regular time checks on the DB if there are devices to push 1 1 2 3 4 5

Main responsibilities:

Handling push :-)

Push Engine Push request WS Admin API Notification Engine Notification Builder Notification Sender DB The NotificationEngine uses a NotificationBuilder to create the Push message The NotificationEngine checks in the DB if there are old push messages The sender is the component used to deliver a push message to a device If the sender is not able to deliver a push message, it is stored in the database 1 2 3 4 Push Engine Push Sender

Builder and sender configurable per device

Two available builder implementations (for SyncML 1.1 and for SyncML 1.2)

Just one sender implementation that tries:

connection-less push

connection-oriented push (if the previous one fails)

SMS Service SMS Provider Push Engine Push Sender

Connection-less push

Main concepts:

the client must be able to run a TCP server waiting for connections from the server

when a push message must be sent, the DS service opens a socket and writes the message

the DS service must know the client IP address

the IP address must be reachable from the DS service

Connection-less push IP Address Pending Push IP Address

Address notification: process used by the client to notify its IP address

The device must notify its address any time it changes ! So it must be able to detect when it changes.

Address Listener Servlet DB If the server is able to send the pending push to the client, the IP address is stored in the DB. 1 2 3 The device sends its IP address to the server The server verifies if the client address is reachable trying to send the pending push messages

Connection-less push

Issues:

The client must be able to run a TCP server

The client must be able to detect when its IP address changes

The IP address must be reachable from the server

Connection-oriented push

Main concepts:

The client opens a connection on the server and keeps it open

The server uses that connection to deliver push messages

...really simple ;-)

Connection-oriented push UDP channel Connecting Pushing Push Connection Service 1 The client opens the connection and sends its credentials DS service WS API Push Engine 2 The Push Connection service calls the DS service WS API to authenticate the client The Push Sender sends a multicast UDP message with the push message 2 1 The Push Connection service receives the UDP message and checks if the recipient is connected 3 If the device is connected, the push message is written on the already open (by the client)connection

Push Connection service

Main challenge:

It must be able to handle millions of connections

Features:

Java Standard Application

Based on javax.nio

Based on Apache Mina (network application framework)

Clusterable

InboxListener and PIMListener

Both do the same work, at regular time they must check the data repository

Both based on the PushListener Framework...

Push Listener Framework InboxListener Task InboxListener Push Listener Framework PimListener Task PIMListener

Push Framework Listener

Framework ready to use to implement any listener. By now used for:

InboxListener

PIMListener

ExchangeListener

OXListener

Pretty similar to a scheduler with basic functionalities for push and for HA/scalability

Java Standard Application (no J2EE)

Based on Jdk 1.5 thread features

JMX interface

Pluggable architecture

Hey, this is not push.....it's polling !

UDP notification

Polling mode works with all mail servers

Notifiable mode can be used just in some cases

InboxListener service Mail Server Polling mode InboxListener service Mail Server Notifiable mode

Push Framework Listener Task executor Tasks queue Scheduled tasks One Shot tasks DB Registry Monitor Plugin Manager UDP Listener At startup it loads from the database the list of tasks to perform and publish it in the tasks queue Thread Pool. Any free thread pickup from the queue a task and starts its execution. It can contain scheduled tasks or “One Shot” tasks It allows notifiable mode ... ... User 1 User 2 User 3 User 4

How does it scale ?

Instance 1 of 3 Instance 2 of 3 Instance 0 of 3 Cluster It executes the tasks with (id % 3) = 0 It executes the tasks with (id % 3) = 1 It executes the tasks with (id % 3) = 2 InboxListener service A InboxListener service C InboxListener service B Any instance know its index and how many instances are in the cluster Tasks Repository The cluster is obtained using JGroups

HA – High availability

Cluster InboxListener service A InboxListener service C InboxListener service B Tasks Repository Instance 1 of 3 Instance 2 of 3 Instance 0 of 3 It executes the tasks with (id % 3) = 0 It executes the tasks with (id % 3) = 1 It executes the tasks with (id % 3) = 2 Instance 1 of 2 Instance 0 of 2 It executes the tasks with (id % 2) = 0 It executes the tasks with (id % 2) = 1

Conclusions Monitoring Monitoring WS API Connection-less push Request for connection-oriented push Connection-oriented push Synchronization DS service Mail Server InboxListener service PIMListener service PIM DB Mail Server WS API Push Engine Push Connection service 1 1 2 3 4 5 Sync Engine 6

References