Funambol Server Architecture

Funambol Server Architecture Stefano Nichele

Overview
Funambol server components
Synchronization overview
Data synchronization service architecture
Push overview
Push Connection service
InboxListener and PIMListener

Funambol Server Architecture We will refer to those as services . The DS service is the primary service , without that the Funambol Server cannot run. And we are going to change some names.... Core component of the Funambol Server. It provides synchronization and push engines Allows push mail by checking email accounts Allows pim push by checking the database Allows Client TCP Push DataSynchronization Server Inbox Listener Pim Listener CTP Server DataSynchronization service Pim Listener service Push Connection service Inbox Listener service

Funambol Server Architecture Synchronization Push DataSynchronization service Pim Listener service Push Connection service Inbox Listener service DB Mail Server

Synchronization
Process to make equal two sets of data (repository, database)
One-way : data is coming just from one source (i.e. backup, restore)
Two-way : data is coming from both sources
PDA Backup Repo PDA My laptop

Synchronization
Two different modes:
Full-sync : the sources are synchronized by comparing the whole sets of data (OMA terminology: slow-sync)
Fast-sync : the sources are synchronized by comparing just the changes that occurred since the last sync.

SyncML
Open standard for data synchronization
XML based
Supported by major companies such as Nokia, Motorola....Funambol :-)
Since 2002 defined by Open Mobile Alliance (http://www.openmobilealliance.org)

SyncML – an example <SyncML xmlns='SYNCML:SYNCML1.2'> <SyncHdr> <VerDTD>1.2</VerDTD> <VerProto>SyncML/1.2</VerProto> <SessionID>1</SessionID> <MsgID>3</MsgID> <Target> <LocURI>IMEI:004400061769830</LocURI> </Target> <Source> <LocURI>http://syncserver.com/servlets/SyncML</LocURI> </Source> ... </SyncHdr> <SyncBody> <Status> <CmdID>1</CmdID> <MsgRef>3</MsgRef> <CmdRef>0</CmdRef> <Cmd>SyncHdr</Cmd> <TargetRef>http://syncserver.com/servlets/SyncML</TargetRef> <SourceRef>IMEI:004400061769830</SourceRef> <Data>200</Data> </Status> </SyncBody> </SyncML>

Data Synchronization service
It's the core and the only required service of the Funambol server (primary service)
It governs synchronization and push
Extensible and highly configurable
SyncML 1.1 and 1.2

Data Synchronization service Admin Manager Pipeline Manager Nokia Synclet Razr Synclet Push Engine Push Sender WebServices API Plugins Manager Plugin A Plugin B Framework Device Inventory SMS Service SMS Provider Synchr. Engine Email Sync Source PIM Sync Source Google Sync Source Officer

It allows additional processing and manipulation of the messages exchanged between the server and the client.
The kind of processing that is performed in the pipeline is a message level processing, such as the manipulation of the message elements.
Possible applications are:
Encoding/decoding of item content
Item filtering
Item ordering
Message decoration adding/removing elements on a custom basis
Pipeline Manager Nokia Synclet Razr Synclet

The idea behind the message processing pipeline is to be able to modify both incoming and outgoing messages:
in the former case, we want to be able to manipulate the message before it goes into the sync engine
instead, in the latter, we want to be able to change the message returned by the sync engine before sending it to the client.
Input and output pipeline components are called Synclet.
Pipeline Manager Nokia Synclet Razr Synclet Synchr. Engine Email Sync Source PIM Sync Source Google Sync Source Officer S E R V L E T Input pipeline Output pipeline Pipeline Manager Raw data SyncML SyncML

How are the synclets to perform selected ?
Synclets for any device
Synclets just for some devices: based on the user-agent of the requests
Example of synclet configuration file <?xml version="1.0" encoding="UTF-8"?> <java version="1.5.0_08" class="java.beans.XMLDecoder"> <object class="com.funambol.foundation.synclet.BeanShellSynclet"> <void property="header"> <string> user-agent </string> </void> <void property="pattern"> <string> NokiaE61 </string> </void> <void property="script"> <string>com/funambol/server/engine/pipeline/phones-support/bsh/NokiaE65in.bsh</string> </void> </object> </java> Pipeline Manager Nokia Synclet Razr Synclet

Main responsibilities:
authenticate and authorize the user
identify the sources and the destinations of data to be synchronized
identify what data needs to be updated/added/deleted
determine how updates must be applied
detect conflicts
resolve conflicts
Sync. Engine Email SS PIM SS Google SS Officer

Deep look at the synchronization engine
SyncML Messages Sync Session Handler Sync4jEngine Sync4jStrategy Email Sync Source PIM Sync Source Officer It represents the handler for a SyncML session. It coordinates and handles messages as dictated by the protocol. It coordinates the sync process delegating almost all the sync logic to Sync4jStrategy Given two sources of data, it performs the real synchronization process It checks if the credentials provided by the client are valid and if the user is authorized. A SyncSource is the interface versus the data store Sync. Engine Email SS PIM SS Google SS Officer

How can you interact with the engine process and how can you synchronize your data repository ?
Officer : it's the component that authenticates and authorizes a user. It's important since implementing your own officer you are able to authenticate a user on your user database.
SyncSource : it's the class used to access to the server data repository. You can develop your own PIM syncsource but also your source for MP3 or for whatever you want synchronization.
For more details, see the Developer Guide.
Sync. Engine Email SS PIM SS Google SS Officer

The Data Synchronization service provides the required infrastructure to allow you an easy integration with your SMS Gateway.
SMS Service
it's the entry point for the SMS infrastructure
no specific SMS gateway logic
SMS Provider
it's the real interface towards your preferred SMS gateway (i.e. SubitoSMS or Clickatell)
SMS Infrastructure SMS Message Recipient Sender SMS Service SMS Provider SMS Gateway Usually this is an HTTP request SMS Service SMS Provider

What can I do in the SMS Service ?
decorate the recipient/sender with additional information (like carrier/country)
work as proxy towards more than one SMSProvider selecting the one to use based on the recipient
provide smart functionalities (for example using the delivery detail/status information, a sms can be discharged if the previous one is still pending)
Out of the box we provide a simple SMS Service that forwards requests to an SMS Provider that just logs the messages.
SMS Service SMS Provider

Plug-ins : configurable objects that can be “deployed” in the DS service in order to extend its functionalities. For instance a simple plug-in could send an email to the system administrator to notify him for startup and shutdown events.
By now the plug-ins are notified just for startup and shutdown events.
As for the other configurable components, they can be deployed without stopping the DS service just copying the xml bean file under com/funambol/server/plugin .
Plugins Manager Plugin A Plugin B

It's the core of the administration API. It can be accessed locally or remotely using web services.
It exposes the administration API via web services.
It's the component that handles devices and their capabilities
....the latest DS service components Admin Manager WebServices API Device Inventory

Funambol Server Architecture Push DataSynchronization service Pim Listener service Push Connection service Inbox Listener service DB Mail Server

Push
Push sequence in summary:
1. the Funambol server detects that a device needs to sync
2. the Funambol server sends a push message to the device. This is pkg #0 in the OMA specs
3. the device starts a sync so that its data is updated with the one on the server
Funambol Server Data Repository Data Repository 1 2 3

Push
How can the server send the push message to the device ?
SMS seems the easiest way but SMSs are not free of charge
The client can run a TCP server so the server can open a TCP connection and sends the push message over that
The client can open a connection versus the server and keep it always open, ready for push message from the server
Funambol Server supports out of the box connection-less and connection-oriented push but SMS Push is easily implementable.
STP Push CTP Push SMS Push SMS Push Connection-less Push Connection-oriented Push

Push
Who does what ?
DS service Mail Server InboxListener service PIMListener service PIM DB Mail Server WS API Push Engine Push Connection service The Inbox Listener at regular time checks on the mail server if there are users to push If there are devices to push, the IL and the PIM listener call the DS service admin API The Push Engine tries to send the push message using connection-less push The Push Connection service tries to push the device using an existing connection The Push Engine asks to the Push Connection service to push the device using connection-oriented push The PIM Listener at regular time checks on the DB if there are devices to push 1 1 2 3 4 5

Main responsibilities:
Handling push :-)
Push Engine Push request WS Admin API Notification Engine Notification Builder Notification Sender DB The NotificationEngine uses a NotificationBuilder to create the Push message The NotificationEngine checks in the DB if there are old push messages The sender is the component used to deliver a push message to a device If the sender is not able to deliver a push message, it is stored in the database 1 2 3 4 Push Engine Push Sender

Builder and sender configurable per device
Two available builder implementations (for SyncML 1.1 and for SyncML 1.2)
Just one sender implementation that tries:
connection-less push
connection-oriented push (if the previous one fails)
SMS Service SMS Provider Push Engine Push Sender

Connection-less push
Main concepts:
the client must be able to run a TCP server waiting for connections from the server
when a push message must be sent, the DS service opens a socket and writes the message
the DS service must know the client IP address
the IP address must be reachable from the DS service

Connection-less push IP Address Pending Push IP Address
Address notification: process used by the client to notify its IP address
The device must notify its address any time it changes ! So it must be able to detect when it changes.
Address Listener Servlet DB If the server is able to send the pending push to the client, the IP address is stored in the DB. 1 2 3 The device sends its IP address to the server The server verifies if the client address is reachable trying to send the pending push messages

Connection-less push
Issues:
The client must be able to run a TCP server
The client must be able to detect when its IP address changes
The IP address must be reachable from the server

Connection-oriented push
Main concepts:
The client opens a connection on the server and keeps it open
The server uses that connection to deliver push messages
...really simple ;-)

Connection-oriented push UDP channel Connecting Pushing Push Connection Service 1 The client opens the connection and sends its credentials DS service WS API Push Engine 2 The Push Connection service calls the DS service WS API to authenticate the client The Push Sender sends a multicast UDP message with the push message 2 1 The Push Connection service receives the UDP message and checks if the recipient is connected 3 If the device is connected, the push message is written on the already open (by the client)connection

Push Connection service
Main challenge:
It must be able to handle millions of connections
Features:
Java Standard Application
Based on javax.nio
Based on Apache Mina (network application framework)
Clusterable

InboxListener and PIMListener
Both do the same work, at regular time they must check the data repository
Both based on the PushListener Framework...
Push Listener Framework InboxListener Task InboxListener Push Listener Framework PimListener Task PIMListener

Push Framework Listener
Framework ready to use to implement any listener. By now used for:
InboxListener
PIMListener
ExchangeListener
OXListener
Pretty similar to a scheduler with basic functionalities for push and for HA/scalability
Java Standard Application (no J2EE)
Based on Jdk 1.5 thread features
JMX interface
Pluggable architecture

Hey, this is not push.....it's polling !
UDP notification
Polling mode works with all mail servers
Notifiable mode can be used just in some cases
InboxListener service Mail Server Polling mode InboxListener service Mail Server Notifiable mode

Push Framework Listener Task executor Tasks queue Scheduled tasks One Shot tasks DB Registry Monitor Plugin Manager UDP Listener At startup it loads from the database the list of tasks to perform and publish it in the tasks queue Thread Pool. Any free thread pickup from the queue a task and starts its execution. It can contain scheduled tasks or “One Shot” tasks It allows notifiable mode ... ... User 1 User 2 User 3 User 4

How does it scale ?
Instance 1 of 3 Instance 2 of 3 Instance 0 of 3 Cluster It executes the tasks with (id % 3) = 0 It executes the tasks with (id % 3) = 1 It executes the tasks with (id % 3) = 2 InboxListener service A InboxListener service C InboxListener service B Any instance know its index and how many instances are in the cluster Tasks Repository The cluster is obtained using JGroups

HA – High availability
Cluster InboxListener service A InboxListener service C InboxListener service B Tasks Repository Instance 1 of 3 Instance 2 of 3 Instance 0 of 3 It executes the tasks with (id % 3) = 0 It executes the tasks with (id % 3) = 1 It executes the tasks with (id % 3) = 2 Instance 1 of 2 Instance 0 of 2 It executes the tasks with (id % 2) = 0 It executes the tasks with (id % 2) = 1

Conclusions Monitoring Monitoring WS API Connection-less push Request for connection-oriented push Connection-oriented push Synchronization DS service Mail Server InboxListener service PIMListener service PIM DB Mail Server WS API Push Engine Push Connection service 1 1 2 3 4 5 Sync Engine 6

References
Funambol Forge - https://www.forge.funambol.org/
Funambol Developer Guide
http://download.forge.objectweb.org/sync4j/funambol-v7-developers-guide.pdf
Funambol design documents
https://core.forge.funambol.org/wiki/FunambolDesignDocuments
Open Mobile Alliance - http://www.openmobilealliance.org
myFunambol - http://my.funambol.com
Funambol mailing lists - https://core.forge.funambol.org/ds/viewForums.do

The End

http://www.slideshare.net	53
https://funcon2008.forge.funambol.org	35
http://translate.googleusercontent.com	3
http://static.slideshare.net	1

Funambol Server Architecture

by Funambol on Dec 04, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 92

Statistics

Funambol Server Architecture — Presentation Transcript