Zarafa SummerCamp 2012 - Z-push 2.0 changes for administrators
Z-PushZ-Push 2.0For administrators
Current situation: Z-Push 1.5.10• Z-Push 1.x code base 5 years old• Development & improvement of 1.5.x for 1,5 years• Considered very stable• Generally good support of devices• Supports ActiveSync 2.5 (comparable with MSEX 2003) – Early days of mobility – Only plaintext email possible (with the exception of iOS)
Not optimal ‚things„ with Z-Push 1.5• Re-creation of the ActiveSync profiles of the users necessary in many cases• Synchronization states are hardly manageable – Broken/missing states cause duplicates on mobile• No effective error detection• Invalid mapi data “drives devices crazy” – Battery drain – „incoming loops‟ (same item created various times on the server) – Never ending synchronisations• Many concurrent push connections (especially from iOS)• Code extensibility limited
Key features for the administrator• More control – About connected users and devices – Remote-wipe even with changed User password or without store• Improved logging – Different log levels – Ability to write full logs for certain users – Fail2ban support• Support friendly – Generally no need to recreate profiles on the mobiles
Stability improvements I• Semantic checks on data Every data object is checked before it is sent to the mobile, e.g. “is the end date of an appointment before the start date” Object can be ignored if a critical criteria is not met. Results in [WARN] level log lines and are also in the device details: SyncObject->Check(): object from type SyncContact: parameter picture is longer than 49152. Check failed (4)• Incoming loop detection (failstate) Assures that created items on the mobile are created ONCE on the server. Does not generate a specific log entry
Stability improvements IILoop detection – Detects when mobiles do not understand received data. The amount of data is reduced to 1 object trying to identify which object is causing the issue. – When problematic item is identified it is ignored. This is logged on [WARN] log level. This message is visible in z-push-admin. – Users could get error messages on the mobile “Synchronization not possible at the moment”. Retrying several time (refresh) generally solves this. – In some cases the “Mail app” (iOS) needs to be stopped and restarted. Device reboot is also a good option.
Stability improvements IIIDetection of loop in different processes• Interprocess loop detection Example: 1. Sync on an invalid folder -> triggers a FolderSync 2. FolderSync does not solve the issue 3. Sync on the invalid folder is issued again -> loops After 2 retries a full re-sync is executed automatically• Segfault detection Process did not terminate successfully. Logs message in [ERROR] log level: LoopDetection->ProcessLoopDetectionPreviousConnectionFailed(): Command Sync at 07/06/2012 17:03:51 with pid 4986 terminated unexpectedly or is still running.
Real push notifications• In combination with ZCP 7.0.6+ real notifications are available by using a “changes sink”! Polling vs. Notification 3 ZCP 1 Z-Push 2 ZCP 4 Z-Push Z-Push registers with the server Z-Push checks (1) with the on which folders it wants to be server every X seconds (2) notified on changes (3). if changes are available On change, server notifies Z-Push (4).
New tools – z-push-top (live statistics)• ZCP 7.0.6+ implements changes sink (see “Sink ...”)• Fallback method is polling (says “Polling ...”)
z-push-top• Several actions available – Type “h” or “help” in z-push-admin for full list – Possible to grep and tail directly on the log from tool – Filter listed output (option “f”) to only see certain commands, user, pid or device – Activate “wide” view for large screens (option “w”)• Hint: You can create a symbolic link on your server to easily access z-push-top and z-push-admin ln –s /usr/share/z-push/z-push-top.php /usr/sbin/z-push-top ln –s /usr/share/z-push/z-push-admin.php /usr/sbin/z-push-admin Note: to execute these commands you need reading and writing permissions to the logs and states
Device Management with z-push-admin• User administration (details about connected devices, remote-wipe, resynchronization)• Administrative clean up (reset loop detection)
New tools – z-push-admin (II)• Also available over SOAP interface
Device managementWith z-push-admin• See which users have connected an ActiveSync device: -a list• See details about one or all devices: -a list –u USERNAME –d DEVICE• Remote wipe a device -a wipe –u USERNAME –d DEVICE• Remove states of all or a single devices of a user -a remove –u USERNAME [-d DEVICE] Note: if devices are still connected they will re-provision and re-sync• Trigger re-sync of a device -a remove –u USERNAME [-d DEVICE] Note: remove and re-sync are similar, but no re-provision is necessary and dates like “first synchronization” stays intact.
Partial & Empty SynchronizationMultiple folders per request (device decision)• Z-Push 1.5: one sync request is always for one folder• Z-Push 2.0: one sync request can be for one or multiple folders“Partial” and “empty” sync• Partial: request sync for one or more folders, but also expects changes for “all known synchronized folders”• Empty: request sync without sending data, which means: give changes for “all known synchronized folders”
Heartbeat Synchronization• Is a combined PING and SYNC request• Initializes like a normal sync but blocks on requested folders if there are no changes atm• On change: the new data is sent immediately in the response• Partial/Empty & Heartbeat can be combined• Much lower latency till change is on the mobile – one response instead of one response and one full sync request• Less bandwidth consumption• May look confusing in z-push-top, as command “Sync”
Real notifications & heartbeat functionality “BlackBerry Experience” with ActiveSync Currently supported by Android 4 and Windows Phone 7+
Configuration options• Timezone configuration is important for Z-Push 2 – If no timezone is set (default), Europe/Amsterdam is used• State directory: STATE_DIR /var/lib/z-push/• Provisioning settings (as in 1.5.x) PROVISIONING and LOOSE_PROVISIONING• Amount of items to be synchronized (default all) SYNC_FILTERTIME_MAX SYNC_FILTERTYPE_ALL
Configuration options – Logging options• Log directory: LOGFILEDIR /var/log/z-push/• Log level (default) LOGLEVEL LOGLEVEL_INFO• Available: LOGLEVEL_OFF, LOGLEVEL_FATAL, LOGLEVEL_ERROR LOGLEVEL_WARN, LOGLEVEL_INFO, LOGLEVEL_DEBUGContaining user data: LOGLEVEL_WBXML wbxml data (for support!) LOGLEVEL_DEVICEID as wbxml but with device id LOGLEVEL_WBXMLSTACK wbxml processing stack
Configuration options – Advanced logging• Separate log for certain users LOGUSERLEVEL LOGLEVEL_DEVICEID Users which should be logged are named: $specialLogUsers = array(‘mo’, ‘firstname.lastname@example.org’);• Fail2ban log: LOGAUTHFAIL true Writes [WARN] message into the default log
Configuration: synchronize Public folder• Before synchronizing a folder, Z-Push will check if user has at least “Secretary permissions” on this folder• Globally configured for all users on system $additionalFolders = array( array( store => "SYSTEM", folderid => "7bd2229f7df92680b0700000000", name => “Learn as SPAM", type => SYNC_FOLDER_TYPE_USER_MAIL, ),• Several folders of several stores can be synchronized• Data can be discovered using a script available for ZCP in z-push/backends/zarafa/listfolders.php
State management• Z-Push 2 states are by default located in /var/lib/z-push• Directory needs to be writeable for apache• Below this directory, several subdirectories are created• States can be deleted manually! – same operation which z-push-admin resync/remove options do!• States are located by device using the last 2 characters of the device id, e.g. Device id States location Appl789789Z7G /var/lib/z-push/g/7• State files begin with the device id, e.g. Appl789789Z7G-74cad955-d390-4f02-b4a1-feb576f02a46-<xx>
States II• The states have a fixed structure <device-id>-<sync-uuid>-[type]-<counter>• Structure in detail – Appl789789Z7G the device id – devicedata main device information – 74cad955-d390-feb576f02a46 Unique key for a folder (uuid) – <no-type> normal synchronization state – hc HierarchyCache – fd FolderData (used e.g. for ping) – fs FailSave data (used by incoming loop protection) – bs Backend storage (IMAP backend)
Ping Tracking• concurrent PING connections – iOS opens a new ping connection every time an ActiveSync item is accessed – These connections run up to 1 hour on the server and block a apache process/thread and memory each – Each ping connection registers an own sink or worse: does polling• PingTracking solves this – Active PING connections check regularly if newer ping processes are available. If so, the older connections terminates. – This is done every refresh cycle (default 30 seconds - parameter PING_INTERVAL in config.php) – Only one ping connection stays active
Upgrading 1.5 -> 2.0• States are incompatible – Old states should be removed or not be copied – All the devices need to fully resynchronize – May cause some higher server load – May take some time (amount of users + items)• All data is automatically removed from the mobile• Data is then resynchronized• Some users will see errors – Device sometimes needs “time” and several sync cycles – Users should press “refresh” repeatedly – On iOS restart mail app or reboot device helps
Upgrading steps• Install & configure z-push 2 – Set timezone – Check/Create state and log folders – State and log folders need to be writeable for apache• Add dummy configuration to apache for testing & reload• Test in browser – Check for errors in /var/log/z-push/z-push.log• Update apache config & reload• Check progress with z-push-top
What usually goes wrong• Missing writing permissions to state & log files – Synchronization will not occur, errors will be in log file• Missing packages – Z-Push 2 uses shared memory for some operations – Some OS like RHEL need package php-process – Z-push-top requires php5-pcntl on some systems, e.g. SUSE• Appointments are listed with wrong time on mobiles – Z-Push server timezone is not set – Timezone on mobile is not set or wrong• High latency – If ZCP server is “too far away” connections can time out
Next step ?• Z-Push 2.0 released today ! Give it a try! Better/More New • Control • Features • Flexibility • Tools • Manageability • Supported devices • Compatibility • User friendlier (profiles) • Performance
Z-Push 1.5 Support• Complementary support until 31.12.2012• Upgrade support afterwards• Be aware that with Z-Push 1.5 more BYOD incompatibilities are to be expected• Currently Android 4 & BlackBerry Playbook are already not supported with 1.5.x
Where does it go?• MDM plugin for Webapp with new features – More synchronization status information – Messages/objects which could not be synchronized – Open shared folders (contacts & calendars only on iOS)• Planning 2.x: – Encrypted & signed emails – Auto-discovery – Policy management tool• Packaging – Easier installation & upgrade
Keep in touch• Announcements on Zarafa Release & Beta list Subscribe at http://zarafa.com/newsletter• Public Bug tracker: http://z-push.sf.net/tracker• Latest downloads: http://z-push.sf.net/download (beta & finals)• Sources at BerliOS: https://svn.berlios.de/wsvn/z-push/• Compatibility list: http://z-push.sf.net/compatibility Always up-to-date list about known devices and potential compatibility issues.