Join Matthew Berg, Synacor E-mail Lead Engineer, to this video series about Operation Zimbra Collaboration for Service Providers. In this second part Matthew will talk about Synacor Operations.

1. SYNACOR MAIL –
OPERATING ZIMBRA FOR SERVICE PROVIDERS
Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Synacor, Inc.
Matthew Berg
Synacor E-mail Lead Engineer
March, 2016
PART 2

2. Synacor Mail – Operating Zimbra for Service Providers
1. Synacor as a Service providers for CSPs
2. Synacor Operations
3. Geo-Redundant Deployment of Zimbra
4. Future Directions
Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.

3. 2Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
SYNACOR OPERATIONS
1. OPERATIONAL PHILOSOPHY
2. DEPLOY AND OPERATE
3. MANAGEMENT
4. MONITORING
5. METRICS
6. MAINTENANCE
7. MIGRATION

4. 3Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Our goal is to reliably detect and mitigate operational issues before
they become end user impacting.
This requires comprehensive understanding of both system and
service performance, long term trending and analysis, and ready
access to historical metrics and log data.
By doing so we improve customers satisfaction, reduce cost of
technical operations, and eliminate call center involvement.
OPERATIONAL PHILOSOPHY

5. 4Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
SYNACOR OPERATIONS
1. PHILOSOPHY
2. DEPLOY AND OPERATE
3. MANAGEMENT
4. MONITORING
5. METRICS
6. MAINTENANCE
7. MIGRATION

6. 5Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Deployment
 Multiple deployment strategies
 All physical
 All virtual
 Hybrid
 Multiple storage backends
 Block storage
 Software defined storage

7. 6Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Operations
 Management
 Routine deployment and operation
 Monitoring
 Reactive and proactive health monitoring
 Metrics
 Data gathering for capacity planning, reporting and
troubleshooting
 Maintenance
 Resolving browser compatibility issues
 Deploying bug fixes and security updates
 Migrations
 Movement of accounts into and between systems

8. 7Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
SYNACOR OPERATIONS
1. PHILOSOPHY
2. DEPLOY AND OPERATE
3. MANAGEMENT
4. MONITORING
5. METRICS
6. MAINTENANCE
7. MIGRATION

9. 8Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Management
 Centralized tooling
 Puppet for config management
 Mcollective for distributed operations
 SCAPI
 Restful API for provisioning and management
 Etools
 Management interface built on SCAPI

10. 9Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Etools: Server Administration

11. 10Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Etools: Queue Monitoring

12. 11Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Etools: Log Access / Abuse Management

13. 12Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
SYNACOR OPERATIONS
1. PHILOSOPHY
2. DEPLOY AND OPERATE
3. MANAGEMENT
4. MONITORING
5. METRICS
6. MAINTENANCE
7. MIGRATION

14. 13Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Monitoring
 24/7 Coverage
 NOC
 Offshore admins
 On-call primary admins
 Synformer
 Network, platform and application monitoring
 Graphing and reporting capabilities

15. 14Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Synformer: Alert View

16. 15Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Synformer: Graph View

17. 16Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Synformer: Ad-hoc Report

18. 17Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
SYNACOR OPERATIONS
1. PHILOSOPHY
2. DEPLOY AND OPERATE
3. MANAGEMENT
4. MONITORING
5. METRICS
6. MAINTENANCE
7. MIGRATION

19. 18Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Metrics
 Graphite
 Centralized storage of system and application metrics
 Ad-hoc visualization
 Dashboards for monitoring and reporting
 Log aggregation and analysis
 Centralized log storage
 Analysis and report done with custom scripts

20. 19Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Graphite: Java GC activity

21. 20Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Graphite: Active User Dashboard

22. 21Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Graphite: Mail Analysis Dashboard

23. 22Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
SYNACOR OPERATIONS
1. PHILOSOPHY
2. DEPLOY AND OPERATE
3. MANAGEMENT
4. MONITORING
5. METRICS
6. MAINTENANCE
7. MIGRATION

24. 23Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Maintenance: Browser issues
 Styles
 Stripped down _base3 skin properties (~ 30 lines)
 Additional IDs added to skin.html for CSS access
 CSS properties converted to LESS; mixin support used to
accommodate browser differences
 Modified view.tag to detect IE
 Scripts
 Modernizr and jquery incorporated
 FixZimbra.js to correct upstream issues
 SynaSkinOverrides.js to augment upstream functionality
 Zimlets
 com_synacor_core_change to apply fixes after application load

25. 24Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Maintenance: Bug fixes and security
 Backport java classes
 Decompile and examine patches
 Apply changes to deployed version
 Update and distribute jars
 Backport dependencies
 Examine dependencies
 Compile new version or backport as appropriate
 Make sure to incorporate any Zimbra customizations in
ZimbraThirdParty repo

26. 25Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
SYNACOR OPERATIONS
1. PHILOSOPHY
2. DEPLOY AND OPERATE
3. MANAGEMENT
4. MONITORING
5. METRICS
6. MAINTENANCE
7. MIGRATION

27. 26Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.Contains proprietary and confidential information owned by Synacor, Inc. © / 2014 Synacor, Inc.
PROVEN EMAIL MIGRATION PROCESS AND PLATFORM
26Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.

28. 27Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Migrations
 Disbatch
 Elastic, distributed batch processing framework
 Used for internal and external mail migrations
 Plugins for ingesting from a variety of platforms
 YAMS (Yet Another Mail Syncer)
 Rewrite of imapsync
 Used to migration millions of mailboxes
 Fully managed and monitored

29. 28Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Migration Overview

30. 29Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Disbatch Management (v1)

31. 30Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.
Disbatch Management (v2)

32. Synacor Mail – Operating Zimbra for Service Providers
1. Synacor as a Service providers for CSPs
2. Synacor Operations
3. Geo-Redundant Deployment of Zimbra
4. Future Directions
Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.

33. THANK YOU
Contains proprietary and confidential information owned by Synacor, Inc. © / 2016 Synacor, Inc.