Louis Zircher & Greg Henderson
February 27th 2018
Global Data Replication with
Galera for Ansell Guardian®
Introductions
Ansell
Guardian®
Technology
Business
Requirement
s
Technical
Architecture
Ansell® Global Leader in Protection Solutions
The Journey
Hazard
Assessments
1975 1995 2011
Application
Assessment Hand Protection
Loss Control
Analysis
International
Expansion
Sales
Transformation
US Patent
Guardian Launch
1990 2002
Safety Net Launch EMEA
2005 2014 2015 2016 2017
Channel
Partnership
Our Solution – Ansell Guardian®
Focus on Safety________________________________________________________
______
An integrated approach to improve
your business performance
An Integrated Solution
Ansell Guardian® partners with industrial and medical organisations to address the challenges in today’s
PPE environment and deliver measurable safety and business improvements.
MariaDB Saved Ansell Guardian®
Introduced to
MariaDB / Galera
• Looking for better ways to move
and synchronize data
• Across 3 Data Centers
• Heavy Data replication
• Long Production Deployment
Current
Synchronization
Process not viable
• Our hub and spoke model
pushed to limits of capability
• Data not replicating
• Pressure from Management to
fix
• Preplanning and evaluation of
MariaDB by Ansell made for a
smooth transition.
Implementation
• Only Delay was our Application
• Easy Implementation
• Major Benefits
• Increased Performance
• Days to Hours Deployment
• User Experience
Ansell Guardian® Today
Channel Sales TeamAnsell Sales Team Digital /E-connector
DIGITAL
Continue helping our customers
through Ansell sales force.
To further differentiate Ansell through our unique safety and productivity platform.
Expansion of Ansell Guardian
coverage through channel partner
sales teams.
Further integration with channel
partners and Ansell customers
And now the technical journey…
• Various global teams with different processes / tools
‑ Excel spreadsheets
‑ Custom Word document apps
‑ .NET desktop apps
‑ PocketPC apps
• Lotus Notes was our global replication tool
‑ Good at replicating objects
‑ But tough to interact with outside Notes
‑ Was on the way out as our corporate messaging platform
• Started working toward a global toolset 2005 to 2008
• Key challenges
‑ Global data synchronization
‑ Centralize “silo” development efforts
First global Guardian application
• Fat Java SWT client developed about 2010
‑ Local DB instance per client
‑ Requirement to synchronize data between all clients
• Adventures in data synchronization
‑ Aim to run sync through a central server-side DB instance
‑ Initial outsource prototype was a “home-grown” sync process
o Only took 17hrs or so… LOL
‑ So internal development team steps in
• Initial global data synchronization solution
‑ Found and installed a COTS DB replication tool
‑ Worked at logical SQL level, extensive manual configuration
‑ A bit painful to implement, but functional
First global Guardian application
Central DB Master
Server Americas
Americas
Multiple DB Slave
Clients
JDBC
Europe,
Africa, East
Asia
Australia, Southeast Asia
Java / web application hybrid
• Additions to Java / DB client solution by 2013
‑ Central Java application stack servers (JBoss / Tomcat)
‑ Local Java application server (Jetty)
‑ Embedded Chrome browser
• The web was the future…
Java / web application hybrid
Central DB Master
Server Americas
Americas
Europe,
Africa, East
Asia
Australia, Southeast Asia
Multiple DB Slave
Clients
JDBC
HTTP
HTTPHTTP
Guardian Next
• Full migration to web application by 2014
‑ Three regional Java DB application server groups across the globe
‑ DB replication via COTS tool between the regions
o Migrated previous central master DB server with numerous slave DB clients hierarchy to a master DB server with two slave DB servers
• Large number of servers to administer globally
‑ Began developing automation scripts
o Python Fabric API was easy… Ansible, Chef and such were a bit irritating
‑ Important!
• Database continues to grow…
‑ Over next couple years DB replication performance continues to degrade
Guardian Next
Central DB Master
(APAC)
Americas

Web Clients
Europe,
Africa, East
Asia

Web Clients
Australia,
Southeast Asia

Web Clients
DB Slave (EMEA)JDBC
HTTP
JDBC
HTTP
HTTP
DB Slave (AM)
MariaDB / Galera Cluster enters the arena
• By 2016, legacy database replication support had become an ongoing issue
‑ FKs were not our friend
‑ Occasional issues could require a full re-synchronization sequence to resolve, minimally 8+ hrs by this point. Fun weekend
work...
o NOT!
‑ Time to find a new synchronization solution
• Check out MariaDB / Galera Cluster
‑ Spent couple months of research, evaluation and testing
‑ Implemented Python Fabric scripting automation to speed up iterations of numerous operational test sequences
• Worked out functional solution without direct MariaDB support
‑ MariaDB offered to help multiple times
‑ But via online docs and various forums, no need
• Deployment planned in mid 2017
Emergency!
• Encountered brick wall DB replication issue early 2017
‑ Regional DBs started to diverge... not good!
‑ Implemented Python scripts to manually sync important data between regional DB instances… this was not sustainable.
• Pushed pending web app updates before deploying MariaDB / Galera
‑ Moving from master/slave configuration to synchronous master servers
MariaDB / Galera to the rescue
• Deployed MariaDB / Galera Cluster as drop-in replacement, included:
‑ Automated scripts for all major cluster ops startup/shutdown/rebuild/etc.
‑ Slave instances attached to cluster for read-intensive processes and backups
‑ Concurrent non-clustered local MariaDB on slave servers for transient data
o Want to minimize ongoing server costs
o Enhanced MariaDB /etc/init.d scripts to allow concurrent instances on different port
▪ Not really recommended... but has worked nicely up to this point
• Cluster build decreased to ~6 hrs and worked reliably
‑ Later… decreased cluster build to <2 hrs by implementing xtrabackup / IST solution instead of built-in SST
MariaDB / Galera to the rescue
MariaDB / Galera
Master (APAC)
Americas

Web Clients Europe,
Africa, East
Asia

Web Clients
Australia, Southeast
Asia

MariaDB / Galera

Master (EMEA)
Slave
HTTP HTTPHTTP
MariaDB / Galera
Master (AM)
Slave
Synchronous Master to
Master Replication
AM / EMEA / APAC
Slave
AM

Non-clustered
Transient Data
EMEA
Transien
t
APAC
Transien
t
Future
• Implement MaxScale to further harden DB infrastructure
• Investigate JSON support
• What else is cool?
‑ I’m here to find out...
Lessons learned
• Due diligence...prepare early
• Test, test, test...
• Automate everything you can
• A few decades of development experience doesn't hurt
‑ MariaDB support is there to help if needed ☺

Global Data Replication with Galera for Ansell Guardian®

  • 1.
    Louis Zircher &Greg Henderson February 27th 2018 Global Data Replication with Galera for Ansell Guardian®
  • 2.
  • 3.
    Ansell® Global Leaderin Protection Solutions
  • 4.
    The Journey Hazard Assessments 1975 19952011 Application Assessment Hand Protection Loss Control Analysis International Expansion Sales Transformation US Patent Guardian Launch 1990 2002 Safety Net Launch EMEA 2005 2014 2015 2016 2017 Channel Partnership
  • 5.
    Our Solution –Ansell Guardian® Focus on Safety________________________________________________________ ______ An integrated approach to improve your business performance
  • 6.
    An Integrated Solution AnsellGuardian® partners with industrial and medical organisations to address the challenges in today’s PPE environment and deliver measurable safety and business improvements.
  • 7.
    MariaDB Saved AnsellGuardian® Introduced to MariaDB / Galera • Looking for better ways to move and synchronize data • Across 3 Data Centers • Heavy Data replication • Long Production Deployment Current Synchronization Process not viable • Our hub and spoke model pushed to limits of capability • Data not replicating • Pressure from Management to fix • Preplanning and evaluation of MariaDB by Ansell made for a smooth transition. Implementation • Only Delay was our Application • Easy Implementation • Major Benefits • Increased Performance • Days to Hours Deployment • User Experience
  • 8.
    Ansell Guardian® Today ChannelSales TeamAnsell Sales Team Digital /E-connector DIGITAL Continue helping our customers through Ansell sales force. To further differentiate Ansell through our unique safety and productivity platform. Expansion of Ansell Guardian coverage through channel partner sales teams. Further integration with channel partners and Ansell customers
  • 9.
    And now thetechnical journey… • Various global teams with different processes / tools ‑ Excel spreadsheets ‑ Custom Word document apps ‑ .NET desktop apps ‑ PocketPC apps • Lotus Notes was our global replication tool ‑ Good at replicating objects ‑ But tough to interact with outside Notes ‑ Was on the way out as our corporate messaging platform • Started working toward a global toolset 2005 to 2008 • Key challenges ‑ Global data synchronization ‑ Centralize “silo” development efforts
  • 10.
    First global Guardianapplication • Fat Java SWT client developed about 2010 ‑ Local DB instance per client ‑ Requirement to synchronize data between all clients • Adventures in data synchronization ‑ Aim to run sync through a central server-side DB instance ‑ Initial outsource prototype was a “home-grown” sync process o Only took 17hrs or so… LOL ‑ So internal development team steps in • Initial global data synchronization solution ‑ Found and installed a COTS DB replication tool ‑ Worked at logical SQL level, extensive manual configuration ‑ A bit painful to implement, but functional
  • 11.
    First global Guardianapplication Central DB Master Server Americas Americas Multiple DB Slave Clients JDBC Europe, Africa, East Asia Australia, Southeast Asia
  • 12.
    Java / webapplication hybrid • Additions to Java / DB client solution by 2013 ‑ Central Java application stack servers (JBoss / Tomcat) ‑ Local Java application server (Jetty) ‑ Embedded Chrome browser • The web was the future…
  • 13.
    Java / webapplication hybrid Central DB Master Server Americas Americas Europe, Africa, East Asia Australia, Southeast Asia Multiple DB Slave Clients JDBC HTTP HTTPHTTP
  • 14.
    Guardian Next • Fullmigration to web application by 2014 ‑ Three regional Java DB application server groups across the globe ‑ DB replication via COTS tool between the regions o Migrated previous central master DB server with numerous slave DB clients hierarchy to a master DB server with two slave DB servers • Large number of servers to administer globally ‑ Began developing automation scripts o Python Fabric API was easy… Ansible, Chef and such were a bit irritating ‑ Important! • Database continues to grow… ‑ Over next couple years DB replication performance continues to degrade
  • 15.
    Guardian Next Central DBMaster (APAC) Americas
 Web Clients Europe, Africa, East Asia
 Web Clients Australia, Southeast Asia
 Web Clients DB Slave (EMEA)JDBC HTTP JDBC HTTP HTTP DB Slave (AM)
  • 16.
    MariaDB / GaleraCluster enters the arena • By 2016, legacy database replication support had become an ongoing issue ‑ FKs were not our friend ‑ Occasional issues could require a full re-synchronization sequence to resolve, minimally 8+ hrs by this point. Fun weekend work... o NOT! ‑ Time to find a new synchronization solution • Check out MariaDB / Galera Cluster ‑ Spent couple months of research, evaluation and testing ‑ Implemented Python Fabric scripting automation to speed up iterations of numerous operational test sequences • Worked out functional solution without direct MariaDB support ‑ MariaDB offered to help multiple times ‑ But via online docs and various forums, no need • Deployment planned in mid 2017
  • 17.
    Emergency! • Encountered brickwall DB replication issue early 2017 ‑ Regional DBs started to diverge... not good! ‑ Implemented Python scripts to manually sync important data between regional DB instances… this was not sustainable. • Pushed pending web app updates before deploying MariaDB / Galera ‑ Moving from master/slave configuration to synchronous master servers
  • 18.
    MariaDB / Galerato the rescue • Deployed MariaDB / Galera Cluster as drop-in replacement, included: ‑ Automated scripts for all major cluster ops startup/shutdown/rebuild/etc. ‑ Slave instances attached to cluster for read-intensive processes and backups ‑ Concurrent non-clustered local MariaDB on slave servers for transient data o Want to minimize ongoing server costs o Enhanced MariaDB /etc/init.d scripts to allow concurrent instances on different port ▪ Not really recommended... but has worked nicely up to this point • Cluster build decreased to ~6 hrs and worked reliably ‑ Later… decreased cluster build to <2 hrs by implementing xtrabackup / IST solution instead of built-in SST
  • 19.
    MariaDB / Galerato the rescue MariaDB / Galera Master (APAC) Americas
 Web Clients Europe, Africa, East Asia
 Web Clients Australia, Southeast Asia
 MariaDB / Galera
 Master (EMEA) Slave HTTP HTTPHTTP MariaDB / Galera Master (AM) Slave Synchronous Master to Master Replication AM / EMEA / APAC Slave AM
 Non-clustered Transient Data EMEA Transien t APAC Transien t
  • 20.
    Future • Implement MaxScaleto further harden DB infrastructure • Investigate JSON support • What else is cool? ‑ I’m here to find out...
  • 21.
    Lessons learned • Duediligence...prepare early • Test, test, test... • Automate everything you can • A few decades of development experience doesn't hurt ‑ MariaDB support is there to help if needed ☺