Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

M|18 How Copart Switched to MariaDB and Reduced Costs During Growth

144 views

Published on

M|18 How Copart Switched to MariaDB and Reduced Costs During Growth

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

M|18 How Copart Switched to MariaDB and Reduced Costs During Growth

  1. 1. Copart at M|18 Copart + MariaDB Feb 27th 2018
  2. 2. WHAT IS COPART?
  3. 3. Copart, Inc. • Copart is a publicly traded company, founded in 1982 by Willis J. Johnson. • Copart, Inc., headquartered in Dallas, Texas and provides real-time online vehicle auction and remarketing services globally. • Copart provides vehicle sellers with a range of services to process and sell salvage and clean title vehicles over the Internet using its patented virtual auction technology. 151,000+ Vehicles on sale now! 200+ locations across 12 countries 120+ Auctions with 18,000 Cars sold per day 250,000 Live bids from 1,000+ Participants per day A car sold every 50 secs • Visit www.Copart.com for more details. 3
  4. 4. Copart, Inc. 4
  5. 5. COPART: A TECHNOLOGY LEADER
  6. 6. SOME OF THE TECHNOLOGIES USED @ Copart 6
  7. 7. DATABASE @ Copart • Our AS400 data is currently estimated at 33 TB and growing. • We currently have 3 TB in MariaDB and growing rapidly. • On an average, we facilitate 300 TPS on AS400 every day. • We currently have 160 TPS on MariaDB every day. • Our technology ecosystem consists of our cutting-edge auctioning system. 7
  8. 8. ORGANIZATIONAL DIRECTION • Light and agile ecosystem to facilitate global expansion. • Existing technology is becoming legacy. • The organization is promoting open source technology. • Operational and administrative costs are high. • International market has stricter data compliance standards. 8
  9. 9. TECHNICAL REQUIREMENTS • Highly scalable database platform. • Technology solution should be relevant for long term. • SQL and logical objects should be compatible to prevent full re- architecture. • Operational costs should be low and system stability and availability should be high. • Need encryption data at rest and in motion for international data compliance. 9
  10. 10. MariaDB + REQUIREMENTS • Technology is open source with an active community. • MariaDB is SQL compliant. • Lower operational cost and high stability. • The platform supports functions and stored procedures. • Supports encryption of data at rest and in motion. • MariaDB fast tracks important features and patches. • Minimum retraining required for the Developers. • DBAs can pick up technology easily. • MariaDB can be installed on commodity hardware. 1 0
  11. 11. WHAT WAS THE SELLING POINT? • MariaDB is based on MySQL which has been a popular RDBMS for a long time. • Platform is becoming more popular and we want to be ahead of the curve. • Migrating structure and data from AS400 to MariaDB is fairly simple. • The cluster solution has very good HA/DR features. • MariaDB enterprise support is GREAT!!! 1 1
  12. 12. LEARNINGS & WORKAROUNDS • As ‘Sequence’ does not exist in MariaDB 10.1/10.2, we had to create additional application logic to generate and manage sequences. • Mandatory primary key requirement for Galera cluster needed tables and application logic redesign.​ • Galera is not optimized to support bulk loads, so ETL needed to be configured for smaller commit sizes. • Hibernate(JPA) needed to be customized to use the Read/Write split feature in Galera. 1 2
  13. 13. MariaDB @ Copart • Its been 3 years since we started with the implementation of MariaDB. • We started with implementing MariaDB for non-critical applications. • Today many of our business critical applications run off of MariaDB. • Realtime pipeline standalone implementation including data replication, StarQuest. Data will flow from source to website within a few seconds. • We currently have 15 instances of Galera Clusters(3 nodes each). • We also have multiple master-master and master-slave implementations. 1 3
  14. 14. MASTER-MASTER/MASTER-SLAVE PROD IMPLEMENTATION
  15. 15. 1 5
  16. 16. GALERA CLUSTER MASTER-SLAVE IMPLEMENTATION
  17. 17. 1 7
  18. 18. GALERA CLUSTER MULTI-INSTANCE IMPLEMENTATION
  19. 19. 1 9
  20. 20. WHY MULTI-INSTANCE CLUSTER? • Copart has multiple Non-Prod environments which need dedicated clusters. • Implementing 14 Non-Prod clusters on 42+14 servers is an administrative overhead. • This is a heavy resource utilization setup. • Reduced to 9+3 powerful machines. • MaxScale for read-write split and full load balanced. 2 0
  21. 21. COPART+MARIADB: FUTURE!!!
  22. 22. WHAT'S BAKING? • ColumnStore and MariaDB AX implementation for reporting to replace our In-Memory solution. Benchmarking results were good and pretty close to that of an In-Memory database platform. o Hardware and Software cost. o System stability. o Better performance. • JSON NoSQL implementation for unstructured data. 2 2
  23. 23. WHAT'S IN THE PIPELINE? • We are planning to use MaxScale as a CDC tool to replace our existing inhouse developed solution. • MariaDB with CentOS Cluster. • MariaDB is working on providing us a connector for Pentaho ETL. • With the next major release of MaxScale, we are looking at implementing: o AD authentication for DB access o Limiting DB user connections 2 3
  24. 24. Q & A 2 4

×