Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copart at M|18
Copart + MariaDB
Feb 27th 2018
WHAT IS COPART?
Copart, Inc.
• Copart is a publicly traded company, founded in 1982 by Willis J. Johnson.
• Copart, Inc., headquartered in...
Copart, Inc.
4
COPART: A TECHNOLOGY LEADER
SOME OF THE TECHNOLOGIES USED @ Copart
6
DATABASE @ Copart
• Our AS400 data is currently estimated at 33 TB and growing.
• We currently have 3 TB in MariaDB and gr...
ORGANIZATIONAL DIRECTION
• Light and agile ecosystem to facilitate global expansion.
• Existing technology is becoming leg...
TECHNICAL REQUIREMENTS
• Highly scalable database platform.
• Technology solution should be relevant for long term.
• SQL ...
MariaDB + REQUIREMENTS
• Technology is open source with an active community.
• MariaDB is SQL compliant.
• Lower operation...
WHAT WAS THE SELLING POINT?
• MariaDB is based on MySQL which has been a popular RDBMS for
a long time.
• Platform is beco...
LEARNINGS & WORKAROUNDS
• As ‘Sequence’ does not exist in MariaDB 10.1/10.2, we had to
create additional application logic...
MariaDB @ Copart
• Its been 3 years since we started with the implementation of MariaDB.
• We started with implementing Ma...
MASTER-MASTER/MASTER-SLAVE PROD
IMPLEMENTATION
1
5
GALERA CLUSTER MASTER-SLAVE
IMPLEMENTATION
1
7
GALERA CLUSTER MULTI-INSTANCE
IMPLEMENTATION
1
9
WHY MULTI-INSTANCE CLUSTER?
• Copart has multiple Non-Prod environments which need dedicated
clusters.
• Implementing 14 N...
COPART+MARIADB: FUTURE!!!
WHAT'S BAKING?
• ColumnStore and MariaDB AX implementation for reporting to
replace our In-Memory solution. Benchmarking r...
WHAT'S IN THE PIPELINE?
• We are planning to use MaxScale as a CDC tool to replace our
existing inhouse developed solution...
Q & A
2
4
Upcoming SlideShare
Loading in …5
×

M|18 How Copart Switched to MariaDB and Reduced Costs During Growth

370 views

Published on

M|18 How Copart Switched to MariaDB and Reduced Costs During Growth

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

M|18 How Copart Switched to MariaDB and Reduced Costs During Growth

  1. 1. Copart at M|18 Copart + MariaDB Feb 27th 2018
  2. 2. WHAT IS COPART?
  3. 3. Copart, Inc. • Copart is a publicly traded company, founded in 1982 by Willis J. Johnson. • Copart, Inc., headquartered in Dallas, Texas and provides real-time online vehicle auction and remarketing services globally. • Copart provides vehicle sellers with a range of services to process and sell salvage and clean title vehicles over the Internet using its patented virtual auction technology. 151,000+ Vehicles on sale now! 200+ locations across 12 countries 120+ Auctions with 18,000 Cars sold per day 250,000 Live bids from 1,000+ Participants per day A car sold every 50 secs • Visit www.Copart.com for more details. 3
  4. 4. Copart, Inc. 4
  5. 5. COPART: A TECHNOLOGY LEADER
  6. 6. SOME OF THE TECHNOLOGIES USED @ Copart 6
  7. 7. DATABASE @ Copart • Our AS400 data is currently estimated at 33 TB and growing. • We currently have 3 TB in MariaDB and growing rapidly. • On an average, we facilitate 300 TPS on AS400 every day. • We currently have 160 TPS on MariaDB every day. • Our technology ecosystem consists of our cutting-edge auctioning system. 7
  8. 8. ORGANIZATIONAL DIRECTION • Light and agile ecosystem to facilitate global expansion. • Existing technology is becoming legacy. • The organization is promoting open source technology. • Operational and administrative costs are high. • International market has stricter data compliance standards. 8
  9. 9. TECHNICAL REQUIREMENTS • Highly scalable database platform. • Technology solution should be relevant for long term. • SQL and logical objects should be compatible to prevent full re- architecture. • Operational costs should be low and system stability and availability should be high. • Need encryption data at rest and in motion for international data compliance. 9
  10. 10. MariaDB + REQUIREMENTS • Technology is open source with an active community. • MariaDB is SQL compliant. • Lower operational cost and high stability. • The platform supports functions and stored procedures. • Supports encryption of data at rest and in motion. • MariaDB fast tracks important features and patches. • Minimum retraining required for the Developers. • DBAs can pick up technology easily. • MariaDB can be installed on commodity hardware. 1 0
  11. 11. WHAT WAS THE SELLING POINT? • MariaDB is based on MySQL which has been a popular RDBMS for a long time. • Platform is becoming more popular and we want to be ahead of the curve. • Migrating structure and data from AS400 to MariaDB is fairly simple. • The cluster solution has very good HA/DR features. • MariaDB enterprise support is GREAT!!! 1 1
  12. 12. LEARNINGS & WORKAROUNDS • As ‘Sequence’ does not exist in MariaDB 10.1/10.2, we had to create additional application logic to generate and manage sequences. • Mandatory primary key requirement for Galera cluster needed tables and application logic redesign.​ • Galera is not optimized to support bulk loads, so ETL needed to be configured for smaller commit sizes. • Hibernate(JPA) needed to be customized to use the Read/Write split feature in Galera. 1 2
  13. 13. MariaDB @ Copart • Its been 3 years since we started with the implementation of MariaDB. • We started with implementing MariaDB for non-critical applications. • Today many of our business critical applications run off of MariaDB. • Realtime pipeline standalone implementation including data replication, StarQuest. Data will flow from source to website within a few seconds. • We currently have 15 instances of Galera Clusters(3 nodes each). • We also have multiple master-master and master-slave implementations. 1 3
  14. 14. MASTER-MASTER/MASTER-SLAVE PROD IMPLEMENTATION
  15. 15. 1 5
  16. 16. GALERA CLUSTER MASTER-SLAVE IMPLEMENTATION
  17. 17. 1 7
  18. 18. GALERA CLUSTER MULTI-INSTANCE IMPLEMENTATION
  19. 19. 1 9
  20. 20. WHY MULTI-INSTANCE CLUSTER? • Copart has multiple Non-Prod environments which need dedicated clusters. • Implementing 14 Non-Prod clusters on 42+14 servers is an administrative overhead. • This is a heavy resource utilization setup. • Reduced to 9+3 powerful machines. • MaxScale for read-write split and full load balanced. 2 0
  21. 21. COPART+MARIADB: FUTURE!!!
  22. 22. WHAT'S BAKING? • ColumnStore and MariaDB AX implementation for reporting to replace our In-Memory solution. Benchmarking results were good and pretty close to that of an In-Memory database platform. o Hardware and Software cost. o System stability. o Better performance. • JSON NoSQL implementation for unstructured data. 2 2
  23. 23. WHAT'S IN THE PIPELINE? • We are planning to use MaxScale as a CDC tool to replace our existing inhouse developed solution. • MariaDB with CentOS Cluster. • MariaDB is working on providing us a connector for Pentaho ETL. • With the next major release of MaxScale, we are looking at implementing: o AD authentication for DB access o Limiting DB user connections 2 3
  24. 24. Q & A 2 4

×