HOW TO MANAGE 1 BILLION+ INSERTS A DAY INTO A MYSQL RESOURCE                                                        Sam La...
SMITH ELECTRIC VEHICLES & TELEMETRY    • Worlds largest manufacturer of Commercial, all-electric vehicles    • Smith Link ...
SYSTEM ARCHITECTURE    How to manage 1 billion + inserts a day into a MySQL resource3
ORIGINAL PROBLEMSUnable to insert data quick enoughProblems with concurrently processing dataData processing scripts takin...
INNODBHuge scaling potentialTransactionalTuneableWell supported/Strong community                     How to manage 1 billi...
MEASUREMeasure, measure, measureAlways measure as many metrics as possibleDe ne a scopeDecide on metrics that can best det...
TOOLSSysbenchCacti (mysql-cacti-templates)IostatIotopmysqltuner.pl                     How to manage 1 billion + inserts a...
RESOURCESLarge amount of RAM (all machines 100GB+) Our latest server roll outshave minimum of 144GBinnodb_buffer_pool_size...
STRIKING A BALANCEMaximum safe RAM utilisationMake the sure buffer pool never swaps                     How to manage 1 bi...
LOAD DATA INFILEOur application pulls messages from the queue and creates les forimportingHave a well optimised perl appli...
OTHER SETTINGSinnodb_ le_per_table=1innodb_ ush_log_at_trx_commit=2innodb_ ush_method=O_DIRECTinnodb_log_ le_size=1G      ...
SANOpen-iSCSI settings:node.session.cmds_maxnode.session.queue_depth                       How to manage 1 billion + inser...
MESSAGE QUEUEMakes it much easier to plan downtimeCan comfortably tweak the live systemCan pull data in different directio...
SUMMARY             www.samlambert.com                     @isamlambert                        Questions?     How to manag...
Upcoming SlideShare
Loading in …5
×

How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk

862
-1

Published on

How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk presented at Super Mondays October http://www.supermondays.org/2011/10/24/lightning-talks-october-31st-2011/

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
862
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk

    1. 1. HOW TO MANAGE 1 BILLION+ INSERTS A DAY INTO A MYSQL RESOURCE Sam Lambert, October, 2011 PHP & Continuous Data Processing, PHPNW 20111
    2. 2. SMITH ELECTRIC VEHICLES & TELEMETRY • Worlds largest manufacturer of Commercial, all-electric vehicles • Smith Link – on-board vehicle telematics system, capturing over 2500 data points each second on the vehicle and broadcasting them over mobile network • ~400 telemetry enabled vehicles on the road • Worlds largest telemetry project outside of F1 • Number of inserts per day increases by around 250,000,000 per month • Record on truck all the way to our web interface in a maximum of 30 seconds How to manage 1 billion + inserts a day into a MySQL resource2
    3. 3. SYSTEM ARCHITECTURE How to manage 1 billion + inserts a day into a MySQL resource3
    4. 4. ORIGINAL PROBLEMSUnable to insert data quick enoughProblems with concurrently processing dataData processing scripts taking an unacceptable amount of timeNo obvious path for scaling How to manage 1 billion + inserts a day into a MySQL resource4
    5. 5. INNODBHuge scaling potentialTransactionalTuneableWell supported/Strong community How to manage 1 billion + inserts a day into a MySQL resource5
    6. 6. MEASUREMeasure, measure, measureAlways measure as many metrics as possibleDe ne a scopeDecide on metrics that can best determine performance improvements How to manage 1 billion + inserts a day into a MySQL resource6
    7. 7. TOOLSSysbenchCacti (mysql-cacti-templates)IostatIotopmysqltuner.pl How to manage 1 billion + inserts a day into a MySQL resource7
    8. 8. RESOURCESLarge amount of RAM (all machines 100GB+) Our latest server roll outshave minimum of 144GBinnodb_buffer_pool_size=??? How to manage 1 billion + inserts a day into a MySQL resource8
    9. 9. STRIKING A BALANCEMaximum safe RAM utilisationMake the sure buffer pool never swaps How to manage 1 billion + inserts a day into a MySQL resource9
    10. 10. LOAD DATA INFILEOur application pulls messages from the queue and creates les forimportingHave a well optimised perl application that loads the data into MySQLInserting data in a batch gave great improvements How to manage 1 billion + inserts a day into a MySQL resource10
    11. 11. OTHER SETTINGSinnodb_ le_per_table=1innodb_ ush_log_at_trx_commit=2innodb_ ush_method=O_DIRECTinnodb_log_ le_size=1G How to manage 1 billion + inserts a day into a MySQL resource11
    12. 12. SANOpen-iSCSI settings:node.session.cmds_maxnode.session.queue_depth How to manage 1 billion + inserts a day into a MySQL resource12
    13. 13. MESSAGE QUEUEMakes it much easier to plan downtimeCan comfortably tweak the live systemCan pull data in different directions for testing etcFast way to receive data How to manage 1 billion + inserts a day into a MySQL resource13
    14. 14. SUMMARY www.samlambert.com @isamlambert Questions? How to manage 1 billion + inserts a day into a MySQL resource14
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×