Your SlideShare is downloading. ×
How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk

818

Published on

How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk presented at Super Mondays October http://www.supermondays.org/2011/10/24/lightning-talks-october-31st-2011/

How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk presented at Super Mondays October http://www.supermondays.org/2011/10/24/lightning-talks-october-31st-2011/

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
818
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Transcript

    • 1. HOW TO MANAGE 1 BILLION+ INSERTS A DAY INTO A MYSQL RESOURCE Sam Lambert, October, 2011 PHP & Continuous Data Processing, PHPNW 20111
    • 2. SMITH ELECTRIC VEHICLES & TELEMETRY • Worlds largest manufacturer of Commercial, all-electric vehicles • Smith Link – on-board vehicle telematics system, capturing over 2500 data points each second on the vehicle and broadcasting them over mobile network • ~400 telemetry enabled vehicles on the road • Worlds largest telemetry project outside of F1 • Number of inserts per day increases by around 250,000,000 per month • Record on truck all the way to our web interface in a maximum of 30 seconds How to manage 1 billion + inserts a day into a MySQL resource2
    • 3. SYSTEM ARCHITECTURE How to manage 1 billion + inserts a day into a MySQL resource3
    • 4. ORIGINAL PROBLEMSUnable to insert data quick enoughProblems with concurrently processing dataData processing scripts taking an unacceptable amount of timeNo obvious path for scaling How to manage 1 billion + inserts a day into a MySQL resource4
    • 5. INNODBHuge scaling potentialTransactionalTuneableWell supported/Strong community How to manage 1 billion + inserts a day into a MySQL resource5
    • 6. MEASUREMeasure, measure, measureAlways measure as many metrics as possibleDe ne a scopeDecide on metrics that can best determine performance improvements How to manage 1 billion + inserts a day into a MySQL resource6
    • 7. TOOLSSysbenchCacti (mysql-cacti-templates)IostatIotopmysqltuner.pl How to manage 1 billion + inserts a day into a MySQL resource7
    • 8. RESOURCESLarge amount of RAM (all machines 100GB+) Our latest server roll outshave minimum of 144GBinnodb_buffer_pool_size=??? How to manage 1 billion + inserts a day into a MySQL resource8
    • 9. STRIKING A BALANCEMaximum safe RAM utilisationMake the sure buffer pool never swaps How to manage 1 billion + inserts a day into a MySQL resource9
    • 10. LOAD DATA INFILEOur application pulls messages from the queue and creates les forimportingHave a well optimised perl application that loads the data into MySQLInserting data in a batch gave great improvements How to manage 1 billion + inserts a day into a MySQL resource10
    • 11. OTHER SETTINGSinnodb_ le_per_table=1innodb_ ush_log_at_trx_commit=2innodb_ ush_method=O_DIRECTinnodb_log_ le_size=1G How to manage 1 billion + inserts a day into a MySQL resource11
    • 12. SANOpen-iSCSI settings:node.session.cmds_maxnode.session.queue_depth How to manage 1 billion + inserts a day into a MySQL resource12
    • 13. MESSAGE QUEUEMakes it much easier to plan downtimeCan comfortably tweak the live systemCan pull data in different directions for testing etcFast way to receive data How to manage 1 billion + inserts a day into a MySQL resource13
    • 14. SUMMARY www.samlambert.com @isamlambert Questions? How to manage 1 billion + inserts a day into a MySQL resource14

    ×