• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk
 

How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk

on

  • 854 views

How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk presented at Super Mondays October http://www.supermondays.org/2011/10/24/lightning-talks-october-31st-2011/

How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk presented at Super Mondays October http://www.supermondays.org/2011/10/24/lightning-talks-october-31st-2011/

Statistics

Views

Total Views
854
Views on SlideShare
639
Embed Views
215

Actions

Likes
0
Downloads
1
Comments
0

3 Embeds 215

http://www.supermondays.org 206
http://www.supermondays.co.uk 5
http://lanyrd.com 4

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk How to manage 1 Billion+ Inserts a day into a MySQL resource -Lightning talk Presentation Transcript

  • HOW TO MANAGE 1 BILLION+ INSERTS A DAY INTO A MYSQL RESOURCE Sam Lambert, October, 2011 PHP & Continuous Data Processing, PHPNW 20111
  • SMITH ELECTRIC VEHICLES & TELEMETRY • Worlds largest manufacturer of Commercial, all-electric vehicles • Smith Link – on-board vehicle telematics system, capturing over 2500 data points each second on the vehicle and broadcasting them over mobile network • ~400 telemetry enabled vehicles on the road • Worlds largest telemetry project outside of F1 • Number of inserts per day increases by around 250,000,000 per month • Record on truck all the way to our web interface in a maximum of 30 seconds How to manage 1 billion + inserts a day into a MySQL resource2
  • SYSTEM ARCHITECTURE How to manage 1 billion + inserts a day into a MySQL resource3
  • ORIGINAL PROBLEMSUnable to insert data quick enoughProblems with concurrently processing dataData processing scripts taking an unacceptable amount of timeNo obvious path for scaling How to manage 1 billion + inserts a day into a MySQL resource4
  • INNODBHuge scaling potentialTransactionalTuneableWell supported/Strong community How to manage 1 billion + inserts a day into a MySQL resource5
  • MEASUREMeasure, measure, measureAlways measure as many metrics as possibleDe ne a scopeDecide on metrics that can best determine performance improvements How to manage 1 billion + inserts a day into a MySQL resource6
  • TOOLSSysbenchCacti (mysql-cacti-templates)IostatIotopmysqltuner.pl How to manage 1 billion + inserts a day into a MySQL resource7
  • RESOURCESLarge amount of RAM (all machines 100GB+) Our latest server roll outshave minimum of 144GBinnodb_buffer_pool_size=??? How to manage 1 billion + inserts a day into a MySQL resource8
  • STRIKING A BALANCEMaximum safe RAM utilisationMake the sure buffer pool never swaps How to manage 1 billion + inserts a day into a MySQL resource9
  • LOAD DATA INFILEOur application pulls messages from the queue and creates les forimportingHave a well optimised perl application that loads the data into MySQLInserting data in a batch gave great improvements How to manage 1 billion + inserts a day into a MySQL resource10
  • OTHER SETTINGSinnodb_ le_per_table=1innodb_ ush_log_at_trx_commit=2innodb_ ush_method=O_DIRECTinnodb_log_ le_size=1G How to manage 1 billion + inserts a day into a MySQL resource11
  • SANOpen-iSCSI settings:node.session.cmds_maxnode.session.queue_depth How to manage 1 billion + inserts a day into a MySQL resource12
  • MESSAGE QUEUEMakes it much easier to plan downtimeCan comfortably tweak the live systemCan pull data in different directions for testing etcFast way to receive data How to manage 1 billion + inserts a day into a MySQL resource13
  • SUMMARY www.samlambert.com @isamlambert Questions? How to manage 1 billion + inserts a day into a MySQL resource14