C O N C U R R E N C Y
A N I N T R O D U C T I O N T O
J O H N E N N E W, S O L U T I O N S A R C H I T E C T, D E E S O N ...
W H AT I S
C O N C U R R E N C Y ?
Database
Filestore
Web service
Application
Database
Filestore
Web service
Application
B R E A K I N G T H E
P R O B L E M D O W N …
– E D WA R D A . L E E
“… a folk definition of insanity is to do the same
thing over and over again and to expect the resu...
T H E D I N I N G P H I L O S O P H E R S
H T T P : / / D E E S O N - O N L I N E . C O . U K / L A B S / M U LT I - P R O...
• lock_acquire()
• drush_invoke_process()
• DB transactions (innodb)
D R U PA L 7 T O O L S
M A K I N G M I G R AT E
M O V E …
W H Y S P E E D U P M I G R A T I O N S ?
• Reduce the time a migration takes
• Discover and fix problems quicker
• Remigr...
T H E P R O C E S S …
• https://github.com/johnennewdeeson/drush-multi-
processing
• Download mt.drush.inc and place in si...
Configuring your migration class
Configuring your migration class
Creating a Drush command
Starting the process thread manager
Process setup …
Process teardown …
L I V E D E M O …
F I N E T U N I N G …
• Ramp up slowly - monitor server load and db load
• /etc/my.cnf

innodb_locks_unsafe_for_binlog = 1...
T H A N K S !
… A N Y Q U E S T I O N S ?
J O H N E N N E W, S O L U T I O N S A R C H I T E C T, D E E S O N O N L I N E
...
Drupal camp london 2014 John Ennew Concurrency Session Deeson Online
Drupal camp london 2014 John Ennew Concurrency Session Deeson Online
Drupal camp london 2014 John Ennew Concurrency Session Deeson Online
Upcoming SlideShare
Loading in …5
×

Drupal camp london 2014 John Ennew Concurrency Session Deeson Online

1,008 views

Published on

Concurrency in Drupal is the description and demonstration of making migrate faster. The slides cover the basics of why concurrency makes programs faster and how to wrapper drush and Migrate to make it execute in parallel.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,008
On SlideShare
0
From Embeds
0
Number of Embeds
614
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Drupal camp london 2014 John Ennew Concurrency Session Deeson Online

  1. 1. C O N C U R R E N C Y A N I N T R O D U C T I O N T O J O H N E N N E W, S O L U T I O N S A R C H I T E C T, D E E S O N O N L I N E C E N G O N D . O . @ D E E S O N _ L A B S H T T P : / / W W W. D E E S O N - O N L I N E . C O . U K / L A B S … O R H O W I M A D E M I G R AT E FA S T E R
  2. 2. W H AT I S C O N C U R R E N C Y ?
  3. 3. Database Filestore Web service Application
  4. 4. Database Filestore Web service Application
  5. 5. B R E A K I N G T H E P R O B L E M D O W N …
  6. 6. – E D WA R D A . L E E “… a folk definition of insanity is to do the same thing over and over again and to expect the results to be different. By this definition, we in fact require that programmers of multithreaded systems be insane. Were they sane, they could not understand their programs …” T H E P R O B L E M W I T H T H R E A D S , 2 0 0 6 T H E H T T P : / / W W W. E E C S . B E R K E L E Y. E D U / P U B S / T E C H R P T S / 2 0 0 6 / E E C S - 2 0 0 6 - 1 . H T M L W I T H T H R E A D S , 2 0 0 6
  7. 7. T H E D I N I N G P H I L O S O P H E R S H T T P : / / D E E S O N - O N L I N E . C O . U K / L A B S / M U LT I - P R O C E S S I N G - PA R T- 1 - H O W - M A K E - D R U S H - R U S H
  8. 8. • lock_acquire() • drush_invoke_process() • DB transactions (innodb) D R U PA L 7 T O O L S
  9. 9. M A K I N G M I G R AT E M O V E …
  10. 10. W H Y S P E E D U P M I G R A T I O N S ? • Reduce the time a migration takes • Discover and fix problems quicker • Remigrate faster / regular migrations
  11. 11. T H E P R O C E S S … • https://github.com/johnennewdeeson/drush-multi- processing • Download mt.drush.inc and place in sites/all/drush • Update to migrate 7.x-2.6-rc1 • Configure your migration class for batched operation • Create a custom drush import command (or modify mtm.drush.inc also at github URL above) • Full instructions online at: http://deeson-online.co.uk/labs/ multi-processing-part-2-how-make-migrate-move
  12. 12. Configuring your migration class
  13. 13. Configuring your migration class
  14. 14. Creating a Drush command
  15. 15. Starting the process thread manager
  16. 16. Process setup …
  17. 17. Process teardown …
  18. 18. L I V E D E M O …
  19. 19. F I N E T U N I N G … • Ramp up slowly - monitor server load and db load • /etc/my.cnf
 innodb_locks_unsafe_for_binlog = 1
 innodb_flush_log_at_trx_commit=2 • Use alternative cache mechanism to ease your database load (e.g. memcache) • $conf['lock_inc'] = 'sites/all/modules/memcache/ memcache-lock.inc'; • $conf['memcache_stampede_protection'] = TRUE;
  20. 20. T H A N K S ! … A N Y Q U E S T I O N S ? J O H N E N N E W, S O L U T I O N S A R C H I T E C T, D E E S O N O N L I N E C E N G O N D . O . @ D E E S O N _ L A B S H T T P : / / W W W. D E E S O N - O N L I N E . C O . U K / L A B S

×