Running memcached clusters  on Amazon EC2  In The Brain of Daniel Sikar Skills Matter – London, UK 01.12.2010
Skills Matter supports the Agile and Open Source developer community, by organising free events, training courses, confere...
Daniel Sikar Consultant Developer Trainer Internet & scientific data acquisition applications Runs Amazon Web Services, Or...
 
LCD memcached? High performance memory object caching system Clusters? Group of computers working together AWS? Infrastruc...
Caching
Caching... To name a few: <ul><li>Application cache
CPU cache
Database query cache </li></ul>
Caching... Used to decrease latency.
Clusters
Clusters... Used to increase: <ul><li>Availability
Reliability
Scalability </li></ul>
Clusters... If a unit falls off, the cluster continues behaving the same way. Units can be added and subtracted as needed.
Clusters... The cluster functions as a whole.
Infrastructure as a Service
AWS ~ IAAS The model: <ul><li>No long term contract
On demand
Pay as you go </li></ul>
EC2
Amazon Machine Image (AMI) >  Instance > EC2 Key Concepts I
One AMI can launch one or more instances EC2 Key Concepts II
 
$ wget http://(..)ec2-api-tools.zip # unpack to /usr # config env vars $ env | grep EC2 (...) EC2 tools
EC2 tools $ ls /usr/bin/ec2* $ ls /usr/lib/ec2* $ ec2ver $ ec2dim -a > amis.txt $ cat amis.txt | grep <...>
$ ec2auth -h $ ec2addgrp -h $ ec2dgrp -h EC2 – Security Groups (firewall)
$ ec2addgrp dbgroup -d &quot;db security group&quot; $ ec2addgrp mcgroup -d &quot;mc security group&quot; $ ec2addgrp wbgr...
EC2 – Security Groups (firewall)
$ ec2auth dbgroup -o wbgroup -u <account id> $ ec2auth wbgroup -o dbgroup -u <account id> $ ec2auth wbgroup -o mcgroup -u ...
EC2 – Security Groups (firewall)
$ ec2auth dbgroup -p 3306 -s 0.0.0.0/0 $ ec2auth wbgroup -p 80 -s 0.0.0.0/0 $ ec2auth wbgroup -p 22 -s 0.0.0.0/0 $ ec2auth...
AMI Instance  Running EC2 Instances
Key Pairs
Generating a key pair $  ec2addkey  aws_key > ~/.ssh/aws_key.pem $ chmod 600 ~/.ssh/aws_key.pem
Running EC2 instances $  ec2run  ami-f4340180 -g dbgroup -t t1.micro  -z eu-west-1a -k aws_key # plus one web and one memc...
Configuring a database server $ sudo apt-get install mysql server $ sudo vim /etc/mysql/my.cnf #bind-address = 127.0.0.1 b...
 
Configuring a memcached server $ ssh -i ~/.ssh/aws_key.pem <user>@<dns> $ sudo apt-get update $ sudo apt-get install memca...
Memcached clients <ul><li>C / C++
PHP
Java
Python
Ruby
Upcoming SlideShare
Loading in...5
×

Itb session v_memcached

1,165

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,165
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Itb session v_memcached

  1. 1. Running memcached clusters on Amazon EC2 In The Brain of Daniel Sikar Skills Matter – London, UK 01.12.2010
  2. 2. Skills Matter supports the Agile and Open Source developer community, by organising free events, training courses, conferences and by publishing thousands of podcasts on ideas and technologies that drive innovation. <ul>116-120 Goswell Road, London, EC1V 7DP Tel: +44 (0)207 183 9040 [email_address] </ul>Welcome to Skills Matter
  3. 3. Daniel Sikar Consultant Developer Trainer Internet & scientific data acquisition applications Runs Amazon Web Services, Oracle Coherence and Intro to Hadoop courses at Skills Matter [email_address]
  4. 5. LCD memcached? High performance memory object caching system Clusters? Group of computers working together AWS? Infrastructure as a Service
  5. 6. Caching
  6. 7. Caching... To name a few: <ul><li>Application cache
  7. 8. CPU cache
  8. 9. Database query cache </li></ul>
  9. 10. Caching... Used to decrease latency.
  10. 11. Clusters
  11. 12. Clusters... Used to increase: <ul><li>Availability
  12. 13. Reliability
  13. 14. Scalability </li></ul>
  14. 15. Clusters... If a unit falls off, the cluster continues behaving the same way. Units can be added and subtracted as needed.
  15. 16. Clusters... The cluster functions as a whole.
  16. 17. Infrastructure as a Service
  17. 18. AWS ~ IAAS The model: <ul><li>No long term contract
  18. 19. On demand
  19. 20. Pay as you go </li></ul>
  20. 21. EC2
  21. 22. Amazon Machine Image (AMI) > Instance > EC2 Key Concepts I
  22. 23. One AMI can launch one or more instances EC2 Key Concepts II
  23. 25. $ wget http://(..)ec2-api-tools.zip # unpack to /usr # config env vars $ env | grep EC2 (...) EC2 tools
  24. 26. EC2 tools $ ls /usr/bin/ec2* $ ls /usr/lib/ec2* $ ec2ver $ ec2dim -a > amis.txt $ cat amis.txt | grep <...>
  25. 27. $ ec2auth -h $ ec2addgrp -h $ ec2dgrp -h EC2 – Security Groups (firewall)
  26. 28. $ ec2addgrp dbgroup -d &quot;db security group&quot; $ ec2addgrp mcgroup -d &quot;mc security group&quot; $ ec2addgrp wbgroup -d &quot;wb security group&quot; EC2 – Security Groups (firewall)
  27. 29. EC2 – Security Groups (firewall)
  28. 30. $ ec2auth dbgroup -o wbgroup -u <account id> $ ec2auth wbgroup -o dbgroup -u <account id> $ ec2auth wbgroup -o mcgroup -u <account id> $ ec2auth mcgroup -o wbgroup -u <account id> EC2 – Security Groups (firewall)
  29. 31. EC2 – Security Groups (firewall)
  30. 32. $ ec2auth dbgroup -p 3306 -s 0.0.0.0/0 $ ec2auth wbgroup -p 80 -s 0.0.0.0/0 $ ec2auth wbgroup -p 22 -s 0.0.0.0/0 $ ec2auth dbgroup -p 22 -s 0.0.0.0/0 $ ec2auth mcgroup -p 22 -s 0.0.0.0/0 $ ec2dgrp dbgroup mcgroup wbgroup EC2 – Security Groups (firewall)
  31. 33. AMI Instance Running EC2 Instances
  32. 34. Key Pairs
  33. 35. Generating a key pair $ ec2addkey aws_key > ~/.ssh/aws_key.pem $ chmod 600 ~/.ssh/aws_key.pem
  34. 36. Running EC2 instances $ ec2run ami-f4340180 -g dbgroup -t t1.micro -z eu-west-1a -k aws_key # plus one web and one memcached server $ ssh -i ~/.ssh/aws_key.pem <user>@<dns>
  35. 37. Configuring a database server $ sudo apt-get install mysql server $ sudo vim /etc/mysql/my.cnf #bind-address = 127.0.0.1 bind-address = 0.0.0.0 # notice caching configuration option $ sudo /etc/init.d/mysql restart $ mysql -u root -p mysql> create user 'dbuser'@'%' identified by <pwd> mysql> grant all privileges on *.* to 'dbuser'@'%' # add some data
  36. 39. Configuring a memcached server $ ssh -i ~/.ssh/aws_key.pem <user>@<dns> $ sudo apt-get update $ sudo apt-get install memcached $ memcached -h | head $ ps aux | grep memcached $ sudo vim /etc/memcached.conf # mods $ sudo /etc/init.d/memcached restart $ ps aux | grep memcached
  37. 40. Memcached clients <ul><li>C / C++
  38. 41. PHP
  39. 42. Java
  40. 43. Python
  41. 44. Ruby
  42. 45. Perl
  43. 46. .NET </li></ul><ul><li>MySQL
  44. 47. PostgreSQL
  45. 48. Erlang
  46. 49. Lua
  47. 50. Lisp
  48. 51. Cold Fusion
  49. 52. OCaml
  50. 53. (...) </li></ul>
  51. 54. Running a memcache(d) php client $ sudo apt-get install php5-memcached $ sudo apt-get install libmemcached-dev $ sudo apt-get install php5-dev php-pear $ sudo apt-get install php5-mysql $ sudo pecl install memcache $ sudo vim /etc/php5/apache2/php.ini extension=memcache.so $ sudo /etc/init.d/apache2 restart $ wget http://localhost/phpinfo.php $ cat phpinfo.php | grep memcache # copy files
  52. 55. What about the memcache cluster?
  53. 56. AWS workflow
  54. 57. Creating a memcached AMI $ ec2din (...) <volume id> (...) $ ec2addsnap <volume id> (...) <snapshot id> (...) $ ec2dim ami-f4340180 $ ec2reg -s <snapshot id> --kernel <kernel id> -n &quot;name&quot; -d &quot;desc&quot; $ ec2dim <ami id> $ ec2run <ami id> -g mcgroup -t t1.micro -z eu-west-1a -k aws_key -n 2 # add to cluster & test
  55. 58. Tidying up $ ec2kill <instance id>
  56. 59. Redemployment gotcha <ul><li>Different ip addresses </li></ul>
  57. 60. Conclusion Setting up and running memcached clusters is easy on IAAS environments such as Amazon EC2
  58. 61. Questions?
  59. 62. Coming up next: Intro to Hadoop 20.12.2010 Amazon Web Services 24.01.2011 Skills Matter – London, UK
  60. 63. Thank you for attending! Running memcached clusters on Amazon EC2 In The Brain of Daniel Sikar Skills Matter – London, UK 01.12.2010

×