Hands On WithAmazon Web ServicesStirTrek 2011May 2011<br />Rob Gillen<br />Cloud Computing Research<br />rob@gillenfamily....
Today	<br />Introduction<br />Amazon Web Services Offerings<br />EC2, S3, SQS, SDB and other TLAs<br />Discussion of recen...
Who am I?<br />Software development background<br />11+ Years with Planet:<br />Senior Developer/Solutions Architect<br />...
AWS Products<br />Elastic Compute Cloud<br />Elastic Map Reduce<br />Auto Scaling<br />Cloud Front<br />SimpleDB<br />Rela...
Amazon S3 Concepts<br />Objects:<br />Opaque data to be stored (1 byte … 5 Gigabytes)<br />Authentication and access contr...
Amazon SQS Concepts<br />Queues:<br />Named message container<br />Persistent<br />Messages:<br />Up to 256KB of data per ...
Amazon EC2 Concepts<br />Amazon Machine Image (AMI):<br />Bootable root disk<br />Pre-defined or user-built<br />Catalog o...
EBS Boot vs. S3 Boot<br />S3 (original)<br />Image copied from S3 to Ephemeral disks<br />Lower IO charges during runtime<...
Tooling<br />Command Line<br />Browser Plug-ins<br />3rd Party Tools<br />Broad SDK Support<br />AWS Management Console<br />
CloudBerry Explorer<br />
AWS Management Console<br />
Stock Windows Instances<br />
Machine Setup Walkthrough<br />
What Happened<br />Incorrect Network Change == Replication Storm for EC2<br />“Affected Availability Zone”<br />Actually a...
Regions and AZs<br />
My AZ != Your AZ<br />
Fallout<br />Survivors<br />NetFlix (Chaos Monkey)<br />SmugMug (Avoids EBS)<br />Recovery.gov, Treasury.gov, others<br />...
Building Blocks for HA<br />Use Elastic IP Addresses for consistent and re-mappable routes<br />Use multiple Amazon EC2 Av...
Build Loosely-Coupled Systems<br />Use independent components<br />Design everything as a black box<br />Load balance and ...
Articles of Interest<br />Amazon:<br />http://aws.amazon.com/message/65648/<br />ShareThis:<br />http://blog.sharethis.com...
Cloud Applications<br />
Computer Forensics<br />
Forensic Toolkit 3<br />Cloud Formation<br />Define deployment environment<br />Bill on Behalf Of<br />Selectable Instance...
Challenge – What to do with mounds of data?<br />What is in there?<br />Are there any threats?<br />What am I missing?<br ...
Trees<br />Forest<br />Can’t See the<br />for the<br />Traditionally, search methods are used to find information at high ...
Keyword Methods<br />Document 1<br />The Army needs sensor technology to help find improvised explosive devices<br />Vecto...
Textual Clustering<br />Vector Space Model<br />Cluster Analysis<br />Similarity Matrix<br />D1<br />D2<br />D3<br />Docum...
Example: Sign of the Crescent1<br />41 Short intelligence reports about a multi-prong terrorist attack<br />Example:<br />...
Piranha Cluster View<br />Report Date: 1 April, 2003. FBI: Abdul Ramazi is the owner of the Select Gourmet Foods shop in S...
Term network with document links<br />1) Report Date: 1 April, 2003. FBI: Abdul Ramaziis the owner of the Select Gourmet F...
Split Tree Over Nodes/Queues<br />
Fault Tolerance<br />
Component Overview<br />
Topics<br />Queue Interactions<br />Blob Interactions<br />Simple DB Interactions<br />
Q & A<br />rob@gillenfamily.net<br />http://rob.gillenfamily.net<br />Please Rate This Talk:<br />http://spkr8.com/t/7409<...
Upcoming SlideShare
Loading in...5
×

Hands On with Amazon Web Services (StirTrek)

2,350

Published on

Hands On AWS session from StirTrek 2011

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,350
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "Hands On with Amazon Web Services (StirTrek)"

  1. 1. Hands On WithAmazon Web ServicesStirTrek 2011May 2011<br />Rob Gillen<br />Cloud Computing Research<br />rob@gillenfamily.net<br />
  2. 2. Today <br />Introduction<br />Amazon Web Services Offerings<br />EC2, S3, SQS, SDB and other TLAs<br />Discussion of recent AWS Outage<br />Applications on AWS<br />Questions<br />
  3. 3. Who am I?<br />Software development background<br />11+ Years with Planet:<br />Senior Developer/Solutions Architect<br /> Federal, Commercial, Foreign, Non-Profit<br />4+ Years at Oak Ridge National Laboratory<br />Software Solutions Architect<br />Cloud Computing/HPC Research<br />
  4. 4. AWS Products<br />Elastic Compute Cloud<br />Elastic Map Reduce<br />Auto Scaling<br />Cloud Front<br />SimpleDB<br />Relational Database Service<br />Simple Storage Svc<br />Elastic Block Storage<br />Simple Queue Service<br />Simple Notification Service<br />CloudWatch<br />Virtual Private Cloud<br />Elastic Load Balancing<br />Import/Export<br />
  5. 5. Amazon S3 Concepts<br />Objects:<br />Opaque data to be stored (1 byte … 5 Gigabytes)<br />Authentication and access controls<br />Buckets:<br />Object container – any number of objects<br />100 buckets per account / buckets are “owned”<br />Keys:<br />Unique object identifier within bucket<br />Up to 1024 bytes long<br />Flat object storage model<br />Standards-Based Interfaces:<br />REST and SOAP<br />URL-Addressability – every object has a URL<br />
  6. 6. Amazon SQS Concepts<br />Queues:<br />Named message container<br />Persistent<br />Messages:<br />Up to 256KB of data per message<br />Peek / Lock access model<br />Scalable:<br />Unlimited number of queues per account<br />Unlimited number of messages per queue<br />
  7. 7. Amazon EC2 Concepts<br />Amazon Machine Image (AMI):<br />Bootable root disk<br />Pre-defined or user-built<br />Catalog of user-built AMIs<br />OS: Fedora, Centos, Gentoo, Debian, Ubuntu, Windows Server<br />App Stack: LAMP, mpiBLAST, Hadoop<br />Instance:<br />Running copy of an AMI<br />Launch in less than 2 minutes<br />Start/stop programmatically<br />Network Security Model:<br />Explicit access control<br />Security groups<br />Inter-service bandwidth is free<br />
  8. 8. EBS Boot vs. S3 Boot<br />S3 (original)<br />Image copied from S3 to Ephemeral disks<br />Lower IO charges during runtime<br />Target: farm of identical machines<br />EBS (newer)<br />Auto Saving (sort of)<br />Performance<br />Ephemeral disks not used<br />Potential higher IO charges<br />Target: single machines<br />Be certain you understand Shutdown & Termination<br />and the ramifications of such<br />
  9. 9. Tooling<br />Command Line<br />Browser Plug-ins<br />3rd Party Tools<br />Broad SDK Support<br />AWS Management Console<br />
  10. 10. CloudBerry Explorer<br />
  11. 11. AWS Management Console<br />
  12. 12. Stock Windows Instances<br />
  13. 13. Machine Setup Walkthrough<br />
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30. What Happened<br />Incorrect Network Change == Replication Storm for EC2<br />“Affected Availability Zone”<br />Actually affected *multiple* AZs (initially)<br />"Each availability zone runs on its own physically distinct, independent infrastructure, and is engineered to be highly reliable. Common points of failures like generators and cooling equipment are not shared across Availability Zones. Additionally, they are physically separate, such that even extremely uncommon disasters such as fires, tornados or flooding would only affect a single Availability Zone."  http://aws.amazon.com/ec2/faqs/#How_isolated_are_Availability_Zones_from_one_another<br />
  31. 31. Regions and AZs<br />
  32. 32. My AZ != Your AZ<br />
  33. 33. Fallout<br />Survivors<br />NetFlix (Chaos Monkey)<br />SmugMug (Avoids EBS)<br />Recovery.gov, Treasury.gov, others<br />ShareThis<br />Casualties<br />Reddit, Foursquare, Quora, Hootsuite<br />OpenEI.org<br />
  34. 34. Building Blocks for HA<br />Use Elastic IP Addresses for consistent and re-mappable routes<br />Use multiple Amazon EC2 Availability Zones<br />Replicate data across multiple Azs<br />Example: Amazon RDS in Multi-AZ mode<br />Use Amazon EBS for persistent file systems<br />Take EBS snapshots and use S3 for backup<br />
  35. 35. Build Loosely-Coupled Systems<br />Use independent components<br />Design everything as a black box<br />Load balance and scale clusters<br />Think about graceful degradation<br />Use queues to isolate components<br />
  36. 36. Articles of Interest<br />Amazon:<br />http://aws.amazon.com/message/65648/<br />ShareThis:<br />http://blog.sharethis.com/2011/04/27/aws-outage-sharethis-survives/<br />http://tipsforsocialmedia.blogspot.com/2011/04/amazon-ec2-troubles-bring-down-reddit.html<br />EC2 Disabled<br />http://ec2disabled.com/<br />Medical Malpractice<br />https://forums.aws.amazon.com/thread.jspa?threadID=65649<br />NetFlix<br />http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.html<br />http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html<br />
  37. 37. Cloud Applications<br />
  38. 38. Computer Forensics<br />
  39. 39. Forensic Toolkit 3<br />Cloud Formation<br />Define deployment environment<br />Bill on Behalf Of<br />Selectable Instance Size<br />
  40. 40. Challenge – What to do with mounds of data?<br />What is in there?<br />Are there any threats?<br />What am I missing?<br />How do I connect the “dots”?<br />How do I find the relevant information I need?<br />
  41. 41. Trees<br />Forest<br />Can’t See the<br />for the<br />Traditionally, search methods are used to find information at high volume levels<br />But, those methods won’t get you here easily<br />
  42. 42. Keyword Methods<br />Document 1<br />The Army needs sensor technology to help find improvised explosive devices<br />Vector Space Model<br />Term List<br />Army<br />Sensor<br />Technology<br />Help<br />Find<br />Improvise<br />Explosive <br />Device<br />ORNL <br />develop <br />homeland <br />Defense<br />Mitre <br />won <br />contract <br />Weight Terms<br />Document 2<br />ORNL has developed sensor technology for homeland defense<br />Document 3<br />Mitre has won a contract to develop homeland defense sensors for explosive devices<br />An index into the document list<br />Term Frequency – Inverse Document Frequency<br />
  43. 43. Textual Clustering<br />Vector Space Model<br />Cluster Analysis<br />Similarity Matrix<br />D1<br />D2<br />D3<br />Documents to Documents<br />Most similar documents<br />TFIDF<br />Euclidean distance<br />Time Complexity<br />O(n2Log n)<br />
  44. 44. Example: Sign of the Crescent1<br />41 Short intelligence reports about a multi-prong terrorist attack<br />Example:<br />Report Date: 1 April, 2003. FBI: Abdul Ramazi is the owner of the Select Gourmet Foods shop in Springfield Mall, Springfield, VA. [Phone number 703-659-2317]. First Union National Bank lists Select Gourmet Foods as holding account number 1070173749003. Six checks totaling $35,000 have been deposited in this account in the past four months and are recorded as having been drawn on accounts at the Pyramid Bank of Cairo, Egypt and the Central Bank of Dubai, United Arab Emirates. Both of these banks have just been listed as possible conduits in money laundering schemes<br />1Intelligence Analysis Case Study by F. J. Hughes, Joint Military Intelligence College<br />
  45. 45. Piranha Cluster View<br />Report Date: 1 April, 2003. FBI: Abdul Ramazi is the owner of the Select Gourmet Foods shop in Springfield Mall, Springfield, VA. [Phone number 703-659-2317]. First Union National Bank lists Select Gourmet Foods as holding account number 1070173749003. Six checks totaling $35,000 have been deposited in this account in the past four months and are recorded as having been drawn on accounts at the Pyramid Bank of Cairo, Egypt and the Central Bank of Dubai, United Arab Emirates. Both of these banks have just been listed as possible conduits in money laundering schemes<br />
  46. 46. Term network with document links<br />1) Report Date: 1 April, 2003. FBI: Abdul Ramaziis the owner of the Select Gourmet Foods shop in Springfield Mall, Springfield, VA. [Phone number 703-659-2317]. First Union National Bank lists Select Gourmet Foods as holding account number 1070173749003. Six checks totaling $35,000 have been deposited in this account in the past four months and are recorded as having been drawn on accounts at the Pyramid Bank of Cairo, Egypt and the Central Bank of Dubai, United Arab Emirates. Both of these banks have just been listed as possible conduits in money laundering schemes.<br />
  47. 47. Split Tree Over Nodes/Queues<br />
  48. 48. Fault Tolerance<br />
  49. 49.
  50. 50.
  51. 51.
  52. 52. Component Overview<br />
  53. 53.
  54. 54. Topics<br />Queue Interactions<br />Blob Interactions<br />Simple DB Interactions<br />
  55. 55. Q & A<br />rob@gillenfamily.net<br />http://rob.gillenfamily.net<br />Please Rate This Talk:<br />http://spkr8.com/t/7409<br />

×