Cloud Computing in PHP With the Amazon Web Services


Published on

Presentation delivered on April 30, 2010 to the Linux Fest in Bellingham, Washington.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • First, it’s useful to provide the context that the way we think about what is, the way we think about it at the highest levels of the Company, is that we have three macro and distinct businesses: our Consumer/Retail business, our Seller business, and our Developer business.
  • Amazon Web Services is a set of primitive building blocks to enable developers to build applications and, ultimately, businesses, on our battle-tested infrastructure. So what kinds of building blocks do developers need?

    First, you need compute resources. The Elastic Compute Cloud, or EC2, allows developers to rent virtual computers, running in our data centers, and pay for them by the hour. Provision as many as you need – 1, 10, 1000, for as long as you need them, and you pay only for what you use (starting at 10 cents/hour).

    Next, you probably need somewhere to durably store and retrieve data. The Simple Storage Service, is, well, a simple storage service that allows to store and retrieve objects in the cloud using simple put and get API calls.

    You might need to index and query for that data, or create other structured data that you want to query, which is what we built SimpleDB.
  • On the application platforms point, good to mention:
    Microsoft .NET
    IBM WebSphere
    Oracle Fusion Middleware

    Also a good place to mention IBM AMIs (bring your own license or pay by the hour)
  • SimpleDB is a very easy to use database service that can be used for a variety of tasks, including indexing information stored in S3, maintaining access and other types of logs for your applications, and storing all kinds of structured data in a reliable, scalable database.

    You can see a quick example of how easy SimpleDB is to use. Here’s your classic products table with an item number, description, and other attributes for each product. You can see the queries we’d use to store and retrieve data from SimpleDB. With SimpleDB, the API for putting data into the database is very flexible. I can add as much data as I’d like to each of the attributes for a given product. Querying the database is just as easy. I specify the “Domain” (or table) and the fields on which I’d like to execute the query. In this case, I’d like to obtain all items in the table whose description matches the given parameter.
  • SQS is an easy to use queueing service. The most typical use case for SQS is to coordinate activity between multiple EC2 instances. Take, for example, a typical web application with a presentation tier, application server or business logic tier, and database tier. You’d typically have multiple EC2 instances on the presentation and application server tiers, which would be working independently to process requests coming to your site.

    Amazon SQS is a queueing service that can provide the glue between your web server and your application server. The most common setup will involve configuring two queues. The first queue will accept messages from the web server hosted on Amazon EC2. Application servers, also hosted on Amazon EC2, will pluck those messages off the queue, process data based on the contents of the message, and then place the equivalent of an “I’m done! Here are the results.” message on the second queue. The web server would then pluck the message off the second queue and return results back to the client that made the initial request. In this way, your Amazon EC2 instances can grow or shrink, startup and fail with impunity, while you can rest assured that all of your data processing happens reliably.
  • Cloud Computing in PHP With the Amazon Web Services

    1. 1. CLOUD COMPUTING IN PHP WITH THE AMAZON WEB SERVICES Jeff Barr, Senior Web Services Evangelist @jeffbarr on Twitter
    2. 2. • Based in Seattle • Unix since 1982; Linux since 1995 • Career path: – Startups – Microsoft – Consultant to VCs and startups – Amazon Web Services • Lead AWS Blogger • Author of “Host Your Website in the Cloud” INTRODUCTION
    3. 3. • Amazon / Cloud / AWS Intro • AWS Overview – Programmable Infrastructure – S3, EC2, SDB, SQS • Programming AWS in PHP – CloudFusion library – Scalable Image Processing – Visualizing Infrastructure – Dynamic Infrastructure • Wrapup / Q & A SESSION OUTLINE
    4. 4. AMAZON’S THREE BUSINESSES Consumer (Retail) Business Tens of millions of active customer accounts Seven countries: US, UK, Germany, Japan, France, Canada, China Seller Business Sell on Amazon websites Use Amazon technology for your own retail website Leverage Amazon’s massive fulfillment center network Developers & IT Professionals On-demand infrastructure for hosting web-scale solutions Hundreds of thousands of registered customers
    5. 5. WHAT IS CLOUD COMPUTING? First, think of your electricity service… Power is available to you on-demand, you pay only for what you use… …and you plug into a vast electrical grid managed by professionals to get you the lowest cost, most reliable power with much greater efficiency and safety than you could probably do on your own.
    6. 6. INTRODUCING AMAZON WEB SERVICES AWS provides flexible, scalable, secure, and cost-effective IT infrastructure for businesses of all sizes around the world. Compute power and storage is available to you on-demand, you pay only for the resources you use… …running on scalable, reliable, and secure infrastructure operated by Amazon Web Services, based on the knowledge gleaned from over a decade of building efficient and dependable infrastructure for
    7. 7. AMAZON WEB SERVICES Compute Amazon Elastic Compute Cloud (EC2) -Elastic Load Balancing -Auto Scaling Storage Amazon Simple Storage Service (S3) -AWS Import/Export Your Custom Applications and Services Content Delivery Amazon CloudFront Messaging Amazon Simple Queue Service (SQS) Amazon Simple Notification Service (SNS) Payments Amazon Flexible Payments Service (FPS) On-Demand Workforce Amazon Mechanical Turk Parallel Processing Amazon Elastic MapReduce Monitoring Amazon CloudWatch Database Amazon RDS Amazon SimpleDB Management AWS Management Console Tools AWS Toolkit for Eclipse Java, PHP, Ruby, Python, .Net Developer Centers Isolated Network Amazon Virtual Private Cloud
    8. 8. • All functionality accessed by APIs • Amazon and third-party libraries • Command-line tools • AWS Management Console • Third-party Tools PROGRAMMABLE INFRASTRUCTURE // Run an instance $EC2 = new AmazonEC2(); $Options = array('KeyName' => "Jeff's Keys", 'InstanceType' => "m1.small"); $Res = $EC2->run_instances("ami-db7b9db2", 1, 1, $Options);
    9. 9. • Scalable data storage in-the-cloud • Highly available and durable • Pay-as-you-go pricing: – Storage: tiered $0.15/GB to $0.055/GB – Data Transfer Out: tiered $0.15/GB to $0.080/GB – Data Transfer In: Free until June 30, 2010 – Requests: nominal charges • Big and busy: – 102 billion objects – 100K requests/second AMAZON SIMPLE STORAGE SERVICE (S3)
    10. 10. • Amazon EC2: on-demand compute power – Obtain and boot new server instances in minutes – Quickly scale capacity up or down • Key features: – Support for Linux, Windows, and OpenSolaris – Supports all major web and application platforms – Deploy across Availability Zones for reliability – Elastic IPs provide greater flexibility – Persistent storage with Amazon Elastic Block Store – Monitoring (CloudWatch), Load Balancing, Auto-Scaling AMAZON ELASTIC COMPUTE CLOUD (EC2)
    11. 11. EC2 PROGRAMMING INTERFACE (API) • Images: – RegisterImage – DescribeImages – DeregisterImage – ModifyImageAttribute – DescribeImageAttribute – ResetImageAttribute • Instances: – RunInstances – DescribeInstances – TerminateInstances – StopInstances – GetConsoleOutput – RebootInstances • IP Addresses: – AllocateAddress – ReleaseAddress – AssociateAddress – DisassociateAddress – DescribeAddresses • Keypairs: – CreateKeyPair – DescribeKeyPairs – DeleteKeyPair • Security Groups: – CreateSecurityGroup – DescribeSecurityGroups – DeleteSecurityGroup – AuthorizeSecurityGroupIngress – RevokeSecurityGroupIngress • Block Storage Volumes: – CreateVolume – DeleteVolume – DescribeVolumes – AttachVolume – DetachVolume – CreateSnapshot – DescribeSnapshots – DeleteSnapshot • VPC: – CreateCustomerGateway – DeleteCustomerGateway – DescribeCustomerGateways – AssociateDhcpOptions – CreateDhcpOptions – DeleteDhcpOptions – DescribeDhcpOptions – CreateSubnet – DeleteSubnet – DescribeSubnets – CreateVpc – DeleteVpc – DescribeVpcs – CreateVpnConnection – DeleteVpnConnection – DescribeVpnConnections – AttachVpnGateway – CreateVpnGateway – DeleteVpnGateway – DescribeVpnGateways – DetachVpnGateway
    12. 12. • Simple, scalable storage solution for structured data – Provides core database functionality for data storage and querying – No schema, no data modeling, no DBA – SQL queries AMAZON SIMPLEDB item description color material 123 Sweater Blue, Red 789 Shoes Black Leather Store: PUT (item, 123), (description, Sweater), (color, Blue), (color, Red) Query: SELECT * FROM Inventory WHERE material='Leather'
    13. 13. • Reliable, highly scalable, hosted queue for messaging • Build automated workflows for all applications • Coordinate multiple Amazon EC2 instances AMAZON SIMPLE QUEUE SERVICE
    14. 14. GETTING STARTED WITH AWS Create Developer Account Enter Payment Information Sign Up for Desired Services Retrieve Private and Public Keys Build & Deploy Application Monitor and Scale Application
    15. 15. MOVING RIGHT ALONG! "Enough with the talk, show us some code, Jeff!"
    16. 16. PROGRAMMING AWS WITH PHP AND CLOUDFUSION We’ll use the CloudFusion libraries from
    17. 17. CLOUDFUSION BASICS  Download from SVN  Add directory to PHP’s include_path  Add AWS keys to define('AWS_KEY', 'J35NTGFCQOIUY3OMNSQQ'); define('AWS_SECRET_KEY', '99pizu2vVOK11rk9UAgWVj7PBGzWwertqJlgLV0c');  Include one file: require_once('cloudfusion.class.php');
    18. 18. CLOUDFUSION XML / PHP SIMPLEXML • CloudFusion methods return a SimpleXML Object: [body] => SimpleXMLElement Object ( [Name] => sitepoint-aws-cloud-book [Prefix] => SimpleXMLElement Object () [Marker] => SimpleXMLElement Object () [MaxKeys] => 1000 [IsTruncated] => false [Contents] => Array ( [0] => SimpleXMLElement Object ( [Key] => images/2008_shiller_housing_projection.jpg • Access as $Res->body->Contents[0]->Key
    20. 20. BUILDING A SCALABLE IMAGE PROCESSING PIPELINE • Fetch a web page, store in Amazon S3 • Parse page and extract links to images • Fetch first 16 images on page, store in Amazon S3 • Render images as composite image
    21. 21. URL Queue Fetch & Store Page Parse Queue Parse Page Image Queue Fetch Images S3 Render Queue Render Images & Pages S3S3 SQS-BASED APPLICATION ARCHITECTURE
    22. 22. ARCHITECTURE ATTRIBUTES • Simple – Each stage is easy to understand – AWS reduces low-level coding • Scalable – Add more processes – Add more instances – Add more stages – Storage any amount of data • Fault Tolerant – Messages remain in queues until processed – Messages reappear in queues if process dies • Asynchronous – Each stage runs at its own speed – Build, test,run one stage at a time
    23. 23. DEFINE QUEUES // Queues define('URL_QUEUE', 'c_url'); define('PARSE_QUEUE', 'c_parse'); define('IMAGE_QUEUE', 'c_image'); define('RENDER_QUEUE', 'c_render'); define('FEED_QUEUE', 'c_feed');
    24. 24. CREATE QUEUES // Create the SQS access object $SQS = new AmazonSQS(); for ($i = 1; $i < count($argv); $i++) { $Queue = $argv[$i]; $Res = $SQS->create_queue($Queue); if ($Res->isOK()) { print("Created queue '${Queue}'n"); } }
    25. 25. CREATE QUEUES & CHECK STATUS $ ./create_queues.php c_url c_parse c_image c_render c_feed $ ./crawl_queue_status.php c_url c_parse c_image c_render ----- ------- ------- -------- 0 0 0 0
    26. 26. LOCATE A QUEUE'S URL // Create the SQS access object $SQS = new AmazonSQS(); // Locate the queue $QueueURL = FindQueueURL($SQS, URL_QUEUE);
    27. 27. POST MESSAGE TO QUEUE // Create message $HistItem = array('Posted by ' . $argv[0] . ' at ' . date('c')); $Message = json_encode(array('Action' => 'FetchPage', 'Origin' => $argv[0], 'Data' => $argv[$i], 'History' => $HistItem)); // Post message $Res = $SQS->send_message($QueueURL, $Message); if ($Res->isOK()) { print("Posted '${Message}' to QueueURL '${QueueURL}'n"); }
    28. 28. RECEIVE MESSAGE FROM QUEUE function PullMessage($SQS, $QueueURL) { while (true) { $Res = $SQS->receive_message($QueueURL); if ($Res->isOk()) { if (IsSet($Res->body->ReceiveMessageResult->Message)) { } else { sleep(1); } } }
    29. 29. RETURN RECEIVED MESSAGE $Message = $Res->body->ReceiveMessageResult->Message; $MessageBody = $Message->Body; $MessageDetail = json_decode($MessageBody, true); $ReceiptHandle = $Message->ReceiptHandle; return array('QueueURL' => $QueueURL, 'Timestamp' => date('c'), 'Message' => $Message, 'MessageBody' => $MessageBody, 'MessageDetail' => $MessageDetail, 'ReceiptHandle' => $ReceiptHandle);
    30. 30. SAMPLE PROCESSING STAGE // Pull, process, post while (true) { // Pull the message from the queue $Message = PullMessage($SQS, $QueueURL); if ($Message != null) { // Extract message detail $MessageDetail = $Message['MessageDetail']; $ReceiptHandle = $Message['ReceiptHandle']; $PageURL = $MessageDetail['Data']; … } }
    31. 31. DETAILED PROCESSING // Fetch the page print("Processing URL '${PageURL}':n"); $HTML = file_get_contents($PageURL); print(" Retrieved " . strlen($HTML) . " bytes of HTMLn"); // Store the page in S3 $Key = 'page_' . md5($PageURL) . '.html'; if (UploadObject($S3, BOOK_BUCKET, $Key, $HTML, S3_ACL_PUBLIC)) { // Get URL in S3 $S3URL = $S3->get_object_url(BOOK_BUCKET, $Key); print(" Uploaded page to S3 as '${Key}'n");
    32. 32. DETAILED PROCESSING // Form message to pass page along to parser $Origin = $MessageDetail['Origin']; $History = $MessageDetail['History']; $History[] = 'Fetched by ' . $argv[0] . ' at ' . date('c'); $Message = json_encode(array('Action' => 'ParsePage', 'Origin' => $Origin, 'Data' => $S3URL, 'PageURL' => $PageURL, 'History' => $History)); // Pass the page along to the parser $Res = $SQS->send_message($QueueParse, $Message);
    33. 33. RENDER IMAGES foreach ($ImageKeys as $ImageKey) { // Fetch the image print(" Fetch image '${ImageKey}'n"); $Image = $S3->get_object(BOOK_BUCKET, $ImageKey); // Convert it to GD format $ImageBits = ImageCreateFromString($Image->body); // Copy it to proper spot in the destination print(" Render image at ${NextX}, ${NextY}n"); ImageCopy($ImageOut, $ImageBits, $NextX, $NextY, 0, 0, ImageSx($ImageBits), ImageSy($ImageBits)); // Update position for next image $NextX += THUMB_SIZE + GAP_SIZE; if (($NextX + THUMB_SIZE) > $OutX) { $NextX = BORDER_LEFT; $NextY += THUMB_SIZE + GAP_SIZE; } }
    35. 35. PROCESSING FLOW Get AWS Metadata Build Object Model Render Object Model as DOT Graph Render Dot Graph as PDF
    36. 36. GET AWS METADATA // Create the service access objects $Service_EC2 = new AmazonEC2); $Service_S3 = new AmazonS3); $Service_SDB = new AmazonSDB); $Service_CF = new AmazonCloudFrontetKey); // Fetch information about all of the EC2 objects $ResElasticIP = $Service_EC2->describe_addresses(); $ResAvailabilityZones = $Service_EC2- >describe_availability_zones(); $ResInstances = $Service_EC2->describe_instances(); $ResVolumes = $Service_EC2->describe_volumes(); $ResSnapshots = $Service_EC2->describe_snapshots();
    37. 37. BUILD OBJECT MODEL // EC2 Instances foreach ($ResInstances->body->reservationSet->item as $ItemSet) { foreach ($ItemSet->instancesSet->item as $Item) { $InstanceId = (string) $Item->instanceId; $ImageId = (string) $Item->imageId; $State = (string) $Item->instanceState->name; $InstanceType = (string) $Item->instanceType; $AvailabilityZone = (string) $Item->placement->availabilityZone; $LaunchTime = (string) $Item->launchTime; $Region->AddInstance(new Instance($AvailabilityZone, $InstanceId, $ImageId, $State, $InstanceType, $LaunchTime)); } }
    38. 38. SAMPLE OBJECT (EC2 INSTANCE) class Instance { var $State; var $ImageId; var $InstanceId; var $LaunchTime; var $InstanceType; var $AvailabilityZone; function __construct(…) {} public function GetAvailabilityZone() {} private function GetLabel() {} function Render() {} function RenderEdges() {} }
    39. 39. RENDER OBJECT MODEL AS DOT GRAPH function WriteNode($FP, $IndentLevel, $Node, $Label, $NodeType) { fwrite($FP, Indent($IndentLevel) . Quote($Node) . ' [' . 'label=' . Quote($Label) . ', ' . GetNodeStyle($NodeType) . "];n"); }
    40. 40. DOT TEXT graph aws { // Region subgraph "cluster_us-east-1" { label="Region us-east-1"; style=filled; fillcolor=yellow; color=black; // Elastic IP Addresses subgraph "cluster_us-east-1_ips" { label="Elastic IP Addresses"; style=filled; fillcolor=darksalmon; color=black; "" [label="Elastic IP", color=black, style=filled, fillcolor=wheat, shape=rect]; "us-east-1_phantom_ips" [label="", shape=point, style=invis]; }
    41. 41. RENDER DOT TEXT TO PDF $ dot –Tpdf > ~jeff/public_html/aws_meta.pdf
    43. 43. EXAMPLE 3 – DYNAMICALLY INSTANTIATE INFRASTRUCTURE EC2 Instance 10 GB EBS Volume 100 GB EBS Volume Elastic IP Address
    44. 44. RUN AN EC2 INSTANCE // Create the EC2 access object $EC2 = new AmazonEC2(AWS_PUBLIC_KEY, AWS_SECRET_KEY); // Run an instance $Options = array('KeyName' => "Jeff's Keys", 'InstanceType' => "m1.small"); $Res = $EC2->run_instances("ami-db7b9db2", 1, 1, $Options);
    45. 45. GET INSTANCE INFO // Get the Id and Availability Zone of the instance $Instances = $Res->body->instancesSet; $InstanceId = $Instances->item->instanceId; $AvailabilityZone = $Instances->item->placement ->availabilityZone; print("Launched instance ${InstanceId} " . "in availability zone ${AvailabilityZone}.n");
    46. 46. ALLOCATE ELASTIC IP ADDRESS // Allocate an Elastic IP address $Res = $EC2->allocate_address(); if (!$Res->isOK()) { exit("Could not allocate public IP address.n"); } // Get the allocated Elastic IP address $PublicIP = $Res->body->publicIp; print("Assigned IP address ${PublicIP}.n");
    47. 47. ATTACH IP ADDRESS TO INSTANCE // Associate the Elastic IP address with the instance $Res = $EC2->associate_address($InstanceId, $PublicIP); if (!$Res->IsOK()) { exit("Could not associate IP address ${PublicIP} " . "with instance ${InstanceId}.n"); } print("Associated IP address ${PublicIP} " . "with instance ${InstanceId}.n");
    48. 48. CREATE 2 EBS VIRTUAL DISK VOLUMES // Create two EBS volumes in the instance's availability zone $Res1 = $EC2->create_volume(10, $AvailabilityZone); $Res2 = $EC2->create_volume(100, $AvailabilityZone); if (!$Res1->isOK() || !$Res2->isOK()) { exit("Could not create EBS volumes.n"); } // Get the volume Ids $VolumeId1 = $Res1->body->volumeId; $VolumeId2 = $Res2->body->volumeId;
    49. 49. ATTACH THE VOLUMES TO THE INSTANCE $Res1 = $EC2->attach_volume($VolumeId1, $InstanceId, '/dev/sdf'); $Res2 = $EC2->attach_volume($VolumeId2, $InstanceId, '/dev/sdg'); if (!$Res1->isOK() || !$Res2->isOK()) { exit("Could not attach EBS volumes " . "${VolumeId1} and ${VolumeId2} " . "to instance ${InstanceId}.n"); }
    50. 50. NOW HIRING • Northern Virginia: – SDE – Virtual Private Cloud – Front End Web Developer – Virtual Private Cloud • Seattle: – SDE – Program Management – Developer Support • More Info: – AWS booth in lobby –
    51. 51. Q & A
    52. 52. • Amazon Web Services: • Virtual Private Cloud: • Relational Database Service: • • FOR MORE INFORMATION
    53. 53. Thanks!