Cloud Computing in PHP With the Amazon Web Services

Cloud Computing in PHP With the Amazon Web Services
Jeff Barr, Senior Web Services Evangelist
jbarr@amazon.com
@jeffbarr on Twitter

Based in Seattle
Unix since 1982; Linux since 1995
Career path:
Startups
Microsoft
Consultant to VCs and startups
Amazon Web Services
Lead AWS Blogger
Author of “Host Your Website in the Cloud”
Introduction

Amazon / Cloud / AWS Intro
AWS Overview
Programmable Infrastructure
S3, EC2, SDB, SQS
Programming AWS in PHP
CloudFusion library
Scalable Image Processing
Visualizing Infrastructure
Dynamic Infrastructure
Wrapup / Q & A
Session Outline

AMAZON’S THREE BUSINESSES
Consumer (Retail)Business
SellerBusiness
Developers &IT Professionals
Tens of millions of active customer accounts
Seven countries: US, UK, Germany, Japan, France, Canada, China
Sell on Amazon websites
Use Amazon technology for your own retail website
Leverage Amazon’s massive fulfillment center network
On-demand infrastructure for hosting web-scale solutions
Hundreds of thousands of registered customers

What is Cloud Computing?
First, think of your electricity service…
Power is available to you on-demand, you pay only for what you use…
…and you plug into a vast electrical grid managed by professionals to get you the lowest cost, most reliable power with much greater efficiency and safety than you could probably do on your own.

Introducing Amazon Web Services
AWS provides flexible, scalable, secure, and cost-effective IT infrastructure for businesses of all sizes around the world.
Compute power and storage is available to you on-demand, you pay only for the resources you use…
…running on scalable, reliable, and secure infrastructure operated by Amazon Web Services, based on the knowledge gleaned from over a decade of building efficient and dependable infrastructure for Amazon.com.

Amazon Web Services
Your Custom Applications and Services
Monitoring
Amazon CloudWatch
Management
AWS Management Console
Tools
AWS Toolkit for Eclipse
Java, PHP, Ruby, Python, .Net Developer Centers
Isolated Network
Amazon Virtual Private Cloud
Content Delivery
Amazon CloudFront
Messaging
Amazon Simple Queue Service (SQS)
Amazon Simple Notification Service (SNS)
Payments
Amazon Flexible Payments Service (FPS)
On-Demand Workforce
Amazon Mechanical Turk
Parallel Processing
Amazon Elastic MapReduce
Compute
Amazon Elastic Compute Cloud (EC2)
Elastic Load Balancing
Auto Scaling
Storage
Amazon Simple Storage Service (S3)
AWS Import/Export
Database
Amazon RDS
Amazon SimpleDB

All functionality accessed by APIs
Amazon and third-party libraries
Command-line tools
AWS Management Console
Third-party Tools
Programmable Infrastructure
// Run an instance
$EC2 = new AmazonEC2();
$Options = array('KeyName' => "Jeff's Keys",
'InstanceType' => "m1.small");
$Res = $EC2->run_instances("ami-db7b9db2", 1, 1, $Options);

Scalable data storage in-the-cloud
Highly available and durable
Pay-as-you-go pricing:
Storage: tiered $0.15/GB to $0.055/GB
Data Transfer Out: tiered $0.15/GB to $0.080/GB
Data Transfer In:Free until June 30, 2010
Requests: nominal charges
Big and busy:
102 billion objects
100K requests/second
Amazon Simple Storage Service (S3)

Amazon EC2: on-demand compute power
Obtain and boot new server instances in minutes
Quickly scale capacity up or down
Key features:
Support for Linux, Windows, and OpenSolaris
Supports all major web and application platforms
Deploy across Availability Zones for reliability
Elastic IPs provide greater flexibility
Persistent storage with Amazon Elastic Block Store
Monitoring (CloudWatch), Load Balancing, Auto-Scaling
Amazon Elastic Compute Cloud (EC2)

EC2 Programming Interface (API)
Images:
RegisterImage
DescribeImages
DeregisterImage
ModifyImageAttribute
DescribeImageAttribute
ResetImageAttribute
Instances:
RunInstances
DescribeInstances
TerminateInstances
StopInstances
GetConsoleOutput
RebootInstances
IP Addresses:
AllocateAddress
ReleaseAddress
AssociateAddress
DisassociateAddress
DescribeAddresses
Keypairs:
CreateKeyPair
DescribeKeyPairs
DeleteKeyPair
Security Groups:
CreateSecurityGroup
DescribeSecurityGroups
DeleteSecurityGroup
AuthorizeSecurityGroupIngress
RevokeSecurityGroupIngress
Block Storage Volumes:
CreateVolume
DeleteVolume
DescribeVolumes
AttachVolume
DetachVolume
CreateSnapshot
DescribeSnapshots
DeleteSnapshot
VPC:
CreateCustomerGateway
DeleteCustomerGateway
DescribeCustomerGateways
AssociateDhcpOptions
CreateDhcpOptions
DeleteDhcpOptions
DescribeDhcpOptions
CreateSubnet
DeleteSubnet
DescribeSubnets
CreateVpc
DeleteVpc
DescribeVpcs
CreateVpnConnection
DeleteVpnConnection
DescribeVpnConnections
AttachVpnGateway
CreateVpnGateway
DeleteVpnGateway
DescribeVpnGateways
DetachVpnGateway

Simple, scalable storage solution for structured data
Provides core database functionality for data storage and querying
No schema, no data modeling, no DBA
SQL queries
Amazon SimpleDB
Store:
PUT (item, 123), (description, Sweater), (color, Blue), (color, Red)
Query:
SELECT * FROM Inventory WHERE material='Leather'

Reliable, highly scalable, hosted queue for messaging
Build automated workflows for all applications
Coordinate multiple Amazon EC2 instances
Amazon Simple Queue Service
Consumer
Producer
Queue
Consumer
Producer
Producer

Getting Started With AWS
Create
Developer Account
Retrieve Private and Public Keys
Enter Payment
Information
Build & Deploy Application
Sign Up for Desired Services
Monitor and Scale Application

Moving Right Along!
"Enough with the talk, show us some code, Jeff!"

Programming AWS With PHP and CloudFusion
We’ll use the CloudFusion libraries from www.getcloudfusion.com

CloudFusion Basics
Download from SVN
Add directory to PHP’s include_path
Add AWS keys to config.inc.php
define('AWS_KEY', 'J35NTGFCQOIUY3OMNSQQ');
define('AWS_SECRET_KEY', '99pizu2vVOK11rk9UAgWVj7PBGzWwertqJlgLV0c');
Include one file:
require_once('cloudfusion.class.php');

CloudFusion XML / PHP SimpleXML
CloudFusion methods return a SimpleXML Object:
[body] => SimpleXMLElement Object
(
[Name] => sitepoint-aws-cloud-book
[Prefix] => SimpleXMLElement Object ()
[Marker] => SimpleXMLElement Object ()
[MaxKeys] => 1000
[IsTruncated] => false
[Contents] => Array
(
[0] => SimpleXMLElement Object
(
[Key] => images/2008_shiller_housing_projection.jpg
Access as $Res->body->Contents[0]->Key

Example 1 – Scalable Processing Pipeline

Building a Scalable Image Processing Pipeline
Fetch a web page, store in Amazon S3
Parse page and extract links to images
Fetch first 16 images on page, store in Amazon S3
Render images as composite image

SQS-Based Application Architecture
S3
URL Queue
Fetch Images
Fetch & Store Page
Render Queue
S3
S3
Parse Queue
Render Images & Pages
Parse Page
Image Queue

Architecture Attributes
Simple
Each stage is easy to understand
AWS reduces low-level coding
Scalable
Add more processes
Add more instances
Add more stages
Storage any amount of data
Fault Tolerant
Messages remain in queues until processed
Messages reappear in queues if process dies
Asynchronous
Each stage runs at its own speed
Build, test,run one stage at a time

Define Queues
// Queues
define('URL_QUEUE', 'c_url');
define('PARSE_QUEUE', 'c_parse');
define('IMAGE_QUEUE', 'c_image');
define('RENDER_QUEUE', 'c_render');
define('FEED_QUEUE', 'c_feed');

Create Queues
// Create the SQS access object
$SQS = new AmazonSQS();
for ($i = 1; $i < count($argv); $i++)
{
$Queue = $argv[$i];
$Res = $SQS->create_queue($Queue);
if ($Res->isOK())
{
print("Created queue '${Queue}' ");
}
}

Create Queues & Check Status
$ ./create_queues.php c_urlc_parsec_imagec_renderc_feed
$ ./crawl_queue_status.php
c_urlc_parsec_imagec_render
----- ------- ------- --------
0 0 0 0

Locate a Queue's URL
// Create the SQS access object
$SQS = new AmazonSQS();
// Locate the queue
$QueueURL = FindQueueURL($SQS, URL_QUEUE);

Post Message to Queue
// Create message
$HistItem = array('Posted by ' . $argv[0] . ' at ' . date('c'));
$Message = json_encode(array('Action' => 'FetchPage',
'Origin' => $argv[0],
'Data' => $argv[$i],
'History' => $HistItem));
// Post message
$Res = $SQS->send_message($QueueURL, $Message);
if ($Res->isOK())
{
print("Posted '${Message}' to QueueURL '${QueueURL}' ");
}

Receive Message From Queue
function PullMessage($SQS, $QueueURL)
{
while (true)
{
$Res = $SQS->receive_message($QueueURL);
if ($Res->isOk())
{
if (IsSet($Res->body->ReceiveMessageResult->Message))
{
}
else
{
sleep(1);
}
}
}

Return Received Message
$Message = $Res->body->ReceiveMessageResult->Message;
$MessageBody = $Message->Body;
$MessageDetail = json_decode($MessageBody, true);
$ReceiptHandle = $Message->ReceiptHandle;
return array('QueueURL' => $QueueURL,
'Timestamp' => date('c'),
'Message' => $Message,
'MessageBody' => $MessageBody,
'MessageDetail' => $MessageDetail,
'ReceiptHandle' => $ReceiptHandle);

Sample Processing Stage
// Pull, process, post
while (true)
{
// Pull the message from the queue
$Message = PullMessage($SQS, $QueueURL);
if ($Message != null)
{
// Extract message detail
$MessageDetail = $Message['MessageDetail'];
$ReceiptHandle = $Message['ReceiptHandle'];
$PageURL = $MessageDetail['Data'];
…
}
}

Detailed Processing
// Fetch the page
print("Processing URL '${PageURL}': ");
$HTML = file_get_contents($PageURL);
print(" Retrieved " . strlen($HTML) .
" bytes of HTML ");
// Store the page in S3
$Key = 'page_' . md5($PageURL) . '.html';
if (UploadObject($S3, BOOK_BUCKET, $Key, $HTML, S3_ACL_PUBLIC))
{
// Get URL in S3
$S3URL = $S3->get_object_url(BOOK_BUCKET, $Key);
print(" Uploaded page to S3 as '${Key}' ");

Detailed Processing
// Form message to pass page along to parser
$Origin = $MessageDetail['Origin'];
$History = $MessageDetail['History'];
$History[] = 'Fetched by ' . $argv[0] . ' at ' . date('c');
$Message = json_encode(array('Action' => 'ParsePage',
'Origin' => $Origin,
'Data' => $S3URL,
'PageURL' => $PageURL,
'History' => $History));
// Pass the page along to the parser
$Res = $SQS->send_message($QueueParse, $Message);

Render Images
foreach ($ImageKeys as $ImageKey)
{
// Fetch the image
print(" Fetch image '${ImageKey}' ");
$Image = $S3->get_object(BOOK_BUCKET, $ImageKey);
// Convert it to GD format
$ImageBits = ImageCreateFromString($Image->body);
// Copy it to proper spot in the destination
print(" Render image at ${NextX}, ${NextY} ");
ImageCopy($ImageOut, $ImageBits, $NextX, $NextY,
0, 0, ImageSx($ImageBits), ImageSy($ImageBits));
// Update position for next image
$NextX += THUMB_SIZE + GAP_SIZE;
if (($NextX + THUMB_SIZE) > $OutX)
{
$NextX = BORDER_LEFT;
$NextY += THUMB_SIZE + GAP_SIZE;
}
}

Example 2 – Create Infrastructure Graph

Processing Flow
Get AWS Metadata
Build Object Model
Render Object Model as DOT Graph
Render Dot Graph as PDF

Get AWS Metadata
// Create the service access objects
$Service_EC2 = new AmazonEC2);
$Service_S3 = new AmazonS3);
$Service_SDB = new AmazonSDB);
$Service_CF = new AmazonCloudFrontetKey);
// Fetch information about all of the EC2 objects
$ResElasticIP = $Service_EC2->describe_addresses();
$ResAvailabilityZones = $Service_EC2->describe_availability_zones();
$ResInstances = $Service_EC2->describe_instances();
$ResVolumes = $Service_EC2->describe_volumes();
$ResSnapshots = $Service_EC2->describe_snapshots();

Build Object Model
// EC2 Instances
foreach ($ResInstances->body->reservationSet->item as $ItemSet)
{
foreach ($ItemSet->instancesSet->item as $Item)
{
$InstanceId = (string) $Item->instanceId;
$ImageId = (string) $Item->imageId;
$State = (string) $Item->instanceState->name;
$InstanceType = (string) $Item->instanceType;
$AvailabilityZone = (string) $Item->placement->availabilityZone;
$LaunchTime = (string) $Item->launchTime;
$Region->AddInstance(new Instance($AvailabilityZone, $InstanceId, $ImageId, $State, $InstanceType, $LaunchTime));
}
}

Sample Object (EC2 Instance)
class Instance
{
var $State;
var $ImageId;
var $InstanceId;
var $LaunchTime;
var $InstanceType;
var $AvailabilityZone;
function __construct(…) {}
public function GetAvailabilityZone() {}
private function GetLabel() {}
function Render() {}
function RenderEdges() {}
}

Render Object Model as DOT Graph
function WriteNode($FP, $IndentLevel, $Node, $Label, $NodeType)
{
fwrite($FP, Indent($IndentLevel) . Quote($Node) .
' [' .
'label=' . Quote($Label) . ', ' .
GetNodeStyle($NodeType) .
"]; ");
}

DOT Text
graph aws
{
// Region
subgraph "cluster_us-east-1"
{
label="Region us-east-1";
style=filled; fillcolor=yellow; color=black;
// Elastic IP Addresses
subgraph "cluster_us-east-1_ips"
{
label="Elastic IP Addresses";
style=filled; fillcolor=darksalmon; color=black;
"75.101.154.199" [label="Elastic IP 75.101.154.199 ec2-75-101-154-199.compute-1.amazonaws.com", color=black, style=filled, fillcolor=wheat, shape=rect];
"us-east-1_phantom_ips" [label="", shape=point, style=invis];
}

Render Dot Text to PDF
$ dot –Tpdf aws_meta.dot > ~jeff/public_html/aws_meta.pdf

Finished Infrastructure Graph

Example 3 – Dynamically Instantiate Infrastructure
10 GB
EBS
Volume
EC2 Instance
Elastic IP
Address
100 GB
EBS
Volume

Run an EC2 Instance
// Create the EC2 access object
$EC2 = new AmazonEC2(AWS_PUBLIC_KEY, AWS_SECRET_KEY);
// Run an instance
$Options = array('KeyName' => "Jeff's Keys",
'InstanceType' => "m1.small");
$Res = $EC2->run_instances("ami-db7b9db2", 1, 1, $Options);

Get Instance Info
// Get the Id and Availability Zone of the instance
$Instances = $Res->body->instancesSet;
$InstanceId = $Instances->item->instanceId;
$AvailabilityZone = $Instances->item->placement ->availabilityZone;
print("Launched instance ${InstanceId} " .
"in availability zone ${AvailabilityZone}. ");

Allocate Elastic IP Address
// Allocate an Elastic IP address
$Res = $EC2->allocate_address();
if (!$Res->isOK())
{
exit("Could not allocate public IP address. ");
}
// Get the allocated Elastic IP address
$PublicIP = $Res->body->publicIp;
print("Assigned IP address ${PublicIP}. ");

Attach IP Address to Instance
// Associate the Elastic IP address with the instance
$Res = $EC2->associate_address($InstanceId, $PublicIP);
if (!$Res->IsOK())
{
exit("Could not associate IP address ${PublicIP} " .
"with instance ${InstanceId}. ");
}
print("Associated IP address ${PublicIP} " .
"with instance ${InstanceId}. ");

Create 2 EBS Virtual Disk Volumes
// Create two EBS volumes in the instance's availability zone
$Res1 = $EC2->create_volume(10, $AvailabilityZone);
$Res2 = $EC2->create_volume(100, $AvailabilityZone);
if (!$Res1->isOK() || !$Res2->isOK())
{
exit("Could not create EBS volumes. ");
}
// Get the volume Ids
$VolumeId1 = $Res1->body->volumeId;
$VolumeId2 = $Res2->body->volumeId;

Attach The Volumes To the Instance
$Res1 = $EC2->attach_volume($VolumeId1, $InstanceId, '/dev/sdf');
$Res2 = $EC2->attach_volume($VolumeId2, $InstanceId, '/dev/sdg');
if (!$Res1->isOK() || !$Res2->isOK())
{
exit("Could not attach EBS volumes " .
"${VolumeId1} and ${VolumeId2} " .
"to instance ${InstanceId}. ");
}

Now Hiring
Northern Virginia:
SDE – Virtual Private Cloud
Front End Web Developer – Virtual Private Cloud
Seattle:
SDE
Program Management
Developer Support
More Info:
AWS booth in lobby
http://aws.amazon.com/jobs

Amazon Web Services: http://aws.amazon.com
Virtual Private Cloud: http://aws.amazon.com/vpc
Relational Database Service: http://aws.amazon.com/rds
jbarr@amazon.com
twitter.com/jeffbarr
For MORE INFORMATION

Thanks!
jbarr@amazozn.com

Cloud Computing in PHP With the Amazon Web Services

by Amazon Web Services on Jul 22, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Cloud Computing in PHP With the Amazon Web Services — Presentation Transcript