Update of the advance of building an AWS SDK for Perl

1. Building an AWS
SDK
Granada Perl Workshop 2014
Jose Luis Martinez
JLMARTIN on CPAN
Twitter: @pplu_io
joseluis.martinez@capside.com

2. About me
 CTO @CAPSiDE
 Training Partner for AWS
 Delivering AWS official training
 All four AWS certifications
 Real experience with real platforms
 Deliver consulting and continuous operation
 Writing Perl for +10 years
 And still having fun with it!

3. About AWS
 Mostly known for EC2 and S3
 But lot's more services! (+30)
 Everything is an API
 “Programmable datacenter”
 Software Defined Everything

4. About AWS
 Mostly known for EC2 and S3
 But lot's more services! (+30)
 Everything is an API
 “Programmable datacenter”
 Software Defined Everything
It’s the your wet dream

5. Perl & AWS
Bad news

 No official SDK
 Official ones: Ruby, PHP,
Python, JS, .Net, Java
 Note: python wasn't
official initially!

6. Perl & AWS
Good news

 CPAN
 And an active
community
 Lots of modules for
different services
(sometimes more than
one)

7. Perl & AWS: The CPAN
 Inconsistencies
 In method naming
 Arbitrary defaults ('eu-west-1', 'us-east-1')
 Not up to date with latest APIs
 Keeping up with AWS is hard!!!
 “Not so popular” services not covered
 IAM, CloudFormation, SWF, RedShift, CloudSearch, etc
 Almost no Role support
 AWS::CLIWrapper is a good generic solution. Not native

8. So lets write an
entire SDK!

10. Hard enough

11. You can end up with…
or

12. Challenges: # of services
 30+ services
 Some with 100+ API calls
 Each call with it’s parameters
 Want to return objects
 Evolution

13. Challenges: # of services

14. So lets start classifying

15. Challenges: Endpoints
Endpoint Type Services
Regional Services available
in a región
the rest
Single Global services 6

16. Challenges: WebService Types
One URL REST
Params: Query
Response: XML
17 3
Params: JSON
Response: JSON
10 1

17. Challenges: Signatures
Type Services
v2 2
v3https 1
cloudfront 1
v4 the rest

18. Challenges: API versions
 More tan one API version is live
 That’s why not-up-to-date callers still work
 RDS
 4 versions
 CloudFront
 3 versions
 EC2
 4 versions

19. AWS Pace of innovation
 Every month there are changes
 Changes mean APIs change (slightly)
 Very hard to keep up!

20. AWS Pace of innovation
 Every week there are announcements
 Changes mean APIs change (slightly)
 Very hard to keep up!

21. Lets start writing!

22. Hand write classes and
methods, and parameters

23. Hand write classes and
methods, and parameters

24. Parse docs
 After all… you would have to read them to write the
classes

25. Parse docs
 After all… you would have to read them to write the
classes

26. No published spec
 Introspect an official SDK!
 After all AWS already has read the docs for you

27. No published spec
 Introspect an official SDK!
 That’s your spec!
 After all AWS already has read the docs for you

28. Data driven SDKs!
 JS SDK has datastructures that define calls
 Proof of concept
 Win++! Boto and PHP2 too!
 Later migrate to using Boto

29. Data driven SDKs!
 JS SDK has datastructures that define calls
 Proof of concept
 Win++! Boto and PHP2 too!
 Later migrate to using Boto

30. The solutions

31. Modern Perl to the rescue
 Try to be very lazy
 Parameter validation: Create Moose classes that validate
the parameters passed to the call. May change in the
future.
 Got very quick results!
 Hide all possible implementation from the user
 Try not to leak implementation
 That way you can change it at will
 Meta lets you inspect the objects!
 Call objects get marshalled into datastructured via meta
 Result datastructures get marshalled into objects via meta

32. Roles

33. Modern Perl: Roles
 All functionality gets pulled in via Roles
 Code is basically in different roles, Classes are
“serialized configurations” of how to call APIs
 Makes it easy to
 Generate service classes from datastructures
 Change behaviour for tests

34. Roles are used for
 One global endpoint vs endpoints per region
 Caller (Query, JSON)
 Response (XML, JSON)
 Signatures (V2, V3, V4)
 IO

35. Aws is a class factory

36. Dynamic class construction
 User never constructs service classes directly
 Aws class method uses default config
my $ec2 = Aws->service(‘EC2’)->new( . . . )

37. Dynamic class construction
 User wants custom functionality
my $aws = Aws->new(
config => AWS::SDK::Config->new(
caller => 'Net::AWS::TestCaller‘
)
);
my $ec2 = $aws->service(‘EC2’)->new( . . . )
service loads Aws::EC2, creates a new class with TestCaller

38. Win: Hello to the async world
 Still playing around in examples
my $aws = Aws->new(
config => AWS::SDK::Config->new(
caller => 'Net::AWS::MojoCaller‘
)
);
my $ec2 = $aws->service(‘EC2’)->new( . . . )
my $asgs = $ec2->DescribeAutoScalingGroups;
See examples/asyncAutoScaling.pl

39. Win: Also going “fully async”
my $aws = Aws->new(
config => AWS::SDK::Config->new(caller => 'Net::AWS::MojoAsyncCaller')
);
my $delay = Mojo::IOLoop->delay(sub {
my ($delay, @responses) = @_;
Dumper($_) for @responses;
});
foreach my $region (
"us-east-1", "ap-northeast-1", "sa-east-1",
"ap-southeast-1", "ap-southeast-2", "us-west-2",
"us-west-1", "eu-west-1", 'invented-region') {
my $ctrail = $aws->service('CloudTrail')->new(
region => $region,
delay => $delay,
);
print "Doing a call for $regionn";
my $list = $ctrail->DescribeTrails;
}
$delay->wait;

41. Lessons Learned

42. Lessons learned
 First shot with MooseX::Declare
 Error messages are HORRIBLE!
 Change to Moops didn't go well
 Finally plain old Moose

43. Lessons learned
 Namespace finding (still in the works)
 AWS vs Aws
 Thanks @clintongormley for the article!

44. Lessons learned
 Enums (still in the works)
 APIs sometimes return unexpected stuff

45. Lessons learned
 Abstract $request object passed around
 Finally marshalled into the UserAgents’ taste

46. Still open stuff
 Load time for big services (EC2)
 Not all classes will be used all the time
 Generate classes on demand
 Calling convention
 Now CamelCased
 Not so perlish
 Calling from objects (not service objects)
 $queue->DeleteQueue(); from a queue object
 Attribute Traits
 Access to Arrays and HashRefs via nicer methods
 Lots more!!!

47. Use AWS? Use Perl? Want to
help?
 Fork your heart out:
https://github.com/pplu/aws-sdk-perl/