The document appears to be a presentation about deploying a graph database on AWS. It discusses using Scala and Akka for building components, Neo4j as the graph database, Packer for creating machine images, and deploying the components on AWS using a t1.micro instance type to build Amazon machine images.

1. Deploying
the
Graph
November 2013
Kevin O’Neill
CTO PlayUp
@kevinoneill
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

2. Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

3. Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

4. Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

5. Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

6. A little History
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

7. Engage & Entertain
Around Live Sport
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

8. Mobile Games
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

9. SMS Based
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

10. Number of Runs
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

11. Complex
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

12. Carrier Integration
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

13. Smart Phone Revolution
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

14. Move from a Content
Creator to a Content
Enabler
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

15. Top 10 in Engagement
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

16. Over 3 Million Likes
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

17. 30% - 40% Active
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

18. Manually Curated
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

19. We Know there's
an Opportunity
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

20. The Graph
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

21. Scala
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

22. Scala
•
•
•
•
•
A Scalable language
Object-Oriented
Functional
Seamless Java Interoperability
Functions are Objects
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

23. AKKA
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

24. AKKA
•
•
•
•
Actors
Fault Tolerance
Location Transparency
Transactors
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

25. Neo4j
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

26. Neo4j
•
•
•
•
Graph Database
Vertices and Edges
Properties
Full Text Index
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

27. Whiteboard
Outlining the bits
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

28. Runtime
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

29. AWS
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

30. AWS
•
•
•
•
us-west
m2.2xlarge
ubuntu 13.04
jre 7
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

31. JVM
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

32. JVM
•
•
•
Graph - 8G of Heap
Others 128M - 1G
Parallel GC
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

33. The remaining memory
is reserved for file
system cache and
memory mapped buffers
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

34. Building the Components
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

35. SBT
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

36. Builds the jars
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

37. Debian Packages
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

38. Debian Packages
•
•
•
Places binaries in opt
Adds an upstart file for the
Permissions on data and log directories
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

39. graph-server.upstart
setuid
ubuntu
setgid
ubuntu
!
console
log
!
chdir
/var/lib/playup/spot/graph-­‐server
!
exec
java
-­‐server
-­‐Xms4096M
-­‐Xmx8192M
-­‐Xss1M
-­‐XX:MaxPermSize=256M
-­‐XX:+UseParallelGC
-­‐jar
/opt/playup/spot/graph-­‐server/run.jar
com.playup.spot.graph.service.NeoKernel
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

40. Base configuration is for
production
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

41. Baking
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

42. Packer
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

43. Create machine images
for multiple platforms
from a single config
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

44. Images have lots of benefits
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

45. Fast Deployment
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

46. Stability
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

47. Testability
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

48. Packer makes creating
images a breeze
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

49. Example
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

50. Builders
"builders":
[
{
"type":
"amazon-­‐ebs",
"instance_type":
"t1.micro",
"region":
"ap-­‐southeast-­‐2",
"ami_name":
"spot-­‐graph-­‐{{user
`version`}}-­‐{{timestamp}}",
"source_ami":
"ami-­‐4bf26f71",
"ssh_username":
"ubuntu",
"tags":
{
"Name":
"spot-­‐graph-­‐{{user
`version`}}
({{isotime}})",
"OS":
"Ubuntu",
"Version":
"{{user
`version`}}-­‐{{timestamp}}"
},
"ami_regions":["us-­‐west-­‐2"]
}
…
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

51. Builders
"builders":
[
{
"type":
"amazon-­‐ebs",
"instance_type":
"t1.micro",
"region":
"ap-­‐southeast-­‐2",
"ami_name":
"spot-­‐graph-­‐{{user
`version`}}-­‐{{timestamp}}",
"source_ami":
"ami-­‐4bf26f71",
"ssh_username":
"ubuntu",
"tags":
{
"Name":
"spot-­‐graph-­‐{{user
`version`}}
({{isotime}})",
"OS":
"Ubuntu",
"Version":
"{{user
`version`}}-­‐{{timestamp}}"
},
"ami_regions":["us-­‐west-­‐2"]
}
…
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

52. Builders
"builders":
[
{
"type":
"amazon-­‐ebs",
"instance_type":
"t1.micro",
"region":
"ap-­‐southeast-­‐2",
"ami_name":
"spot-­‐graph-­‐{{user
`version`}}-­‐{{timestamp}}",
"source_ami":
"ami-­‐4bf26f71",
"ssh_username":
"ubuntu",
"tags":
{
"Name":
"spot-­‐graph-­‐{{user
`version`}}
({{isotime}})",
"OS":
"Ubuntu",
"Version":
"{{user
`version`}}-­‐{{timestamp}}"
},
"ami_regions":["us-­‐west-­‐2"]
}
…
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

53. Builders
"builders":
[
{
"type":
"amazon-­‐ebs",
"instance_type":
"t1.micro",
"region":
"ap-­‐southeast-­‐2",
"ami_name":
"spot-­‐graph-­‐{{user
`version`}}-­‐{{timestamp}}",
"source_ami":
"ami-­‐4bf26f71",
"ssh_username":
"ubuntu",
"tags":
{
"Name":
"spot-­‐graph-­‐{{user
`version`}}
({{isotime}})",
"OS":
"Ubuntu",
"Version":
"{{user
`version`}}-­‐{{timestamp}}"
},
"ami_regions":["us-­‐west-­‐2"]
}
…
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

54. Builders
"builders":
[
{
"type":
"amazon-­‐ebs",
"instance_type":
"t1.micro",
"region":
"ap-­‐southeast-­‐2",
"ami_name":
"spot-­‐graph-­‐{{user
`version`}}-­‐{{timestamp}}",
"source_ami":
"ami-­‐4bf26f71",
"ssh_username":
"ubuntu",
"tags":
{
"Name":
"spot-­‐graph-­‐{{user
`version`}}
({{isotime}})",
"OS":
"Ubuntu",
"Version":
"{{user
`version`}}-­‐{{timestamp}}"
},
"ami_regions":["us-­‐west-­‐2"]
}
…
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

55. Variables
"variables":
{
"version":
"development"
}
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

56. Provisioners
"provisioners":
[
{
"script":
"scripts/update-­‐packages.sh",
"type":
"shell"
},
{
"script":
"scripts/install-­‐puppet.sh",
"type":
"shell"
},
{
"type":
"file",
"source":
"packages",
"destination":
"/tmp"
},
{
"script":
"scripts/install-­‐spot-­‐packages.sh",
"type":
"shell"
},
{
"type":
"puppet-­‐masterless",
"manifest_file":
"puppet/spot-­‐graph.pp",
"module_paths":
["puppet/local",
"puppet/common"]
},
{
"script":
"scripts/build-­‐date.sh",
"type":
"shell"
}
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

57. Provisioners
"provisioners":
[
{
"script":
"scripts/update-­‐packages.sh",
"type":
"shell"
},
{
"script":
"scripts/install-­‐puppet.sh",
"type":
"shell"
},
{
"type":
"file",
"source":
"packages",
"destination":
"/tmp"
},
{
"script":
"scripts/install-­‐spot-­‐packages.sh",
"type":
"shell"
},
{
"type":
"puppet-­‐masterless",
"manifest_file":
"puppet/spot-­‐graph.pp",
"module_paths":
["puppet/local",
"puppet/common"]
},
{
"script":
"scripts/build-­‐date.sh",
"type":
"shell"
}
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

58. Provisioners
"provisioners":
[
{
"script":
"scripts/update-­‐packages.sh",
"type":
"shell"
},
{
"script":
"scripts/install-­‐puppet.sh",
"type":
"shell"
},
{
"type":
"file",
"source":
"packages",
"destination":
"/tmp"
},
{
"script":
"scripts/install-­‐spot-­‐packages.sh",
"type":
"shell"
},
{
"type":
"puppet-­‐masterless",
"manifest_file":
"puppet/spot-­‐graph.pp",
"module_paths":
["puppet/local",
"puppet/common"]
},
{
"script":
"scripts/build-­‐date.sh",
"type":
"shell"
}
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

59. Provisioners
"provisioners":
[
{
"script":
"scripts/update-­‐packages.sh",
"type":
"shell"
},
{
"script":
"scripts/install-­‐puppet.sh",
"type":
"shell"
},
{
"type":
"file",
"source":
"packages",
"destination":
"/tmp"
},
{
"script":
"scripts/install-­‐spot-­‐packages.sh",
"type":
"shell"
},
{
"type":
"puppet-­‐masterless",
"manifest_file":
"puppet/spot-­‐graph.pp",
"module_paths":
["puppet/local",
"puppet/common"]
},
{
"script":
"scripts/build-­‐date.sh",
"type":
"shell"
}
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

60. Builder Provisioners
"provisioners":
[
!
…
!
{
"type":
"shell",
"script":
"scripts/vagrant.sh",
"only":
["virtualbox"]
},
{
"type":
"shell",
"script":
"scripts/guest-­‐additions.sh",
"only":
["virtualbox"]
},
!
…
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

61. Build
$
packer
build
-­‐-­‐only
"amazon-­‐ebs"
template.json
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

62. In a few minutes you
have a shiny new image
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

63. CI
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

64. CI
•
•
•
•
Variables
source ami
region
default build is for local development
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

65. Starting an instance
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

66. Remember
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

67. Backed config is for
Production*
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

68. We have staging plus multiple
developer environments
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

69. CloudInit
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

70. Configure values in
/etc/playup/…
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

71. Configure the
puppet master*
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

72. Cloud Formation
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

73. Which images
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

74. What CloudInit values
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

75. Challenges
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

76. Too many to list …
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

77. Persistant Data
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

78. Service Registration
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

79. Cluster Membership
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

80. Investigating
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

81. Docker
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

82. Apache Mesos
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

83. Come Join Us
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

84. Disrupting
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

85. Hiring Selectively
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

86. Talk to Me or Andrea
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

87. You miss 100 percent
of the the shots you
don't take
– Wayne Gretzky
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013

88. Questions?
Kevin O’Neill
CTO - PlayUp
@kevinoneill
kevin@playup.com
Kevin O’Neill - CTO PlayUp - @kevinoneill
Infrastructure Coders - Melbourne - November 2013