by Ian Robinson - In the mid-2000s I was using the web as a platform for building enterprise apps. This led to my co-authoring 'REST in Practice', a guide to applying the tenets of REST in the enterprise. Then, in 2011, I joined a graph database company. Here, we applied the ideas that inspired 'REST in Practice' to a quite different set of architectural problems. In this session I'll discuss some the things I've learnt in implementing a database server Web API, and building the infrastructure we use for testing clusters, reproducing customer scenarios, and benchmarking the database.

1. MILAN November 28th/29th, 2014
Ian Robinson
REST in Practice, in Practice
@iansrobinson

2. MILAN November 28th/29th, 2014 – Ian Robinson
Work
in
the
1950s
Office
Internal Mail
Business

3. MILAN November 28th/29th, 2014 – Ian Robinson
RESTful
Web
App
Domain Model
Web Technologies
http://app
http://app/abc
http://app/xyz
HTTP
HTTP

4. MILAN November 28th/29th, 2014 – Ian Robinson
Protocols
Coordinate
Processes
HTTP v
? v

5. MILAN November 28th/29th, 2014 – Ian Robinson
Protocols

6. MILAN November 28th/29th, 2014 – Ian Robinson
Protocol
SpecificaBon

7. MILAN November 28th/29th, 2014 – Ian Robinson
Protocol
Discovery

8. MILAN November 28th/29th, 2014 – Ian Robinson
Everyday
Protocol
Discovery

9. MILAN November 28th/29th, 2014 – Ian Robinson
Hypermedia
Constraint
Hypermedia
As
The
Engine
Of
Application
State

10. MILAN November 28th/29th, 2014 – Ian Robinson
Graph database
Clustered

11. MILAN November 28th/29th, 2014 – Ian Robinson
Infrastructure
and
Tooling
• Performance
tesBng
• Soak
tesBng
• Stress
tesBng
• Proof
of
concept
• Reproduce
customer
issues
http://www.danzigergallery.com/artists/yuji-obata/7

12. MILAN November 28th/29th, 2014 – Ian Robinson

13. MILAN November 28th/29th, 2014 – Ian Robinson
TestLab
Manager
Workflows
Application State

14. MILAN November 28th/29th, 2014 – Ian Robinson
1.
Design
Around
CapabiliBes
Provision
Estate
Install
Database
Configure
Database
Install
Dataset
Install
Extension
Start
Database
Archive
Logs
Backup
Database
Check
Database
Status
Capture
Heap
Dump
Install
Workload
Execute
Workload
Publish
Results
Analyze
Results

15. MILAN November 28th/29th, 2014 – Ian Robinson
Implement
as
Resource
Families
Provision
Estate
Install
Database
Configure
Database
Install
Dataset
Install
Extension
Start
Database
Archive
Logs
Backup
Database
Check
Database
Status
Capture
Heap
Dump
Install
Workload
Execute
Workload
Publish
Results
Analyze
Results

16. MILAN November 28th/29th, 2014 – Ian Robinson
Deploy
in
Services
Provision
Estate
Database
Agent
Install
Database
Configure
Database
Install
Dataset
Backup
Database
Check
Database
Status
Capture
Heap
Dump
Install
Extension
Start
Database
Archive
Logs
Load
Generator
Install
Workload
Execute
Workload
Results
Service
Publish
Results
Analyze
Results

17. MILAN November 28th/29th, 2014 – Ian Robinson
Client
Creates
an
ApplicaBon
• Client-­‐specific
applicaBon
goal
• Applies
resources
to
saBsfy
this
goal
• Intelligence
at
the
edges
• Client
understands/reconstructs
state
of
the
applicaBon

18. MILAN November 28th/29th, 2014 – Ian Robinson
Database
Agent
Neo-­‐Workbench
Java
SDK
Database
Agent
Command-­‐Line
Tools
Tests
(e.g.
Soak
Tests)
Database
Agent
Servers
Clients

19. MILAN November 28th/29th, 2014 – Ian Robinson
Database
Agent
Demo
• Browser
• Command-­‐line
• Code

20. MILAN November 28th/29th, 2014 – Ian Robinson
2.
Choose
a
Hypermedia
Format
<table>
<tr>
<td>
<a href="http://localhost:38000/database-agent/001"
rel="database-admin">001</a>
</td>
</tr>
...
</table>
<form class="install-database"
method="POST"
action="http://localhost:38000/database-agent/"
enctype="application/x-www-form-urlencoded">
<input type="hidden" name="_redirect_"
value="http://localhost:38000/database-agent/"></input>
<input type="text" name="installUri"></input>
<button type="submit" id="submit-form-install-database">Install</button>
</form>

21. MILAN November 28th/29th, 2014 – Ian Robinson
<table>
<tr>
<td>
<a href="http://localhost:38000/database-agent/001"
rel="database-admin">001</a>
</td>
</tr>
...
</table>
<form class="install-database"
method="POST"
action="http://localhost:38000/database-agent/"
enctype="application/x-www-form-urlencoded">
<input type="hidden" name="_redirect_"
value="http://localhost:38000/database-agent/"></input>
<input type="text" name="installUri"></input>
<button type="submit" id="submit-form-install-database">Install</button>
</form>
Link
–
Safe
OperaBons
SemanBc
context

22. MILAN November 28th/29th, 2014 – Ian Robinson
<table>
<tr>
<td>
<a href="http://localhost:38000/database-agent/001"
rel="database-admin">001</a>
</td>
</tr>
...
</table>
<form class="install-database"
method="POST"
action="http://localhost:38000/database-agent/"
enctype="application/x-www-form-urlencoded">
<input type="hidden" name="_redirect_"
value="http://localhost:38000/database-agent/"></input>
<input type="text" name="installUri"></input>
<button type="submit" id="submit-form-install-database">Install</button>
</form>
Form
–
Unsafe
OperaBons
SemanBc
context

23. MILAN November 28th/29th, 2014 – Ian Robinson
<table>
<tr>
<td>
<a href="http://localhost:38000/database-agent/001"
rel="database-admin">001</a>
</td>
</tr>
...
</table>
<form class="install-database"
method="POST"
action="http://localhost:38000/database-agent/"
enctype="application/x-www-form-urlencoded">
<input type="hidden" name="_redirect_"
value="http://localhost:38000/database-agent/"></input>
<input type="text" name="installUri"></input>
<button type="submit" id="submit-form-install-database">Install</button>
</form>
Program
the
Client

24. MILAN November 28th/29th, 2014 – Ian Robinson
Machines
First,
Browsers
Second
<table>
<tr>
<td>
<a href="http://localhost:38000/database-agent/001"
rel="database-admin">001</a>
</td>
</tr>
...
</table>
<form class="install-database"
method="POST"
action="http://localhost:38000/database-agent/"
enctype="application/x-www-form-urlencoded">
<input type="hidden" name="_redirect_"
value="http://localhost:38000/database-agent/"></input>
<input type="text" name="installUri"></input>
<button type="submit" id="submit-form-install-database">Install</button>
</form>
Conven&on
Fields
with
_underscores_
target
the
browser

25. MILAN November 28th/29th, 2014 – Ian Robinson
POST /database-agent
installUri=file:///neo4j-enterprise-2.1.5-unix.tar.gz
Client
Database
Agent
ProgrammaBc
Client
201 CREATED
Location: /database-agent/001/

26. MILAN November 28th/29th, 2014 – Ian Robinson
Database
Agent
Browser
Client
POST /database-agent
installUri=file:///neo4j-enterprise-2.1.5-unix.tar.gz
_redirect_=/database-agent
303 SEE OTHER
Location: /database-agent
GET/database-agent
200 OK
Content-Type: text/html

27. MILAN November 28th/29th, 2014 – Ian Robinson
3.
Long-­‐Running
Processes
• Request
triggers
long-­‐running
operaBon
(e.g.
backup)
• Client
waits
for
response:
• What
happens
if
the
connecBon
breaks?

28. MILAN November 28th/29th, 2014 – Ian Robinson
202
Accepted
• Indicates
request
has
been
accepted
for
processing
• Response
includes
URI
of
status
monitor
• Client
polls
status
monitor

29. MILAN November 28th/29th, 2014 – Ian Robinson
Client
Database
Agent
202
Accepted
POST /database-agent/001/stores
installUri=s3://.../graph.db.zip
202 ACCEPTED
Location: /database-agent/jobs/3e4d0
GET /database-agent/jobs/3e4d0
202 ACCEPTED
Location: /database-agent/jobs/3e4d0
GET /database-agent/jobs/3e4d0
201 CREATED
Location: /database-agent/002/stores/graph.db
Install
Store

30. MILAN November 28th/29th, 2014 – Ian Robinson
Polling
Demo

31. MILAN November 28th/29th, 2014 – Ian Robinson
4.
Client-­‐Side
OrchestraBon
Neo-­‐Workbench
Java
APIs
Command-­‐Line
Tools
Tests
(e.g.
Soak
Tests)
Servers
Clients
Sequential = Slow
1. Install Database 1
2. Install Database 2
3. Install Database 3
4. Upload Store 1-A
5. Upload Store 2-A
6. Upload Store 3-A
7. Start Database 1
8. Start Database 2
9. Start Database 3
10. etc…

32. MILAN November 28th/29th, 2014 – Ian Robinson
Client
Database
Agent
Database
Agent
Database
Agent
Install
Database
Install
Database
Install
Database
Upload
Dataset
Upload
Dataset
Upload
Dataset

33. MILAN November 28th/29th, 2014 – Ian Robinson
ReacBve
Extensions
Iterables
Observables
Synchronous
Asynchronous
Pull
Push
for ( Integer integer : stats.counters() )
{
System.out.println( integer );
}
stats.counters().subscribe(
new Action1<Integer>()
{
@Override
public void call( Integer integer )
{
System.out.println( integer );
}
});
h^ps://github.com/Ne`lix/RxJava/wiki

34. MILAN November 28th/29th, 2014 – Ian Robinson
new StartCluster()
.execute( cluster )
.subscribe( new Action1<URI>()
{
@Override
public void call( URI uri )
{
System.out.println(
format( "Database started: %s", uri ) );
}
} );
Client-­‐Side
OrchestraBon

35. MILAN November 28th/29th, 2014 – Ian Robinson
new StartCluster()
.execute( cluster )
.subscribe( new Action1<URI>()
{
@Override
public void call( URI uri )
{
System.out.println(
format( "Database started: %s", uri ) );
}
} );
Client-­‐Side
OrchestraBon

36. MILAN November 28th/29th, 2014 – Ian Robinson
5.
Bookmark
ApplicaBon
State
{
"estate" : {
"estateId" : "est51b65",
"owner" : "neo-workbench",
"scenario" : "soak-test",
"blocks" : [ {
"blockId" : "blke16a8",
"name" : "db-cluster",
"size" : 3,
"machines" : [ {
"id" : "mace5508",
"publicIpAddress" : "127.0.0.1",
"publicDnsName" : "localhost",
"links" : [ {
"rel" : "database-agent",
"uri" : "http://localhost:38001/database-agent"
}, {
"rel" : "database-admin",
"uri" : "http://localhost:38001/database-agent/003"
}, {
"rel" : "database",
"uri" : "http://localhost:7475"
} ],

37. MILAN November 28th/29th, 2014 – Ian Robinson
5.
Bookmark
ApplicaBon
State
{
"estate" : {
"estateId" : "est51b65",
"owner" : "neo-workbench",
"scenario" : "soak-test",
"blocks" : [ {
"blockId" : "blke16a8",
"name" : "db-cluster",
"size" : 3,
"machines" : [ {
"id" : "mace5508",
"publicIpAddress" : "127.0.0.1",
"publicDnsName" : "localhost",
"links" : [ {
"rel" : "database-agent",
"uri" : "http://localhost:38001/database-agent"
}, {
"rel" : "database-admin",
"uri" : "http://localhost:38001/database-agent/003"
}, {
"rel" : "database",
"uri" : "http://localhost:7475"
} ],

38. MILAN November 28th/29th, 2014 – Ian Robinson
6.
Automate
Deployment
Load
Database
Results

39. MILAN November 28th/29th, 2014 – Ian Robinson
Automated
Provisioning
Provisioning
API
AWS
CloudFormation
SDK
Local
Provisioning
Provider
Vagrant
Azure
SDK

40. MILAN November 28th/29th, 2014 – Ian Robinson
Fluent
API
Estate estate = new Estate()
.addBlock( new Block<>(
"results", 1,
ResultsServiceConfig.DEFAULT,
AwsPlatformConfig.DEFAULT )
.addBlock( new Block<>(
"db-cluster", 3,
DatabaseAgentConfig.builder().withPackageDownloadUri(
URI.create( "http://dist.neo4j.org/neo4j-enterprise-2.1.2-unix.tar.gz" ) ).build(),
AwsPlatformConfig.DEFAULT ) )
.addBlock( new Block<>(
"load-generator", 1,
LoadGeneratorConfig.DEFAULT,
LocalPlatformConfig.DEFAULT ) );
EstateProvisioning provisioning = new EstateProvisioning();
provisioning.provision( estate );

41. MILAN November 28th/29th, 2014 – Ian Robinson
Graphite
Block
in
AWS
Estate estate = new Estate()
.addBlock( new Block<>(
"results", 1,
ResultsServiceConfig.DEFAULT,
AwsPlatformConfig.DEFAULT )
.addBlock( new Block<>(
"db-cluster", 3,
DatabaseAgentConfig.builder().withPackageDownloadUri(
URI.create( "http://dist.neo4j.org/neo4j-enterprise-2.1.2-unix.tar.gz" ) ).build(),
AwsPlatformConfig.DEFAULT ) )
.addBlock( new Block<>(
"load-generator", 1,
LoadGeneratorConfig.DEFAULT,
LocalPlatformConfig.DEFAULT ) );
EstateProvisioning provisioning = new EstateProvisioning();
provisioning.provision( estate );

42. MILAN November 28th/29th, 2014 – Ian Robinson
Database
Block
in
AWS
Estate estate = new Estate()
.addBlock( new Block<>(
"results", 1,
ResultsServiceConfig.DEFAULT,
AwsPlatformConfig.DEFAULT )
.addBlock( new Block<>(
"db-cluster", 3,
DatabaseAgentConfig.builder().withPackageDownloadUri(
URI.create( "http://dist.neo4j.org/neo4j-enterprise-2.1.2-unix.tar.gz" ) ).build(),
AwsPlatformConfig.DEFAULT ) )
.addBlock( new Block<>(
"load-generator", 1,
LoadGeneratorConfig.DEFAULT,
LocalPlatformConfig.DEFAULT ) );
EstateProvisioning provisioning = new EstateProvisioning();
provisioning.provision( estate );

43. MILAN November 28th/29th, 2014 – Ian Robinson
Local
Load
Generator
Block
Estate estate = new Estate()
.addBlock( new Block<>(
"results", 1,
ResultsServiceConfig.DEFAULT,
AwsPlatformConfig.DEFAULT )
.addBlock( new Block<>(
"db-cluster", 3,
DatabaseAgentConfig.builder().withPackageDownloadUri(
URI.create( "http://dist.neo4j.org/neo4j-enterprise-2.1.2-unix.tar.gz" ) ).build(),
AwsPlatformConfig.DEFAULT ) )
.addBlock( new Block<>(
"load-generator", 1,
LoadGeneratorConfig.DEFAULT,
LocalPlatformConfig.DEFAULT ) );
EstateProvisioning provisioning = new EstateProvisioning();
provisioning.provision( estate );

44. MILAN November 28th/29th, 2014 – Ian Robinson
Fluent
API
Estate estate = new Estate()
.addBlock( new Block<>(
"results", 1,
ResultsServiceConfig.DEFAULT,
AwsPlatformConfig.DEFAULT )
.addBlock( new Block<>(
"db-cluster", 3,
DatabaseAgentConfig.builder().withPackageDownloadUri(
URI.create( "http://dist.neo4j.org/neo4j-enterprise-2.1.2-unix.tar.gz" ) ).build(),
AwsPlatformConfig.DEFAULT ) )
.addBlock( new Block<>(
"load-generator", 1,
LoadGeneratorConfig.DEFAULT,
LocalPlatformConfig.DEFAULT ) );
EstateProvisioning provisioning = new EstateProvisioning();
provisioning.provision( estate );

45. MILAN November 28th/29th, 2014 – Ian Robinson
Compliments
of Neo Technology
Graph
Databases
Ian Robinson,
h
Jim Webber & Emil Eifrem
@iansrobinson

46. MILAN November 28th/29th, 2014 – Ian Robinson
Database
Agent
Soak
Test
Database
Agent
Database
Agent
Load
Server
Results
Server
Graphite
Graphite
Database
Cluster
Estate
Workload