Monica Wilkinson gave a presentation on building an activity stream engine on Cloud Foundry using MongoDB and Node.js. Cloud Foundry is an open platform as a service that allows deploying and scaling applications across multiple languages, frameworks, services, and cloud providers without vendor lock-in. The presentation covered an overview of Cloud Foundry, what activity streams are, and building an activity stream engine that leverages Cloud Foundry's flexibility across clouds and services.

1. Activity Streams on Cloud Foundry with
MongoDB and Node.js
MongoSF - 2012
By Monica Wilkinson
© 2012 VMware, Inc. All rights reserved
Friday, May 4, 12

2. About Monica Wilkinson Loves the web and data portability.
Developer Advocate @ Cloud Foundry
12 years development experience.
Last 5 years in Social Web
Open Web Standards Advocate
Contact Me: @ciberch
mwilkinson@vmware.com
CONFIDENTIAL
2
Friday, May 4, 12

3. Agenda
1.Overview of Cloud Foundry
2.What is Activity Streams ?
3.Building an Activity Stream Engine
CONFIDENTIAL
3
Friday, May 4, 12

4. About Cloud Foundry The first Open PaaS
Multi(n) Languages, Frameworks,
Services & Clouds
Open Source
CONFIDENTIAL
4
Friday, May 4, 12

5. The Open
Platform as a Service
Deploy and scale applications in
seconds, without locking yourself
p en,
e, O lable
Simpl Sca
Flex ible,
5
Friday, May 4, 12

6. Cloud Foundry open PaaS - Choice of frameworks
Friday, May 4, 12

7. Cloud Foundry open PaaS - Choice of frameworks
OSS community
Friday, May 4, 12

8. Cloud Foundry open PaaS - Choice of application services
Ap
p lica
Data
'o
Services
n
S
erv
ice
In
ter
fac
Msg
e
Services
Other
Services
Friday, May 4, 12

9. Cloud Foundry open PaaS - Choice of application services
vFabric
Postgres
Ap
p lica
Data
'o
Services
n
S
erv
ice
In
vFabric
ter
fac
RabbitMQTM Msg
e
Services
Other
Services
Additional partners
services …
Friday, May 4, 12

10. Cloud Foundry open PaaS - Choice of clouds
Ap
p lica
Data Private
'o
Services Clouds
n
S
erv
ice
In
e
f ac
ter
ter
fac
Msg Public
r
In
e
Clouds
ide
Services
ov
Pr
ud
Cl o
Micro
Other Clouds
Services
Friday, May 4, 12

11. Cloud Foundry open PaaS - Choice of clouds
d
Avoi n
Ap
p lica Lo ck-i
Data Private
'o
Services Clouds
n
S
erv
ice
In
e
f ac
ter
ter
fac
Msg Public
r
In
e
Clouds
ide
Services
ov
Pr
ud
Cl o
Micro
Other Clouds
Services
Friday, May 4, 12

12. Multi-cloud flexibility is critical to your long-term success
9
Friday, May 4, 12

13. Multi-cloud flexibility is critical to your long-term success
§ Make use of both public and private clouds without rewriting your
applications
9
Friday, May 4, 12

14. Multi-cloud flexibility is critical to your long-term success
§ Make use of both public and private clouds without rewriting your
applications
§ Protect against vendor lock-in
9
Friday, May 4, 12

15. Multi-cloud flexibility is critical to your long-term success
§ Make use of both public and private clouds without rewriting your
applications
§ Protect against vendor lock-in
§ Meet different compliance and geographical needs
9
Friday, May 4, 12

16. Multi-cloud flexibility is critical to your long-term success
§ Make use of both public and private clouds without rewriting your
applications
§ Protect against vendor lock-in
§ Meet different compliance and geographical needs
§ Accommodate peak loads while optimizing costs
9
Friday, May 4, 12

17. Multi-cloud flexibility is critical to your long-term success
§ Make use of both public and private clouds without rewriting your
applications
§ Protect against vendor lock-in
§ Meet different compliance and geographical needs
§ Accommodate peak loads while optimizing costs
§ Manage your growth and changing needs over time
9
Friday, May 4, 12

18. CloudFoundry.COM - Multi-tenant PaaS operated by VMware
CloudFoundry.COM (beta)
Runtimes & Frameworks
Services
vCenter / vSphere
Infrastructure
10
Friday, May 4, 12

19. Micro Cloud FoundryTM – Industry first downloadable PaaS
Micro Cloud Foundry
Runtimes & Frameworks
Services
Your Laptop/PC
Single VM instance of
Cloud Foundry
that runs on a developer’s
MAC or PC
11
Friday, May 4, 12

20. CloudFoundry.ORG - Community open-source project
CloudFoundry.ORG
Your Infrastructure
Download Setup Setup Deploy Behind
Code Environment Scripts Firewall
Apache2 license
12
Friday, May 4, 12

21. Cloud Foundry: you can trade-off effort vs flexibility
Less
Less
Public PaaS
....
.COM
Private PaaS
Effort .... Flexibility
Custom Private PaaS
git clone git://github.com/cloudfoundry/
More More
13
Friday, May 4, 12

22. Why PaaS Matters
CONFIDENTIAL
14
Friday, May 4, 12

23. Traditional web application architecture
Web Application
StoreFront
Accounting
Service
MySQL
Desktop Database
Apache
Browser InventoryService
Shipping
Service
Simple to Container (e.g. Tomcat)
develop
test
deploy
scale
15
Friday, May 4, 12

24. Traditional web application architecture
Web Application
StoreFront
Accounting
Service
MySQL
Desktop Database
Apache
Browser InventoryService
Shipping
Service
Simple to Container (e.g. Tomcat)
develop
test
deploy
scale Or is it? 15
Friday, May 4, 12

25. Let’s imagine you want to run a
performance test...
?
How long to get the servers?
Who is going to set up the servers?
Who is going to set up Apache,
Tomcat, and MySQL?
Friday, May 4, 12

26. And things are changing:
This simple, monolithic architecture is
inadequate
17
Friday, May 4, 12

27. New kinds of clients
Smart phones overtake PCs in Q4 2010
18
Friday, May 4, 12

28. Users expect a rich, dynamic and interactive experience on mobile
devices and desktop
HTTP Request
Desktop
Web Application
Browser HTML/Javascript
19
Friday, May 4, 12

29. Users expect a rich, dynamic and interactive experience on mobile
devices and desktop
h
oug
d en
oo
’t g
HTTP Request
e isn
r
ctu
Desktop
Web Application
e
hit
Browser HTML/Javascript
I arc
ty le U
s
Old
19
Friday, May 4, 12

30. Need to handle massive loads and the data explosion
Need scalable architectures
§ Application tier:
• Replicated/clustered servers
• Modular so that components can be scaled differently
• Asynchronous architecture - communication via a message
broker
§ Database tier:
• Replication
• Sharding
• Polyglot persistence: Relational, NoSQL, NewSQL
databases
20
Friday, May 4, 12

31. Scaling development
Web application
StoreFront
Accounting
Service Scalable
!=
InventoryService development
Shipping
Service
§ Strongly coupled components makes development difficult
§ Forces multiple developers/teams to synchronize development efforts
§ Obstacle to frequent, independent deployments
§ Increases risk of failure - need to redeploy everything to change one thing 21
Friday, May 4, 12

32. Modern application architecture
billing web application
Desktop Accounting
Browser Service
StoreUI
Native Mobile NodeJS
NodeJS inventory web application
Application front-end
MySQL
application
StoreUI
StoreUI InventoryService
HTML5 mobile
application
StoreUI Redis shipping web application
RabbitMQ
Mongo
ShippingService
22
Friday, May 4, 12

33. Developing and testing these applications
is even more challenging
23
Friday, May 4, 12

34. Let’s imagine you are fixing a bug
and want to run some JUnit
integration tests...
?
Who is going to setup your sandbox:
MySQL, RabbitMQ, MongoDB, ....?
Friday, May 4, 12

35. Let’s imagine you have fixed a bug
and want to run some functional
tests...
?
How long to purchase the servers?
Who is going to set up the servers?
Who is going to setup MySQL,
RabbitMQ, MongoDB, ....?
Friday, May 4, 12

36. Platform-as-a-Service is the solution
Deployment Services
Easy deployment SQL databases
Application management + NoSQL databases
Easy scaling up and down Message Broker
26
Friday, May 4, 12

37. Getting Started on Cloud Foundry
CONFIDENTIAL
27
Friday, May 4, 12

38. STEP 1 - Get a Cloud Foundry Account
§ https://my.cloudfoundry.com/signup/hack
CONFIDENTIAL
28
Friday, May 4, 12

39. STEP 1 - Fetch password from email
29
Friday, May 4, 12

40. STEP 2 - Login to Cloud Foundry
•sudo gem
install vmc
•vmc login
<username>
CONFIDENTIAL
30
Friday, May 4, 12

41. STEP 3 - Push your code to Cloud Foundry
vmc push
• Detects runtime automatically.
• In addition if you have manifest.yml vmc will read:
• App Name
• App Instances
• Services to bind to App
•Demo
CONFIDENTIAL
31
Friday, May 4, 12

42. Services on Cloud Foundry
CONFIDENTIAL
32
Friday, May 4, 12

43. Current Services Available on Cloud Foundry
33
Friday, May 4, 12

44. Creating a service instance
$ vmc create-service mysql --name mysql1
Creating Service: OK
$ vmc services
============== System Services ==============
+------------+---------+---------------------------------------+
| Service | Version | Description |
+------------+---------+---------------------------------------+
| mongodb | 1.8 | MongoDB NoSQL store |
| mysql | 5.1 | MySQL database service |
| postgresql | 9.0 | PostgreSQL database service (vFabric) |
| rabbitmq | 2.4 | RabbitMQ messaging service |
| redis | 2.2 | Redis key-value store service |
+------------+---------+---------------------------------------+
=========== Provisioned Services ============
+-------------+---------+
| Name | Service |
+-------------+---------+
| mysql1 | mysql |
+-------------+---------+
Friday, May 4, 12

45. Binding a service to an application
$ vmc push cer-spring --path web/target/
Application Deployed URL [cer-spring.cloudfoundry.com]:
Detected a Java SpringSource Spring Application, is this correct? [Yn]:
Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]:
Creating Application: OK
Would you like to bind any services to 'cer-spring'? [yN]: y
Would you like to use an existing provisioned service? [yN]: y
The following provisioned services are available
1: mysql1
2: mysql-135e0
Please select one you wish to use: 1
Binding Service [mysql1]: OK
Uploading Application:
Checking for available resources: OK
Processing resources: OK
Packing application: OK
Uploading (12K): OK
Friday, May 4, 12

46. Binding a service to an application
$ vmc push cer-spring --path web/target/
Application Deployed URL [cer-spring.cloudfoundry.com]:
Detected a Java SpringSource Spring Application, is this correct? [Yn]:
Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]:
Creating Application: OK
Would you like to bind any services to 'cer-spring'? [yN]: y [yN]: y
Would you like
to bind any services to 'cer-spring'?
Would you like to use anan existing provisioned service? [yN]: y
Would you like
to use existing provisioned service? [yN]: y
The The following provisioned servicesavailable
following provisioned services are are available
1: mysql1
1: mysql1
2: mysql-135e0
2: mysql-135e0
Please select one you wish to use:use: 1
Please select one you wish to 1
Binding Service [mysql1]: OK OK
Binding Service [mysql1]:
Uploading Application:
Checking for available resources: OK
Processing resources: OK
Packing application: OK
Uploading (12K): OK
Friday, May 4, 12

47. Binding a service to an application
$ vmc push cer-spring --path web/target/
Application Deployed URL [cer-spring.cloudfoundry.com]:
Detected a Java SpringSource Spring Application, is this correct? [Yn]:
Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]:
Creating Application: OK
Would you like to bind any services to 'cer-spring'? [yN]: y
Would you like to use an existing provisioned service? [yN]: y
The following provisioned services are available
1: mysql1
2: mysql-135e0
Please select one you wish to use: 1
Binding Service [mysql1]: OK
Uploading Application:
Checking for available resources: OK
Processing resources: OK
Packing application: OK
Uploading (12K): OK
Friday, May 4, 12

48. Binding a service to an application
$ vmc bind-service mysql1 myapp
Binding Service [mysql1]: OK
Stopping Application 'myapp': OK
Staging Application 'myapp': OK
Starting Application 'myapp': OK
$
Friday, May 4, 12

49. Every day development on Cloud Foundry
CONFIDENTIAL
37
Friday, May 4, 12

50. Every day development
§ Debugging and accessing the data locally
• Caldecott --> Service tunneling. Access your Cloud Foundry service as if it was local.
CONFIDENTIAL
38
Friday, May 4, 12

51. Tunneling
gem install
caldecott
vmc tunnel
<mongodb>
CONFIDENTIAL
39
Friday, May 4, 12

52. Using your favorite tools
CONFIDENTIAL
40
Friday, May 4, 12

53. CONFIDENTIAL
41
Friday, May 4, 12

54. What is Activity Streams ?
CONFIDENTIAL
42
Friday, May 4, 12

55. http://activityStrea.ms
§ Activity Streams is a simple open specification used to describe social
actions around the web.
§ The goal of Activity Streams is data portability
§ Some publishers of Activity Streams: Socialcast, MySpace, Facebook
and G+
§ The default format is JSON but Atom is also supported. The JSON
format was added after we saw some initial limitations with JSON
§ Core concepts are: Actor, Verb, Object and Target
CONFIDENTIAL
43
Friday, May 4, 12

56. Actor -> Monica
Verb -> Pinned(Bookmarked)
Object -> Rosa Parks’ Bio Page
Target -> Female Leaders Board
Source -> Pinterest
Published -> MLK Day 2012
© 2012 VMware, Inc. All rights reserved
Friday, May 4, 12

57. Example Activity
John posted “SCALA 2012 Recap” on his blog
“John OSS”
10 minutes ago
CONFIDENTIAL
45
Friday, May 4, 12

58. Activity in JSON
{
"published": "2011-02-10T15:04:55Z",
"actor": {
"url": "http://example.org/john",
"objectType" : "person",
"id": "tag:example.org,2011:john",
"image": {
"url": "http://example.org/john/image",
"width": 250,
"height": 250
},
"displayName": "John Smith"
},
"verb": "post",
"object" : {
“objectType” : “blog-entry”,
“url": "http://example.org/blog/2011/02/entry",
"id": "tag:example.org,2011:abc123/xyz"
},
"target" : {
"url": "http://example.org/blog/",
"objectType": "blog",
"id": "tag:example.org,2011:abc123",
"displayName": "John OSS"
}
} CONFIDENTIAL
46
Friday, May 4, 12

59. What is an Activity Stream Engine ?
§ An activity stream engine allows you to publish events and subscribe to
events.
§ Many activity stream engine support aggregation of events via Streams.
§ Many activity stream engines support fanning out activities to subscribers.
§ Many activity stream engines tailor the stream to the reader providing Key
Stats and Contextual Info. Example: Monica with 1000 followers or Bank
of America established in 1989
CONFIDENTIAL
47
Friday, May 4, 12

60. Building an Activity Streams Engine
CONFIDENTIAL
48
Friday, May 4, 12

61. What should we use to store Activities ?
Key-Value Column Document Graph
Redis, Cassandra, MongoDB Neo4J
Riak HBase
CONFIDENTIAL
49
Friday, May 4, 12

62. Mongo DB schema-less documents are perfect
for Activities
Key-Value Column Document Graph
MongoDB
(so easy)
CONFIDENTIAL
50
Friday, May 4, 12

63. About MongoDB
1. No-SQL database development : {
2. Stores JSON-style tools: many,
documents with language_support: superb,
embedded documents
=
agility: high
3. Horizontally scalable },
production: {
4. Full Indexing Support
speed: fast
5. Open Source so great fault_tolerance: true
ORMs and drivers.
scalability: high
CONFIDENTIAL
51
Friday, May 4, 12

64. Step 1- Build the persistence Model
var MediaLinkHash = {
duration: Number,
height: Number,
width: Number,
url: String
};
var ActivityObjectHash = {
id: {type: String},
image: MediaLinkHash,
displayName: {type: String},
summary: {type: String},
content: {type: String},
url: {type:String},
author: {type: ObjectId, ref: 'activityObject'},
published: {type: Date, default: Date.now},
updated: {type: Date, default: Date.now},
objectType: {type: String},
attachments: [{type: ObjectId, ref: 'activityObject'}],
upstreamDuplicates: [String],
downstreamDuplicates: [String]
};
var ActivityObjectSchema = new Schema(ActivityObjectHash)
this.ActivityObject = mongoose.model('activityObject', ActivityObjectSchema);
CONFIDENTIAL
52
Friday, May 4, 12

65. Step 1- Persistence Model
var ActivitySchema = new Schema({
id: {type: String},
verb: {type: String},
url: {type: String},
title: {type: String},
content: {type: String},
icon: MediaLinkHash,
object: ActivityObjectHash,
actor: ActivityObjectHash,
target: {type: ObjectId, ref: 'activityObject'},
published: { type: Date, default: Date.now},
updated: { type: Date, default: Date.now},
inReplyTo: {type: ObjectId, ref: 'activity'}
});
this.Activity = mongoose.model('activity', ActivitySchema);
return this;
};
CONFIDENTIAL
53
Friday, May 4, 12

66. Step 2 - Expose helpers for the queries
§ The most important is to be able to list the activities in
descending order by published time.
§ It is also important to hydrate any objects for which we
have references
§ With Mongoose you can do:
§ this.getActivityStream = function(n, fx)
{ Activity.find().sort('published',
'descending').limit(n).populate('target').run
(fx);
}
§ More at: https://github.com/cloudfoundry-samples/
activity-streams-mongoose
CONFIDENTIAL
54
Friday, May 4, 12

67. How do we make our engine faster and more
scalable ?
§ Add support for PubSub
§ In a PubSub model as soon as an event is published it
gets sent to all the subscribers.
§ The Publisher doesn’t know who the subscribers are, it
simply knows where to publish.
§ Redis has nice support for Pub Sub http://redis.io/topics/
pubsub
CONFIDENTIAL
55
Friday, May 4, 12

68. Exposing the Engine as Service
CONFIDENTIAL
56
Friday, May 4, 12

69. Using Socket.io
§ Server Side:
• Subscribe to a Stream
• When there is a new event send it to the client as JSON
§ Client Side
• Add support to publish activities to the Stream
• Change the client rendering to use ActivityStrea.ms
§ Benefits
• Richer messaging
• Activity Syndication
CONFIDENTIAL
57
Friday, May 4, 12

70. Demo
http://asms.cloudfoundry.com
CONFIDENTIAL
58
Friday, May 4, 12

71. Thank You - Questions ?
Cloud Foundry Activity Streams Libraries are at:
https://github.com/organizations/cloudfoundry-samples
http://cloudfoundry.com
Questions: @cloudfoundry - @ciberch or
mwilkinson@vmware.com
© 2012 VMware, Inc. All rights reserved
Friday, May 4, 12