MongoSF 2012

About Monica Wilkinson Loves the web and data portability. Developer Advocate @ Cloud Foundry 12 years development experience. Last 5 years in Social Web Open Web Standards Advocate Contact Me: @ciberch mwilkinson@vmware.com CONFIDENTIAL 2Friday, May 4, 12

Agenda 1.Overview of Cloud Foundry 2.What is Activity Streams ? 3.Building an Activity Stream Engine CONFIDENTIAL 3Friday, May 4, 12

About Cloud Foundry The first Open PaaS Multi(n) Languages, Frameworks, Services & Clouds Open Source CONFIDENTIAL 4Friday, May 4, 12

The Open Platform as a Service Deploy and scale applications in seconds, without locking yourself p en, e, O lable Simpl Sca Flex ible, 5Friday, May 4, 12

Cloud Foundry open PaaS - Choice of frameworksFriday, May 4, 12

Cloud Foundry open PaaS - Choice of frameworks OSS communityFriday, May 4, 12

Cloud Foundry open PaaS - Choice of application services Ap p lica Data o Services n S erv ice In ter fac Msg e Services Other ServicesFriday, May 4, 12

Cloud Foundry open PaaS - Choice of application services vFabric Postgres Ap p lica Data o Services n S erv ice In vFabric ter fac RabbitMQTM Msg e Services Other Services Additional partners services …Friday, May 4, 12

Cloud Foundry open PaaS - Choice of clouds Ap p lica Data Private o Services Clouds n S erv ice In e f ac ter ter fac Msg Public r In e Clouds ide Services ov Pr ud Cl o Micro Other Clouds ServicesFriday, May 4, 12

Cloud Foundry open PaaS - Choice of clouds d Avoi n Ap p lica Lo ck-i Data Private o Services Clouds n S erv ice In e f ac ter ter fac Msg Public r In e Clouds ide Services ov Pr ud Cl o Micro Other Clouds ServicesFriday, May 4, 12

Multi-cloud flexibility is critical to your long-term success 9Friday, May 4, 12

Multi-cloud flexibility is critical to your long-term success § Make use of both public and private clouds without rewriting your applications 9Friday, May 4, 12

Multi-cloud flexibility is critical to your long-term success § Make use of both public and private clouds without rewriting your applications § Protect against vendor lock-in 9Friday, May 4, 12

Multi-cloud flexibility is critical to your long-term success § Make use of both public and private clouds without rewriting your applications § Protect against vendor lock-in § Meet different compliance and geographical needs 9Friday, May 4, 12

Multi-cloud flexibility is critical to your long-term success § Make use of both public and private clouds without rewriting your applications § Protect against vendor lock-in § Meet different compliance and geographical needs § Accommodate peak loads while optimizing costs 9Friday, May 4, 12

Multi-cloud flexibility is critical to your long-term success § Make use of both public and private clouds without rewriting your applications § Protect against vendor lock-in § Meet different compliance and geographical needs § Accommodate peak loads while optimizing costs § Manage your growth and changing needs over time 9Friday, May 4, 12

CloudFoundry.COM - Multi-tenant PaaS operated by VMware CloudFoundry.COM (beta) Runtimes & Frameworks Services vCenter / vSphere Infrastructure 10Friday, May 4, 12

Micro Cloud FoundryTM – Industry first downloadable PaaS Micro Cloud Foundry Runtimes & Frameworks Services Your Laptop/PC Single VM instance of Cloud Foundry that runs on a developer’s MAC or PC 11Friday, May 4, 12

CloudFoundry.ORG - Community open-source project CloudFoundry.ORG Your Infrastructure Download Setup Setup Deploy Behind Code Environment Scripts Firewall Apache2 license 12Friday, May 4, 12

Cloud Foundry: you can trade-off effort vs flexibility Less Less Public PaaS .... .COM Private PaaS Effort .... Flexibility Custom Private PaaS git clone git://github.com/cloudfoundry/ More More 13Friday, May 4, 12

Why PaaS Matters CONFIDENTIAL 14Friday, May 4, 12

Traditional web application architecture Web Application StoreFront Accounting Service MySQL Desktop Database Apache Browser InventoryService Shipping Service Simple to Container (e.g. Tomcat) develop test deploy scale 15Friday, May 4, 12

Traditional web application architecture Web Application StoreFront Accounting Service MySQL Desktop Database Apache Browser InventoryService Shipping Service Simple to Container (e.g. Tomcat) develop test deploy scale Or is it? 15Friday, May 4, 12

Let’s imagine you want to run a performance test... ? How long to get the servers? Who is going to set up the servers? Who is going to set up Apache, Tomcat, and MySQL?Friday, May 4, 12

And things are changing: This simple, monolithic architecture is inadequate 17Friday, May 4, 12

New kinds of clients Smart phones overtake PCs in Q4 2010 18Friday, May 4, 12

Users expect a rich, dynamic and interactive experience on mobile devices and desktop HTTP Request Desktop Web Application Browser HTML/Javascript 19Friday, May 4, 12

Users expect a rich, dynamic and interactive experience on mobile devices and desktop h oug d en oo ’t g HTTP Request e isn r ctu Desktop Web Application e hit Browser HTML/Javascript I arc ty le U s Old 19Friday, May 4, 12

Need to handle massive loads and the data explosion Need scalable architectures § Application tier: • Replicated/clustered servers • Modular so that components can be scaled differently • Asynchronous architecture - communication via a message broker § Database tier: • Replication • Sharding • Polyglot persistence: Relational, NoSQL, NewSQL databases 20Friday, May 4, 12

Scaling development Web application StoreFront Accounting Service Scalable != InventoryService development Shipping Service § Strongly coupled components makes development difficult § Forces multiple developers/teams to synchronize development efforts § Obstacle to frequent, independent deployments § Increases risk of failure - need to redeploy everything to change one thing 21Friday, May 4, 12

Modern application architecture billing web application Desktop Accounting Browser Service StoreUI Native Mobile NodeJS NodeJS inventory web application Application front-end MySQL application StoreUI StoreUI InventoryService HTML5 mobile application StoreUI Redis shipping web application RabbitMQ Mongo ShippingService 22Friday, May 4, 12

Developing and testing these applications is even more challenging 23Friday, May 4, 12

Let’s imagine you are fixing a bug and want to run some JUnit integration tests... ? Who is going to setup your sandbox: MySQL, RabbitMQ, MongoDB, ....?Friday, May 4, 12

Let’s imagine you have fixed a bug and want to run some functional tests... ? How long to purchase the servers? Who is going to set up the servers? Who is going to setup MySQL, RabbitMQ, MongoDB, ....?Friday, May 4, 12

Platform-as-a-Service is the solution Deployment Services Easy deployment SQL databases Application management + NoSQL databases Easy scaling up and down Message Broker 26Friday, May 4, 12

Getting Started on Cloud Foundry CONFIDENTIAL 27Friday, May 4, 12

STEP 1 - Get a Cloud Foundry Account § https://my.cloudfoundry.com/signup/hack CONFIDENTIAL 28Friday, May 4, 12

STEP 1 - Fetch password from email 29Friday, May 4, 12

STEP 2 - Login to Cloud Foundry •sudo gem install vmc •vmc login <username> CONFIDENTIAL 30Friday, May 4, 12

STEP 3 - Push your code to Cloud Foundry vmc push • Detects runtime automatically. • In addition if you have manifest.yml vmc will read: • App Name • App Instances • Services to bind to App •Demo CONFIDENTIAL 31Friday, May 4, 12

Services on Cloud Foundry CONFIDENTIAL 32Friday, May 4, 12

Current Services Available on Cloud Foundry 33Friday, May 4, 12

Binding a service to an application $ vmc push cer-spring --path web/target/ Application Deployed URL [cer-spring.cloudfoundry.com]: Detected a Java SpringSource Spring Application, is this correct? [Yn]: Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]: Creating Application: OK Would you like to bind any services to cer-spring? [yN]: y Would you like to use an existing provisioned service? [yN]: y The following provisioned services are available 1: mysql1 2: mysql-135e0 Please select one you wish to use: 1 Binding Service [mysql1]: OK Uploading Application: Checking for available resources: OK Processing resources: OK Packing application: OK Uploading (12K): OKFriday, May 4, 12

Binding a service to an application $ vmc push cer-spring --path web/target/ Application Deployed URL [cer-spring.cloudfoundry.com]: Detected a Java SpringSource Spring Application, is this correct? [Yn]: Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]: Creating Application: OK Would you like to bind any services to cer-spring? [yN]: y [yN]: y Would you like to bind any services to cer-spring? Would you like to use anan existing provisioned service? [yN]: y Would you like to use existing provisioned service? [yN]: y The The following provisioned servicesavailable following provisioned services are are available 1: mysql1 1: mysql1 2: mysql-135e0 2: mysql-135e0 Please select one you wish to use:use: 1 Please select one you wish to 1 Binding Service [mysql1]: OK OK Binding Service [mysql1]: Uploading Application: Checking for available resources: OK Processing resources: OK Packing application: OK Uploading (12K): OKFriday, May 4, 12

Binding a service to an application $ vmc push cer-spring --path web/target/ Application Deployed URL [cer-spring.cloudfoundry.com]: Detected a Java SpringSource Spring Application, is this correct? [Yn]: Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]: Creating Application: OK Would you like to bind any services to cer-spring? [yN]: y Would you like to use an existing provisioned service? [yN]: y The following provisioned services are available 1: mysql1 2: mysql-135e0 Please select one you wish to use: 1 Binding Service [mysql1]: OK Uploading Application: Checking for available resources: OK Processing resources: OK Packing application: OK Uploading (12K): OKFriday, May 4, 12

Binding a service to an application $ vmc bind-service mysql1 myapp Binding Service [mysql1]: OK Stopping Application myapp: OK Staging Application myapp: OK Starting Application myapp: OK $Friday, May 4, 12

Every day development on Cloud Foundry CONFIDENTIAL 37Friday, May 4, 12

Every day development § Debugging and accessing the data locally • Caldecott --> Service tunneling. Access your Cloud Foundry service as if it was local. CONFIDENTIAL 38Friday, May 4, 12

Tunneling gem install caldecott vmc tunnel <mongodb> CONFIDENTIAL 39Friday, May 4, 12

Using your favorite tools CONFIDENTIAL 40Friday, May 4, 12

CONFIDENTIAL 41Friday, May 4, 12

What is Activity Streams ? CONFIDENTIAL 42Friday, May 4, 12

http://activityStrea.ms § Activity Streams is a simple open specification used to describe social actions around the web. § The goal of Activity Streams is data portability § Some publishers of Activity Streams: Socialcast, MySpace, Facebook and G+ § The default format is JSON but Atom is also supported. The JSON format was added after we saw some initial limitations with JSON § Core concepts are: Actor, Verb, Object and Target CONFIDENTIAL 43Friday, May 4, 12

Actor -> Monica Verb -> Pinned(Bookmarked) Object -> Rosa Parks’ Bio Page Target -> Female Leaders Board Source -> Pinterest Published -> MLK Day 2012 © 2012 VMware, Inc. All rights reservedFriday, May 4, 12

Example Activity John posted “SCALA 2012 Recap” on his blog “John OSS” 10 minutes ago CONFIDENTIAL 45Friday, May 4, 12

Activity in JSON { "published": "2011-02-10T15:04:55Z", "actor": { "url": "http://example.org/john", "objectType" : "person", "id": "tag:example.org,2011:john", "image": { "url": "http://example.org/john/image", "width": 250, "height": 250 }, "displayName": "John Smith" }, "verb": "post", "object" : { “objectType” : “blog-entry”, “url": "http://example.org/blog/2011/02/entry", "id": "tag:example.org,2011:abc123/xyz" }, "target" : { "url": "http://example.org/blog/", "objectType": "blog", "id": "tag:example.org,2011:abc123", "displayName": "John OSS" } } CONFIDENTIAL 46Friday, May 4, 12

What is an Activity Stream Engine ? § An activity stream engine allows you to publish events and subscribe to events. § Many activity stream engine support aggregation of events via Streams. § Many activity stream engines support fanning out activities to subscribers. § Many activity stream engines tailor the stream to the reader providing Key Stats and Contextual Info. Example: Monica with 1000 followers or Bank of America established in 1989 CONFIDENTIAL 47Friday, May 4, 12

Building an Activity Streams Engine CONFIDENTIAL 48Friday, May 4, 12

What should we use to store Activities ? Key-Value Column Document Graph Redis, Cassandra, MongoDB Neo4J Riak HBase CONFIDENTIAL 49Friday, May 4, 12

Mongo DB schema-less documents are perfect for Activities Key-Value Column Document Graph MongoDB (so easy) CONFIDENTIAL 50Friday, May 4, 12

About MongoDB 1. No-SQL database development : { 2. Stores JSON-style tools: many, documents with language_support: superb, embedded documents = agility: high 3. Horizontally scalable }, production: { 4. Full Indexing Support speed: fast 5. Open Source so great fault_tolerance: true ORMs and drivers. scalability: high CONFIDENTIAL 51Friday, May 4, 12

Step 1- Build the persistence Model var MediaLinkHash = { duration: Number, height: Number, width: Number, url: String }; var ActivityObjectHash = { id: {type: String}, image: MediaLinkHash, displayName: {type: String}, summary: {type: String}, content: {type: String}, url: {type:String}, author: {type: ObjectId, ref: activityObject}, published: {type: Date, default: Date.now}, updated: {type: Date, default: Date.now}, objectType: {type: String}, attachments: [{type: ObjectId, ref: activityObject}], upstreamDuplicates: [String], downstreamDuplicates: [String] }; var ActivityObjectSchema = new Schema(ActivityObjectHash) this.ActivityObject = mongoose.model(activityObject, ActivityObjectSchema); CONFIDENTIAL 52Friday, May 4, 12

Step 1- Persistence Model var ActivitySchema = new Schema({ id: {type: String}, verb: {type: String}, url: {type: String}, title: {type: String}, content: {type: String}, icon: MediaLinkHash, object: ActivityObjectHash, actor: ActivityObjectHash, target: {type: ObjectId, ref: activityObject}, published: { type: Date, default: Date.now}, updated: { type: Date, default: Date.now}, inReplyTo: {type: ObjectId, ref: activity} }); this.Activity = mongoose.model(activity, ActivitySchema); return this; }; CONFIDENTIAL 53Friday, May 4, 12

Step 2 - Expose helpers for the queries § The most important is to be able to list the activities in descending order by published time. § It is also important to hydrate any objects for which we have references § With Mongoose you can do: § this.getActivityStream = function(n, fx) { Activity.find().sort(published, descending).limit(n).populate(target).run (fx); } § More at: https://github.com/cloudfoundry-samples/ activity-streams-mongoose CONFIDENTIAL 54Friday, May 4, 12

How do we make our engine faster and more scalable ? § Add support for PubSub § In a PubSub model as soon as an event is published it gets sent to all the subscribers. § The Publisher doesn’t know who the subscribers are, it simply knows where to publish. § Redis has nice support for Pub Sub http://redis.io/topics/ pubsub CONFIDENTIAL 55Friday, May 4, 12

Exposing the Engine as Service CONFIDENTIAL 56Friday, May 4, 12

Using Socket.io § Server Side: • Subscribe to a Stream • When there is a new event send it to the client as JSON § Client Side • Add support to publish activities to the Stream • Change the client rendering to use ActivityStrea.ms § Benefits • Richer messaging • Activity Syndication CONFIDENTIAL 57Friday, May 4, 12

Demo http://asms.cloudfoundry.com CONFIDENTIAL 58Friday, May 4, 12

Thank You - Questions ? Cloud Foundry Activity Streams Libraries are at: https://github.com/organizations/cloudfoundry-samples http://cloudfoundry.com Questions: @cloudfoundry - @ciberch or mwilkinson@vmware.com © 2012 VMware, Inc. All rights reservedFriday, May 4, 12

http://www.10gen.com	1103
http://www.scoop.it	732
http://www.twylah.com	49
http://lanyrd.com	44
http://drupal1.10gen.cc	18
http://www.linkedin.com	9
https://twimg0-a.akamaihd.net	4
https://si0.twimg.com	4
http://pdvqwpb.mongodb.org	3
http://apy.mongodb.org	3
http://archive.10gen.com	2
http://coderwall.com	2
http://w.mongodb.org	2
http://a0.twimg.com	2
http://leapf.org	1
http://presentations.10gen.com	1
http://twitter.com	1

MongoSF 2012

by Monica Wilkinson

Accessibility

Categories

Upload Details

Usage Rights

17 Embeds 1,980

Statistics

MongoSF 2012 Presentation Transcript