Your SlideShare is downloading. ×
Cannibalising The  Google  App  Engine
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Cannibalising The Google App Engine

6,481

Published on

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,481
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
63
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Cannibalising Google App Engine Kevin Noonan Calbane Ltd. IT@Cork, 30th September, 2009
  • 2. Cannibalising Google App Engine Introduction
  • 3. A working definition of Cloudware • Outsourced Virtualization o Virtual server running on outsourced server farm
  • 4. The Cloud Computing Pyramid •Michael Sheehan of GoGrid devised this classification
  • 5. What's Cloudware good for? (I) • Guerilla development o Pay-per-use model is ideal for startups o Also good for guerillas inside the enterprise...
  • 6. What's Cloudware good for? (II) • Scaling up o Clone the virtual server o pay for extra resources (bandwidth, storage, RAM, etc.)
  • 7. What's Cloudware good for? (III) • In general, outsourcing IT infrastructure o pay someone else to run your servers, network, database...
  • 8. Python: The Original of the Species
  • 9. Java: the All New Flavour
  • 10. Languages on App Engine's JVM • Java • JRuby • Jython • BeanShell • JavaScript • Scala • Groovy • … • Will it Play in App Engine? http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine
  • 11. What is Google App Engine? • Complete web stack • Runs on Google's infrastructure • Bigtable Datastore o Like a sorted hashtable o Not a relational database o Use JDO or JPA (with Java) for datastore work • Memcache • Image API • Mail API • SDK for development on Windows/Mac/Linux o Run & test locally before uploading to Cloud
  • 12. Limitations of Google App Engine • Applications run in “The Sandbox”: http://code.google.com/appengine/docs/java/runtime.html#The_Sandbox • Many Java libraries will not work (but many have been patched or will be). See the JRE whitelist: http://code.google.com/appengine/docs/java/jrewhitelist.html No “long-lived” processing (requests must return in 30 seconds). • HTTPS only on XYZ.appspot.com o and only with Google's certificate. o Due to limitation of SSL protocol, as applied to Google's Cloud.
  • 13. Billing on Google App Engine • True pay-per-use o free quotas & no bill to pay if quotas not exceeded. o quite different model to AWS • Developer can assign budget per resource. • Prices o $0.10 per CPU core-hour o $0.15 per GB-month of storage o $0.12 per GB outgoing bandwidth o $0.10 per GB incoming bandwidth o $0.0001 per email recipient for emails sent by the application
  • 14. Free Quotas on App Engine •Daily outgoing bandwidth: 10 GB •Daily incoming bandwidth 10 GB •Daily requests 1.3 million • Daily CPU-hours: 46 • More quotas (mail, datastore, images, memcache) http://code.google.com/appengine/docs/quotas.html
  • 15. Other limitations on App Engine • 1 GB of free storage per app. •100 MB Memcache storage limit • App Engine apps can now receive requests and send responses of up to 10MB. • Caveat: 1 MB limit on datastore/memcache API call • You can deploy up to 1,000 code and 1,000 static files with an app. Workaround: use ZipImport (with Python) to store files in a Zip-archive: http://code.google.com/appengine/articles/django10_zipimport.html • 150MB cumulative limit on an app's code and static files (as opposed to storage in the datastore).
  • 16. Cannibalising Google App Engine JIQL for JDBC emulation
  • 17. What is JIQL? o o Abstraction layer over the Google datastore, Bigtable o o Bigtable is non-relational, but JIQL simulates a relational database o o JIQL enables the use of JDBC and SQL with Google Bigtable
  • 18. How to use JIQL o Download Jiql.jar from http://jiql.googlecode.com/files/jiql.jar o Place Jiql.jar under WEB-INFlib o Create a connection as follows: String url = "jdbc:jiql://local"; String password = ""; String user = ""; Properties props = new Properties(); props.put("user",user); props.put("password",password); Class clazz = Class.forName("org.jiql.jdbc.Driver"); Driver driver = (Driver) clazz.newInstance(); Connection Conn = driver.connect(url,props);
  • 19. Supported SQL syntax on JIQL o SUPPORTED SQL SYNTAX o INSERT o SELECT o DROP o CREATE o CONSTRAINT o DISTINCT o FOREIGN KEY o PRIMARY KEY o ALTER o AND o OR o JOINS o STRING FUNCTIONS o ORDER
  • 20. Supported JDBC statements in JIQL o * Connection o * Statement o * ResultSet o * PreparedStatement o * Driver o * DatabaseMetaData o * ResultSetMetaData
  • 21. Castles in the Cloud GaeVFS: a virtual filesystem on Bigtable
  • 22. What is GaeVFS ? o Filesystem abstraction over Google BigTable (To enable writes to the “local filesystem” in Google's Cloud) o o Implemented as an Apache Commons VFS plugin o o Uses Memcache too under the hood o o The URI format is gae://path o o Best to use relative paths
  • 23. Using GaeVFS import org.apache.commons.vfs.FileSystemManager; import org.apache.commons.vfs.FileObject; import com.newatlanta.commons.vfs.provider.gae.GaeVFS; . GaeVFS.setRootPath( getServletContext.getRealPath( "/" ) ); . FileSystemManager fsManager = GaeVFS.getManager(); FileObject tmpDir = fsManager.resolveFile( "gae://WEB-INF/tmp" ); if ( !tmpDir.exists() ) { tmpDir.createFolder(); }
  • 24. Clear the Cache for GAEVFS Clear the cache at the end of every servlet request! public void doGet( HttpServletRequest req, HttpServletResponse res ) throws ServletException, IOException { try { // ...process request... } finally { GaeVFS.clearFilesCache(); } }
  • 25. Close the connection to GaeVFS It's good practice to close the connection when your servlet is destroyed. public void destroy() { GaeVFS.close(); // this is not strictly required, but good practice }
  • 26. Castles in the Cloud RESTlets 2/27/2009
  • 27. What is the RESTlet project? o An open source project to make RESTful web services easy to implement in Java. o o There's a special build for App Engine, a snapshot tagged “unstable” o
  • 28. How to use RESTlets on App Engine o Create a new project with the Eclipse plugin o o Add "org.restlet.gae.jar" to the war/WEB-INF/lib directoy o o Add "org.restlet.gae.jar" to the project build path o o o
  • 29. Create a simple RESTful resource package simpleREST; import org.restlet.resource.Get; import org.restlet.resource.ServerResource; //simple resource public class simpleResource extends ServerResource { @Get public String represent() { return "A word from the Cloud gods "; } }
  • 30. Create a parent app for resource package simpleREST; import org.restlet.Application; import org.restlet.Restlet; import org.restlet.routing.Router; public class simpleRESTApplication extends Application { @Override public synchronized Restlet createRoot() { Router router = new Router(getContext()); router.attachDefault(simpleResource.class); return router; } }
  • 31. Castles in the Cloud Scheduled tasks 2/27/2009
  • 32. Scheduled tasks on App Engine o Aka “Cron jobs” o Vital for housekeeping and maintenance tasks, especially on the database o Useful for various polling exercises o Significant in a system that is single-threaded and usually activated by incoming HTTP request. o o “Web hook pattern”: Cron jobs work by invoking a URL, just like an incoming web-request o Maximum 20 cron jobs per app o Max one per minute (use “webcron” service to overcome limit)
  • 33. cron.xml in WEB-INF directory <?xml version="1.0" encoding="UTF-8"?> <cronentries> <cron> <url>/polling</url> <description>Poll for some event every couple of minutes</description> <schedule>every 2 minutes</schedule> </cron> <cron> <url>/stats</url> <description>Send out statistics every week</description> <schedule>every monday 07:30</schedule> <timezone>America/New_York</timezone> </cron> </cronentries>
  • 34. English syntax in the cron schedule every 5 minutes every 12 hours 2nd,third mon,wed,thu of march 17:00 every monday 09:00 1st monday of sep,oct,nov 17:00 every day 00:00
  • 35. Updating your cron jobs o Update the whole “app” via Eclipse plugin or on the command-line: appcfg update Update just your cron-jobs appcfg update-cron
  • 36. Cannibalising Google App Engine JRuby-on-Rails Photo credit: http://www.flickr.com/photos/geoftheref/
  • 37. JRuby o One of the popular new dynamic languages on the JVM o Highest performing Ruby implementation o Excellent compatibility with “the standard Ruby” (MRI) o Rails runs without code-changes o Native Rails plugins will not run on Jruby o Core developers lately moved from SUN to Engine Yard o o The next four slides are a recipe for JRuby-on-Rails...
  • 38. Generate a Rails app o Install the gems $ sudo gem install google-appengine o Assign an app identifier Simply create an application at appspot.com o Generate your rails app $ rails my-app; cd my-app;
  • 39. Configure Rack (middleware for Rails) o Generate a rackup file: In the same folder, save this as config.ru. require 'appengine-rack' AppEngine::Rack.configure_app( :application => 'my-app', :version => 1 ) %w{R db/ doc/ log/ script/ test/ tmp/}.each do |x| AppEngine::Rack.app.resource_files.exclude "/#{x}**" end ENV['RAILS_ENV'] = AppEngine::Rack.environment require 'config/environment' run ActionController::Dispatcher.new
  • 40. Install gems (libraries) & generate model o Check gem sources for datamapper $ gem sources o Only when missing, add it like this... $ sudo gem source -a http://gems.datamapper.org o Install gems into your app $ appcfg.rb gem install rails dm-appengine rails_datamapper o #Generate a model $ ./script/generate dm_model book title:string summary:text image_url:string available:boolean pages:integer -f --skip-timestamps --skip-migration
  • 41. Run app locally & deploy to Cloud o Run your app locally $ dev_appserver.rb . o Open up the console $ appcfg.rb run -S irb > require 'config/environment' o Deploy to App Engine $ appcfg.rb update .
  • 42. Cannibalising Google App Engine Task Queues for background processing :Image courtesy the SeaWiFS Project, NASA/Goddard Space Flight Center & ORBIMAGE Visible Earth Project: http://visibleearth.nasa.gov/view_rec.php?id=2600
  • 43. Task Queues: Background processing o Motivation: App Engine is single-threaded. o Place small, discrete units of work on a queue. o App Engine monitors work-queue & executes task in background. o o Configure queues in web.xml (in WEB-INF directory) o Without any configuration, each app gets a queue named “default” o Default rate: 5 tasks per second o o Tasks comprises: data payload and code to process it. o Web-hook pattern o Asynchronous, parallel processing in background
  • 44. Add a task to a queue import com.google.appengine.api.labs.taskqueue.Queue; import com.google.appengine.api.labs.taskqueue.QueueFactory; import static com.google.appengine.api.labs.taskqueue.TaskOptions.Builder.*; // ... Queue queue = QueueFactory.getDefaultQueue(); queue.add(url("/worker").param("key", key)) queue.add(url("/path/to/my/worker")); queue.add(url("/path?a=b&c=d").method("GET"));
  • 45. Cannibalising Google App Engine XMPP: instant messages & peer-to-peer Photo credit: NOAA Photo Library and NSSL: http://www.photolib.noaa.gov/nssl/nssl0113.htm
  • 46. XMPP – Instant message protocol o “Jabber” peer-to-peer protocol for instant messaging o Standardised as “XML Message Passing Protocol“ o o IM protocol in use by “Google Talk” o Twitter uses XMPP under-the-hood o o An enabling technology of “The realtime Web” o Makes peer-to-peer architectures possible with App Engine o o XMPP is useful for all kinds of chat / IM applications o Also for interprocess communication.
  • 47. Construct an instant message... import com.google.appengine.api.xmpp.JID; import com.google.appengine.api.xmpp.Message; import com.google.appengine.api.xmpp.MessageBuilder; import com.google.appengine.api.xmpp.SendResponse; import com.google.appengine.api.xmpp.XMPPService; import com.google.appengine.api.xmpp.XMPPServiceFactory; // ... JID jid = new JID("somebody@gmail.com"); String msgBody = "Hello from the Cloud!"; Message msg = new MessageBuilder() .withRecipientJids(jid) .withBody(msgBody) .build();
  • 48. Check on “presence” & send message boolean messageSent = false; XMPPService xmpp = XMPPServiceFactory.getXMPPService(); if (xmpp.getPresence(jid).isAvailable()) { SendResponse status = xmpp.sendMessage(msg); messageSent = (status.getStatusMap().get(jid) == SendResponse.Status.SUCCESS); } if (!messageSent) { // Send an email message instead... }
  • 49. Configuration to receive messages o Web hook pattern again o Configuration: edit appengine-web.xml file; include these lines: <inbound-services> <service>xmpp_message</service> </inbound-services> o when App Engine receives a chat message for the app, it makes an HTTP POST to: /_ah/xmpp/message/chat/
  • 50. Code to receive messages o import java.io.IOException; o import javax.servlet.http.*; o import com.google.appengine.api.xmpp.JID; o import com.google.appengine.api.xmpp.Message; o import com.google.appengine.api.xmpp.XMPPService; o import com.google.appengine.api.xmpp.XMPPServiceFactory; o @SuppressWarnings("serial") o public class XMPPReceiverServlet extends HttpServlet { o public void doPost(HttpServletRequest req, HttpServletResponse res) o throws IOException { o XMPPService xmpp = XMPPServiceFactory.getXMPPService(); o Message message = xmpp.parseMessage(req); o JID fromJid = message.getFromJid(); o String body = message.getBody(); // ....... o } }
  • 51. Map the URL to the servlet To map the servlet to the XMPP URL path, add the following in the web.xml file, inside the <web-app> element: <servlet> <servlet-name>xmppreceiver</servlet-name> <servlet-class>XMPPReceiverServlet</servlet-class> </servlet> <servlet-mapping> <servlet-name>xmppreceiver</servlet-name> <url-pattern>/_ah/xmpp/message/chat/</url-pattern> </servlet-mapping>
  • 52. Clojure • Elegant • Functional • Dynamic • Fast • Excellent concurrency (unusable on App Engine, unfortunately) • Direct access to Java • Lisp Reloaded (Feature-list above by Stuart Halloway.)
  • 53. Running Clojure on App Engine • How to run Servlets on App Engine with Clojure http://fragmental.tw/2009/04/08/clojure-on-google-app-engine/ (“Getting Cloudy”, by Phillip Calçado of Thoughtworks) •How to run Compojure (a web-framework for Clojure) on App Engine http://elhumidor.blogspot.com/2009/04/clojure-on-google-appengine.html (“Clojure on Google App Engine”, by John Hume of Thoughtworks)
  • 54. Servlets with Clojure (sample-code) Code from Phillip Calçado's BestMateServlet.clj (ns BestMateServlet (:gen-class :extends javax.servlet.http.HttpServlet) (:use mate)) (defn -doGet [_ req resp] (process-request req resp))
  • 55. Resources for Clojure o Disclojure: http://disclojure.org/where-to-start/ – tutorial – Book – main Clojure site – Plugins for IDEs – Videos • VimClojure: http://kotka.de/projects/clojure/vimclojure.html
  • 56. Cannibalising Google App Engine Store Static Content on App Engine Photo credit: Nikolai, http://www.blogger.com/profile/13624955858359692394
  • 57. Getting started with App Engine To use App Engine, you can begin with your current Gmail address and password or with a new Google identity you create for the purpose. Sign up for App Engine: http://appengine.google.com (You'll need to receive an SMS on your mobile 'phone.)
  • 58. Create a project on App Engine Log into http://appengine.google.com and create a new app. (Many project-names will be taken already, since App Engine shares its namespace with Gmail.) The project name you choose will determine the default domain-name. For example, if "blissedbride" is available, your app's default domain will be: http://blissedbride.appspot.com
  • 59. Store static web-assets in Google's Cloud Install the plugin from e.g. (for Galileo) http://dl.google.com/eclipse/plugin/3.5 Select the File menu > New > Web Application Project Supposing you picked "blissedbride" as your project-name. Navigate to this directory: workspace/blissedbride/war" Dump all of your web-assets in this directory. Your index.html will overwrite the placeholder file. And the rest of your HTML files can be here too. You can put other web-assets in subdirectories.
  • 60. Cannibalising Google App Engine Wrapping up Photo credit: NOAA Photo Library, NOAA Central Library; OAR/ERL/National Severe Storms Lab
  • 61. Book (I) Developing with Google App Engine by Eugene Ciurana Published: Feb 2009 http://apress.com/
  • 62. Book (II) Google App Engine in Action by Noah Gift & Michael Orr To be published: May 2009 http://www.manning.com/gift/
  • 63. Book (III) Google App Engine by Dan Sanderson To be published: Sept 2009 http://oreilly.com/
  • 64. Resources for Google App Engine • Google App Engine: http://appengine.google.com • Videos from conference Google held in 2009: o http://sites.google.com/site/io/ • Chat with developers from the App Engine team o first Wednesday of the month: 3am next day (Dublin/London time). o third Wednesday of the month: 5pm (Dublin/London time). o IRC channel: #appengine on irc.freenode.net o http://en.wikipedia.org/wiki/List_of_IRC_clients • Google Group o http://groups.google.com/group/google-appengine/ • Paper on Bigtable: o http://labs.google.com/papers/bigtable.html
  • 65. Credits • Photos used under Creative Commons license: • Photo Credits o http://www.flickr.com/photos/notsogoodphotography/ o http://www.flickr.com/photos/barto/ o http://www.flickr.com/photos/kitoy/ o http://www.flickr.com/photos/wolfgangstaudt/ o http://www.flickr.com/photos/motumboe/ o http://www.flickr.com/photos/ragnar1984/ o http://www.flickr.com/photos/janekm/ o http://www.flickr.com/photos/raylopez/ o http://www.flickr.com/photos/pagedooley/ o http://www.flickr.com/photos/gustty/ o http://www.flickr.com/photos/mike9alive/ o http://www.flickr.com/photos/tambako/ o http://www.flickr.com/photos/nate o http://www.flickr.com/photos/freewine/
  • 66. Contact Information Kevin Noonan kevin_noonan on Twitter http://www.calbane.com

×