Cannibalising The Google App Engine


Published on

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Cannibalising The Google App Engine

  1. 1. Cannibalising Google App Engine Kevin Noonan Calbane Ltd. IT@Cork, 30th September, 2009
  2. 2. Cannibalising Google App Engine Introduction
  3. 3. A working definition of Cloudware • Outsourced Virtualization o Virtual server running on outsourced server farm
  4. 4. The Cloud Computing Pyramid •Michael Sheehan of GoGrid devised this classification
  5. 5. What's Cloudware good for? (I) • Guerilla development o Pay-per-use model is ideal for startups o Also good for guerillas inside the enterprise...
  6. 6. What's Cloudware good for? (II) • Scaling up o Clone the virtual server o pay for extra resources (bandwidth, storage, RAM, etc.)
  7. 7. What's Cloudware good for? (III) • In general, outsourcing IT infrastructure o pay someone else to run your servers, network, database...
  8. 8. Python: The Original of the Species
  9. 9. Java: the All New Flavour
  10. 10. Languages on App Engine's JVM • Java • JRuby • Jython • BeanShell • JavaScript • Scala • Groovy • … • Will it Play in App Engine?
  11. 11. What is Google App Engine? • Complete web stack • Runs on Google's infrastructure • Bigtable Datastore o Like a sorted hashtable o Not a relational database o Use JDO or JPA (with Java) for datastore work • Memcache • Image API • Mail API • SDK for development on Windows/Mac/Linux o Run & test locally before uploading to Cloud
  12. 12. Limitations of Google App Engine • Applications run in “The Sandbox”: • Many Java libraries will not work (but many have been patched or will be). See the JRE whitelist: No “long-lived” processing (requests must return in 30 seconds). • HTTPS only on o and only with Google's certificate. o Due to limitation of SSL protocol, as applied to Google's Cloud.
  13. 13. Billing on Google App Engine • True pay-per-use o free quotas & no bill to pay if quotas not exceeded. o quite different model to AWS • Developer can assign budget per resource. • Prices o $0.10 per CPU core-hour o $0.15 per GB-month of storage o $0.12 per GB outgoing bandwidth o $0.10 per GB incoming bandwidth o $0.0001 per email recipient for emails sent by the application
  14. 14. Free Quotas on App Engine •Daily outgoing bandwidth: 10 GB •Daily incoming bandwidth 10 GB •Daily requests 1.3 million • Daily CPU-hours: 46 • More quotas (mail, datastore, images, memcache)
  15. 15. Other limitations on App Engine • 1 GB of free storage per app. •100 MB Memcache storage limit • App Engine apps can now receive requests and send responses of up to 10MB. • Caveat: 1 MB limit on datastore/memcache API call • You can deploy up to 1,000 code and 1,000 static files with an app. Workaround: use ZipImport (with Python) to store files in a Zip-archive: • 150MB cumulative limit on an app's code and static files (as opposed to storage in the datastore).
  16. 16. Cannibalising Google App Engine JIQL for JDBC emulation
  17. 17. What is JIQL? o o Abstraction layer over the Google datastore, Bigtable o o Bigtable is non-relational, but JIQL simulates a relational database o o JIQL enables the use of JDBC and SQL with Google Bigtable
  18. 18. How to use JIQL o Download Jiql.jar from o Place Jiql.jar under WEB-INFlib o Create a connection as follows: String url = "jdbc:jiql://local"; String password = ""; String user = ""; Properties props = new Properties(); props.put("user",user); props.put("password",password); Class clazz = Class.forName("org.jiql.jdbc.Driver"); Driver driver = (Driver) clazz.newInstance(); Connection Conn = driver.connect(url,props);
  20. 20. Supported JDBC statements in JIQL o * Connection o * Statement o * ResultSet o * PreparedStatement o * Driver o * DatabaseMetaData o * ResultSetMetaData
  21. 21. Castles in the Cloud GaeVFS: a virtual filesystem on Bigtable
  22. 22. What is GaeVFS ? o Filesystem abstraction over Google BigTable (To enable writes to the “local filesystem” in Google's Cloud) o o Implemented as an Apache Commons VFS plugin o o Uses Memcache too under the hood o o The URI format is gae://path o o Best to use relative paths
  23. 23. Using GaeVFS import org.apache.commons.vfs.FileSystemManager; import org.apache.commons.vfs.FileObject; import com.newatlanta.commons.vfs.provider.gae.GaeVFS; . GaeVFS.setRootPath( getServletContext.getRealPath( "/" ) ); . FileSystemManager fsManager = GaeVFS.getManager(); FileObject tmpDir = fsManager.resolveFile( "gae://WEB-INF/tmp" ); if ( !tmpDir.exists() ) { tmpDir.createFolder(); }
  24. 24. Clear the Cache for GAEVFS Clear the cache at the end of every servlet request! public void doGet( HttpServletRequest req, HttpServletResponse res ) throws ServletException, IOException { try { // ...process request... } finally { GaeVFS.clearFilesCache(); } }
  25. 25. Close the connection to GaeVFS It's good practice to close the connection when your servlet is destroyed. public void destroy() { GaeVFS.close(); // this is not strictly required, but good practice }
  26. 26. Castles in the Cloud RESTlets 2/27/2009
  27. 27. What is the RESTlet project? o An open source project to make RESTful web services easy to implement in Java. o o There's a special build for App Engine, a snapshot tagged “unstable” o
  28. 28. How to use RESTlets on App Engine o Create a new project with the Eclipse plugin o o Add "org.restlet.gae.jar" to the war/WEB-INF/lib directoy o o Add "org.restlet.gae.jar" to the project build path o o o
  29. 29. Create a simple RESTful resource package simpleREST; import org.restlet.resource.Get; import org.restlet.resource.ServerResource; //simple resource public class simpleResource extends ServerResource { @Get public String represent() { return "A word from the Cloud gods "; } }
  30. 30. Create a parent app for resource package simpleREST; import org.restlet.Application; import org.restlet.Restlet; import org.restlet.routing.Router; public class simpleRESTApplication extends Application { @Override public synchronized Restlet createRoot() { Router router = new Router(getContext()); router.attachDefault(simpleResource.class); return router; } }
  31. 31. Castles in the Cloud Scheduled tasks 2/27/2009
  32. 32. Scheduled tasks on App Engine o Aka “Cron jobs” o Vital for housekeeping and maintenance tasks, especially on the database o Useful for various polling exercises o Significant in a system that is single-threaded and usually activated by incoming HTTP request. o o “Web hook pattern”: Cron jobs work by invoking a URL, just like an incoming web-request o Maximum 20 cron jobs per app o Max one per minute (use “webcron” service to overcome limit)
  33. 33. cron.xml in WEB-INF directory <?xml version="1.0" encoding="UTF-8"?> <cronentries> <cron> <url>/polling</url> <description>Poll for some event every couple of minutes</description> <schedule>every 2 minutes</schedule> </cron> <cron> <url>/stats</url> <description>Send out statistics every week</description> <schedule>every monday 07:30</schedule> <timezone>America/New_York</timezone> </cron> </cronentries>
  34. 34. English syntax in the cron schedule every 5 minutes every 12 hours 2nd,third mon,wed,thu of march 17:00 every monday 09:00 1st monday of sep,oct,nov 17:00 every day 00:00
  35. 35. Updating your cron jobs o Update the whole “app” via Eclipse plugin or on the command-line: appcfg update Update just your cron-jobs appcfg update-cron
  36. 36. Cannibalising Google App Engine JRuby-on-Rails Photo credit:
  37. 37. JRuby o One of the popular new dynamic languages on the JVM o Highest performing Ruby implementation o Excellent compatibility with “the standard Ruby” (MRI) o Rails runs without code-changes o Native Rails plugins will not run on Jruby o Core developers lately moved from SUN to Engine Yard o o The next four slides are a recipe for JRuby-on-Rails...
  38. 38. Generate a Rails app o Install the gems $ sudo gem install google-appengine o Assign an app identifier Simply create an application at o Generate your rails app $ rails my-app; cd my-app;
  39. 39. Configure Rack (middleware for Rails) o Generate a rackup file: In the same folder, save this as require 'appengine-rack' AppEngine::Rack.configure_app( :application => 'my-app', :version => 1 ) %w{R db/ doc/ log/ script/ test/ tmp/}.each do |x| "/#{x}**" end ENV['RAILS_ENV'] = AppEngine::Rack.environment require 'config/environment' run
  40. 40. Install gems (libraries) & generate model o Check gem sources for datamapper $ gem sources o Only when missing, add it like this... $ sudo gem source -a o Install gems into your app $ appcfg.rb gem install rails dm-appengine rails_datamapper o #Generate a model $ ./script/generate dm_model book title:string summary:text image_url:string available:boolean pages:integer -f --skip-timestamps --skip-migration
  41. 41. Run app locally & deploy to Cloud o Run your app locally $ dev_appserver.rb . o Open up the console $ appcfg.rb run -S irb > require 'config/environment' o Deploy to App Engine $ appcfg.rb update .
  42. 42. Cannibalising Google App Engine Task Queues for background processing :Image courtesy the SeaWiFS Project, NASA/Goddard Space Flight Center & ORBIMAGE Visible Earth Project:
  43. 43. Task Queues: Background processing o Motivation: App Engine is single-threaded. o Place small, discrete units of work on a queue. o App Engine monitors work-queue & executes task in background. o o Configure queues in web.xml (in WEB-INF directory) o Without any configuration, each app gets a queue named “default” o Default rate: 5 tasks per second o o Tasks comprises: data payload and code to process it. o Web-hook pattern o Asynchronous, parallel processing in background
  44. 44. Add a task to a queue import; import; import static*; // ... Queue queue = QueueFactory.getDefaultQueue(); queue.add(url("/worker").param("key", key)) queue.add(url("/path/to/my/worker")); queue.add(url("/path?a=b&c=d").method("GET"));
  45. 45. Cannibalising Google App Engine XMPP: instant messages & peer-to-peer Photo credit: NOAA Photo Library and NSSL:
  46. 46. XMPP – Instant message protocol o “Jabber” peer-to-peer protocol for instant messaging o Standardised as “XML Message Passing Protocol“ o o IM protocol in use by “Google Talk” o Twitter uses XMPP under-the-hood o o An enabling technology of “The realtime Web” o Makes peer-to-peer architectures possible with App Engine o o XMPP is useful for all kinds of chat / IM applications o Also for interprocess communication.
  47. 47. Construct an instant message... import; import; import; import; import; import; // ... JID jid = new JID(""); String msgBody = "Hello from the Cloud!"; Message msg = new MessageBuilder() .withRecipientJids(jid) .withBody(msgBody) .build();
  48. 48. Check on “presence” & send message boolean messageSent = false; XMPPService xmpp = XMPPServiceFactory.getXMPPService(); if (xmpp.getPresence(jid).isAvailable()) { SendResponse status = xmpp.sendMessage(msg); messageSent = (status.getStatusMap().get(jid) == SendResponse.Status.SUCCESS); } if (!messageSent) { // Send an email message instead... }
  49. 49. Configuration to receive messages o Web hook pattern again o Configuration: edit appengine-web.xml file; include these lines: <inbound-services> <service>xmpp_message</service> </inbound-services> o when App Engine receives a chat message for the app, it makes an HTTP POST to: /_ah/xmpp/message/chat/
  50. 50. Code to receive messages o import; o import javax.servlet.http.*; o import; o import; o import; o import; o @SuppressWarnings("serial") o public class XMPPReceiverServlet extends HttpServlet { o public void doPost(HttpServletRequest req, HttpServletResponse res) o throws IOException { o XMPPService xmpp = XMPPServiceFactory.getXMPPService(); o Message message = xmpp.parseMessage(req); o JID fromJid = message.getFromJid(); o String body = message.getBody(); // ....... o } }
  51. 51. Map the URL to the servlet To map the servlet to the XMPP URL path, add the following in the web.xml file, inside the <web-app> element: <servlet> <servlet-name>xmppreceiver</servlet-name> <servlet-class>XMPPReceiverServlet</servlet-class> </servlet> <servlet-mapping> <servlet-name>xmppreceiver</servlet-name> <url-pattern>/_ah/xmpp/message/chat/</url-pattern> </servlet-mapping>
  52. 52. Clojure • Elegant • Functional • Dynamic • Fast • Excellent concurrency (unusable on App Engine, unfortunately) • Direct access to Java • Lisp Reloaded (Feature-list above by Stuart Halloway.)
  53. 53. Running Clojure on App Engine • How to run Servlets on App Engine with Clojure (“Getting Cloudy”, by Phillip Calçado of Thoughtworks) •How to run Compojure (a web-framework for Clojure) on App Engine (“Clojure on Google App Engine”, by John Hume of Thoughtworks)
  54. 54. Servlets with Clojure (sample-code) Code from Phillip Calçado's BestMateServlet.clj (ns BestMateServlet (:gen-class :extends javax.servlet.http.HttpServlet) (:use mate)) (defn -doGet [_ req resp] (process-request req resp))
  55. 55. Resources for Clojure o Disclojure: – tutorial – Book – main Clojure site – Plugins for IDEs – Videos • VimClojure:
  56. 56. Cannibalising Google App Engine Store Static Content on App Engine Photo credit: Nikolai,
  57. 57. Getting started with App Engine To use App Engine, you can begin with your current Gmail address and password or with a new Google identity you create for the purpose. Sign up for App Engine: (You'll need to receive an SMS on your mobile 'phone.)
  58. 58. Create a project on App Engine Log into and create a new app. (Many project-names will be taken already, since App Engine shares its namespace with Gmail.) The project name you choose will determine the default domain-name. For example, if "blissedbride" is available, your app's default domain will be:
  59. 59. Store static web-assets in Google's Cloud Install the plugin from e.g. (for Galileo) Select the File menu > New > Web Application Project Supposing you picked "blissedbride" as your project-name. Navigate to this directory: workspace/blissedbride/war" Dump all of your web-assets in this directory. Your index.html will overwrite the placeholder file. And the rest of your HTML files can be here too. You can put other web-assets in subdirectories.
  60. 60. Cannibalising Google App Engine Wrapping up Photo credit: NOAA Photo Library, NOAA Central Library; OAR/ERL/National Severe Storms Lab
  61. 61. Book (I) Developing with Google App Engine by Eugene Ciurana Published: Feb 2009
  62. 62. Book (II) Google App Engine in Action by Noah Gift & Michael Orr To be published: May 2009
  63. 63. Book (III) Google App Engine by Dan Sanderson To be published: Sept 2009
  64. 64. Resources for Google App Engine • Google App Engine: • Videos from conference Google held in 2009: o • Chat with developers from the App Engine team o first Wednesday of the month: 3am next day (Dublin/London time). o third Wednesday of the month: 5pm (Dublin/London time). o IRC channel: #appengine on o • Google Group o • Paper on Bigtable: o
  65. 65. Credits • Photos used under Creative Commons license: • Photo Credits o o o o o o o o o o o o o o
  66. 66. Contact Information Kevin Noonan kevin_noonan on Twitter