Google Cloud Technologies Overview


Published on

This is a presentation on Google Cloud technologies by Google Developer Advocate, Chris Schalk at Cloud Computing Expo on Nov 4th 2010.

Published in: Technology, Business

Google Cloud Technologies Overview

  1. Google Cloud Technologies Overview  Chris Schalk  Google Developer Advocate 
  2. 2 What is  cloud  compu/ng? 
  3. 3 Just Kidding ;‐) 
  4. Google Cloud Development Technologies Google BigQuery  Google   Predic/on API  Google Storage  Google App Engine  Google App Engine for Business (new)  ExisAng  New! 
  5. •  Part I Intro to App Engine  •  App Engine Details  •  Development Tools  •  App Engine for Business  • Part II – Google’s new cloud technologies  Agenda 
  6. 6  IaaS  APaaS  SaaS  Source: Gartner AADI Summit Dec 2009  Cloud Computing as Gartner Sees It
  7. 7  IaaS value proposition… 
  8. 8  The PaaS value proposition
  9. • Easy to build  • Easy to maintain  • Easy to scale  9 Google App Engine 
  10. 10 500M+  daily  Pageviews  250,000+  Developers  100,000+  Apps  10 By the numbers 
  11. 11  Some App Engine Partners 
  12. 12  App Engine @ 
  13. 13  >62M Users  Social Networking at Scale 
  14. gigy   Socialize  14 
  15. gigy   Socialize  15  "Although we typically host all our services in‐house,   on our own infrastructure, we felt that GAE would be a beer fit   for the live chat feature because of its unique traffic paern, which   is characterized by very low traffic most of the Ame with very high   bursts during high profile events."  Raviv Pavel, Gigya VP of Research and Development  hp://‐means‐flexibility.html#links  Flexible Scalability 
  16. App Engine Details  16
  17. Cloud development in a box 17 •  Downloadable SDK  •  ApplicaAon runAmes  •  Java, Python  •  Local development tools  •  Eclipse plugin, AppEngine Launcher  •  Specialized applicaAon services   •  Cloud based dashboard  •  Ready to scale   •  Built in fault tolerance, load balancing 
  18. Specialized Services 18  Blobstore Images  Mail  XMPP  Task Queue  Memcache  Datastore  URL Fetch  User Service 
  19. Language runtimes 19  Duke, the Java mascot  Copyright © Sun Microsystems Inc., all rights reserved. 
  20. Ensuring Portability 20 
  21. Extended Language support through JVM •  Java •  Scala •  JRuby (Ruby) •  Groovy •  Quercus (PHP) •  Rhino (JavaScript) •  Jython (Python) 21  Duke, the Java mascot  Copyright © Sun Microsystems Inc., all rights reserved. 
  22. Always free to get started •  ~5M pageviews/month  •  6.5 CPU hrs/day  •  1 GB storage  •  650K URL Fetch calls/day  •  2,000 recipients emailed  •  1 GB/day bandwidth  •  100,000 tasks enqueued  •  650K XMPP messages/day  22
  23. Application Platform Management 23 
  24. App Engine Dashboard 24 
  25. App Engine Health History 25 
  26. Development Tools for App Engine 26 
  27. Google App Engine Launcher  27 
  28. SDK Console 28 
  29. Google Plugin for Eclipse  29 
  30. Two+ years in review  30  Apr 2008 Python launch May 2008 Memcache, Images API Jul 2008 Logs export Aug 2008 Batch write/delete Oct 2008 HTTPS support Dec 2008 Status dashboard, quota details Feb 2009 Billing, larger files Apr 2009 Java launch, DB import, cron support, SDC May 2009 Key-only queries Jun 2009 Task queues Aug 2009 Kindless queries Sep 2009 XMPP Oct 2009 Incoming email Dec 2009 Blobstore Feb 2010 Datastore cursors, Appstats Mar 2010 Read policies, IPv6 May 2010 App Engine for Business
  31. Introducing App Engine for Business  31 Same scalable cloud pla3orm, but designed for the Enterprise  App Engine for Business 
  32. Google App Engine for Business  Details  •  Enterprise applicaAon management  –  Centralized domain console (preview available today)  •  Enterprise reliability and support  –  99.9% Service Level Agreement  –   Direct support  •  Hosted SQL  –  RelaAonal SQL database in the cloud (preview available today)  •  SSL on your domain  •  Secure by default  –  Integrated Single Sign On (SSO)  •  Pricing that makes sense  –  Apps cost $8 per user, up to $1000 max per month  32  Google App Engine for Business
  33. Enterprise App Development with Google  33  Build your own Google App Engine for Business Buy from others Google Apps Marketplace Enterprise Firewall  Enterprise Data  AuthenAcaAon  Enterprise Services  User Management  Buy from Google Google Apps for Business Enterprise Application Platform
  34. 34  App Engine for Business Roadmap Enterprise Administration Console Preview (signups available) Direct Support Preview (signups available) Hosted SQL Preview (signups available) Service Level Agreement Available Q4 2010 (Draft published) Enterprise billing Available Q4 2010 Custom Domain SSL Limited Release EOY 2010
  35. App Engine Resources Get started with App Engine  •  hp://  Read up on App Engine for Business and become a trusted tester  •  hp://  • <‐ sign up! 
  36. Part II - Google’s new Cloud Technologies Topics covered •  Google Storage for Developers •  Prediction API (machine learning) •  BigQuery
  37. Google Storage for Developers Store your data in Google's cloud
  38. What Is Google Storage? •  Store your data in Google's cloud  o  any format, any amount, any Ame  •  You control access to your data  o  private, shared, or public  •   Access via Google APIs or 3rd party tools/libraries 
  39. Sample Use Cases StaAc content hosAng  e.g. staAc html, images, music, video  Backup and recovery  e.g. personal data, business records  Sharing  e.g. share data with your customers  Data storage for applicaAons  e.g. used as storage backend for Android, AppEngine, Cloud based apps  Storage for ComputaAon  e.g. BigQuery, PredicAon API 
  40. Google Storage Benefits High Performance and Scalability         Backed by Google infrastructure   Strong Security and Privacy          Control access to your data  Easy to Use  Get started fast with Google & 3rd party tools 
  41. Google Storage Technical Details •  RESTful API   o  Verbs: GET, PUT, POST, HEAD, DELETE   o  Resources: identified by URI  o  Compatible with S3   •  Buckets   o  Flat containers   •  Objects   o  Any type  o  Size: 100 GB / object  •  Access Control for Google Accounts   o  For individuals and groups   •  Two Ways to Authenticate Requests   o  Sign request using access keys   o  Web browser login
  42. Performance and Scalability •  Objects of any type and 100 GB / Object  •  Unlimited numbers of objects, 1000s of buckets  •  All data replicated to mulAple US data centers  •  Leveraging Google's worldwide network for data delivery   •  Only you can use bucket names with your domain names   •  Read‐your‐writes data consistency  •  Range Get  
  43. Security and Privacy Features •  Key‐based authenAcaAon  •  AuthenAcated downloads from a web browser  •  Sharing with individuals  •  Group sharing via Google Groups   •  Access control for buckets and objects  •  Set Read/Write/List permissions   
  44. Demo •  Tools:  o  GS Manager  o  GSUAl  •  Upload / Download 
  45. Google Storage usage within Google HaiA Relief Imagery  USPTO data  Partner ReporAng  Google BigQuery  Google   Predic/on API  Partner ReporAng 
  46. Some Early Google Storage Adopters 
  47. Google Storage - Pricing o  Storage   $0.17/GB/Month  o  Network   Upload - $0.10/GB   Download   $0.15/GB Americas / EMEA   $0.30/GB  APAC  o  Requests   PUT, POST, LIST - $0.01 / 1000 Requests   GET, HEAD - $0.01 / 10000 Requests
  48. Google Storage - Availability •  Limited preview in US currently   o  100GB free storage and network from Google per account  o  Sign up for waitlist at hp://  •  Note: Non US preview available on case‐by‐case basis 
  49. Google Storage Summary •  Store any kind of data using Google's cloud infrastructure  •  Easy to Use APIs  •  Many available tools and libraries  o  gsuAl, GS Manager  o  3rd party:   Boto, CloudBerry, CyberDuck, JetS3t, and more 
  50. Google Prediction API Google's prediction engine in the cloud
  51. Introducing the Google Prediction API •  Google's sophisticated machine learning technology •  Available as an on-demand RESTful HTTP web service
  52. How does it work?  "english"  The quick brown fox jumped over the lazy  dog.  "english"  To err is human, but to really foul things up  you need a computer.  "spanish"  No hay mal que por bien no venga.  "spanish"  La tercera es la vencida.  ?  To be or not to be, that is the quesAon.  ?  La fe mueve montañas.  The Prediction API finds relevant features in the sample data during training. The PredicAon API  later searches for  those features  during predicAon. 
  53. A virtually endless number of applicaAons...  Customer Sentiment TransacAon  Risk  Species  IdenAficaAon  Message  RouAng  Legal Docket  ClassificaAon  Suspicious  AcAvity  Work Roster  Assignment  Recommend  Products  PoliAcal  Bias  UpliS  MarkeAng  Email  Filtering  DiagnosAcs  Inappropriate  Content  Career  Counselling  Churn  PredicAon  ... and many more ... 
  54. A PredicAon API Example  AutomaAcally categorize and respond to emails by language  •  Customer: ACME Corp, a multinational organization •  Goal: Respond to customer emails in their language •  Data: Many emails, tagged with their languages •  Outcome: Predict language and respond accordingly
  55. Using the Prediction API 1. Upload  2. Train  Upload your training data to  Google Storage   Build a model from your data  Make new predicAons 3. Predict  A simple three step process... 
  56. Step 1: Upload  Upload your training data to Google Storage  •  Training data: outputs and input features •  Data format: comma separated value format (CSV) "english","To err is human, but to really ..."  "spanish","No hay mal que por bien no venga."  ... Upload to Google Storage  gsutil cp ${data} gs://yourbucket/${data}
  57. Step 2: Train  Create a new model by training on data  To train a model: POST prediction/v1.1/training?data=mybucket%2Fmydata Training runs asynchronously. To see if it has finished: GET prediction/v1.1/training/mybucket %2Fmydata {"data":{ "data":"mybucket/mydata", "modelinfo":"estimated accuracy: 0.xx"}}}
  58. Step 3: Predict  Apply the trained model to make predicAons on new data  POST prediction/v1.1/query/mybucket%2Fmydata/predict { "data":{ "input": { "text" : [ "J'aime X! C'est le meilleur" ]}}}
  59. Step 3: Predict  Apply the trained model to make predicAons on new data  POST prediction/v1.1/query/mybucket%2Fmydata/predict { "data":{ "input": { "text" : [ "J'aime X! C'est le meilleur" ]}}} { data : { "kind" : "prediction#output", "outputLabel":"French", "outputMulti" :[ {"label":"French", "score": x.xx} {"label":"English", "score": x.xx} {"label":"Spanish", "score": x.xx}]}}
  60. Step 3: Predict  Apply the trained model to make predicAons on new data  import httplib header = {"Content-Type" : "application/json"} #...put new data in JSON format in params variable conn = httplib.HTTPConnection("")conn.request("POST", "/prediction/v1.1/query/mybucket%2Fmydata/predict”, params, header) print conn.getresponse() An example using Python 
  61. PredicAon API CapabiliAes  Data •  Input Features: numeric or unstructured text •  Output: up to hundreds of discrete categories Training •  Many machine learning techniques •  Automatically selected •  Performed asynchronously Access from many platforms: •  Web app from Google App Engine •  Apps Script (e.g. from Google Spreadsheet) •  Desktop app
  62. PredicAon API v1.1  ‐ new features  •  Updated Syntax •  Multi-category prediction o  Tag entry with multiple labels •  Continuous Output o  Finer grained prediction rankings based on multiple labels •  Mixed Inputs o  Both numeric and text inputs are now supported Can combine continuous output with mixed inputs
  63. Google BigQuery Interactive analysis of large datasets in Google's cloud
  64. Introducing Google BigQuery •  Google's large data adhoc analysis technology o  Analyze massive amounts of data in seconds •  Simple SQL-like query language •  Flexible access o  REST APIs, JSON-RPC, Google Apps Script
  65. Why BigQuery?   Working with large data is a challenge 
  66. Many Use Cases ...  Spam Trends  DetecAon  Web Dashboards  Network  OpAmizaAon  InteracAve Tools 
  67. Key CapabiliAes of BigQuery  •  Scalable: Billions of rows •  Fast: Response in seconds •  Simple: Queries in SQL •  Web Service o  REST o  JSON-RPC o  Google App Scripts
  68. Using BigQuery 1. Upload  2. Import  Upload your raw data to  Google Storage   Import raw data into BigQuery table  Perform SQL queries on table 3. Query  Another simple three step process... 
  69. WriAng Queries  Compact subset of SQL o  SELECT ... FROM ... WHERE ... GROUP BY ... ORDER BY ... LIMIT ...; Common functions o  Math, String, Time, ... Statistical approximations o  TOP o  COUNT DISTINCT
  70. BigQuery via REST  GET /bigquery/v1/tables/{table name} GET /bigquery/v1/query?q={query} Sample JSON Reply: { "results": { "fields": { [ {"id":"COUNT(*)","type":"uint64"}, ... ] }, "rows": [ {"f":[{"v":"2949"}, ...]}, {"f":[{"v":"5387"}, ...]}, ... ] } } Also supports JSON-RPC
  71. Security and Privacy  Standard Google Authentication •  Client Login •  OAuth •  AuthSub HTTPS support •  protects your credentials •  protects your data Relies on Google Storage to manage access
  72. Large Data Analysis Example  Wikimedia Revision history data from: hp://‐ latest‐pages‐meta‐history.xml.7z  Wikimedia Revision History 
  73. Using BigQuery Shell  Python DB API 2.0 + B. Clapper's sqlcmd
  74. BigQuery from a Spreadsheet 
  75. BigQuery from a Spreadsheet 
  76. Recap  •  Google Storage o  High speed data storage on Google Cloud •  Prediction API o  Google's machine learning technology able to predict outcomes based on sample data •  BigQuery o  Interactive analysis of very large data sets o  Simple SQL query language access
  77. Further info available at:  •  Google Storage for Developers o •  Prediction API o •  BigQuery o
  78. Q&A
  79. Thank You!  Chris Schalk  Google Developer  Advocate  hp://