Google Cloud for Developers - Devfest Manila

21,747 views
18,975 views

Published on

Overview of Google Cloud products for Developers, to build, sell and monetize web apps: Google Apps Marketplace, App Engine, App Engine for Business, Google Storage, Prediction and BigQuery APIs.

Published in: Technology, Business
3 Comments
42 Likes
Statistics
Notes
No Downloads
Views
Total views
21,747
On SlideShare
0
From Embeds
0
Number of Embeds
23
Actions
Shares
0
Downloads
854
Comments
3
Likes
42
Embeds 0
No embeds

No notes for slide

Google Cloud for Developers - Devfest Manila

  1. Google Cloud for Developers Apps, Apps Marketplace, App Engine for Business, Storage, Prediction, Bigquery Patrick Chanezon Developer Advocate #devfest Manila chanezon@google.com http://twitter.com/chanezon July 6 2010 2 Tuesday, July 6, 2010
  2. Tuesday, July 6, 2010
  3. The benefits of Cloud Computing Economics Pay for only what you use TCO OPEX vs CAPEX Operations Day to day: no maintenance Fighting fires: no Pagers Elasticity Focus on your Business 3 Tuesday, July 6, 2010
  4. Build and Buy all your enterprise cloud apps... Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010
  5. Build and Buy all your enterprise cloud apps... Buy from Google Google Apps for Business Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010
  6. Build and Buy all your enterprise cloud apps... Buy from others Buy from Google Google Apps Google Apps Marketplace for Business Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010
  7. Build and Buy all your enterprise cloud apps... Buy from others Buy from Google Build your own Google Apps Google Apps Google App Engine Marketplace for Business for Business Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010
  8. Customers want more Apps Business in the cloud Google Apps 5 Tuesday, July 6, 2010
  9. Google Apps Marketplace Tuesday, July 6, 2010
  10. Marketplace Overview Tuesday, July 6, 2010
  11. Google Apps is Focused on Messaging & Collaboration Gives every employee powerful messaging and collaboration tools without the usual IT hassle and cost Tuesday, July 6, 2010
  12. These customers want more great cloud apps... Messaging Document Management Collaboration Productivity Accounting & Finance Project Management Admin Tools Sales & Marketing Calendaring / Meetings Security & Compliance Customer Management Workflow Tuesday, July 6, 2010
  13. Steps to sell your app to Google Apps customers 1. Build your app: - with any tools and hosting provider you want 2. Integrate your app: - add Single Sign On using OpenID (required) - access over a dozen integration points from Calendar, Contacts, Docs, etc. using OAuth (optional) 3. Sell your app: - to 2M+ businesses & 25M+ users - through Google Apps resellers - to wherever Google Apps goes - One-time fee of $100, 20% rev share starting 2H'10 Tuesday, July 6, 2010
  14. Everything in the Cloud Tuesday, July 6, 2010
  15. Integrating with Google Apps & the Marketplace Manifest Tuesday, July 6, 2010
  16. Demo Google Apps Marketplace Tuesday, July 6, 2010
  17. How it's done - Single Sign On Tuesday, July 6, 2010
  18. Single Sign On OpenID with Google Apps bob@foo.com Welcome Bob! *********** Sounds complicated, but not hard in practice! Tuesday, July 6, 2010
  19. Single Sign On OpenID Libraries Language Libraries Java OpenID4Java, Step2 .NET DotNetOpenAuth php-openid, PHP php-openid-apps-discovery ruby-openid, Ruby ruby-openid-apps-discovery Any RPX, Ping Identity Tuesday, July 6, 2010
  20. Single Sign On - Code Step 1 of 3 - Initialize library function getOpenIDConsumer() {   // Initialize client, storing associations   // in memcache   $store = new Auth_OpenID_MemcachedStore($MEMCACHE);   $consumer = new Auth_OpenID_Consumer($store);      // Enable Google Apps support    GApps_OpenID_EnableDiscovery($consumer);   return $consumer; } Tuesday, July 6, 2010
  21. Single Sign On - Code Step 2 of 3 - Make the request // Create an auth request to the user's domain  $auth_request = getOpenIDConsumer()->begin($domain); // Request email address & name during login  $ax = new Auth_OpenID_AX_FetchRequest;  $attr = Auth_OpenID_AX_AttrInfo::make(       $AX_SCHEMA_EMAIL, 1, 1, 'email'); $attr = Auth_OpenID_AX_AttrInfo::make(       $AX_SCHEMA_FIRSTNAME, 1, 1, 'first'); $attr = Auth_OpenID_AX_AttrInfo::make(       $AX_SCHEMA_LASTNAME, 1, 1, 'last'); $auth_request->addExtension($ax); // Render Javascript/form to post request $form_html = $auth_request->htmlMarkup($REALM,       $RETURN_TO, false,        array('id' => 'openid_message'));  print $form_html; Tuesday, July 6, 2010
  22. Single Sign On - Code Step 3 of 3 - Handle response // Parse the response from identity provider  $response = getOpenIDConsumer()->complete($RETURN_TO);  if ($response->status == Auth_OpenID_SUCCESS) {   // Extract data from response    $openid = $response->getDisplayIdentifier();   $ax = new Auth_OpenID_AX_FetchResponse();   $ax_resp = $ax->fromSuccessResponse($response);   $email = $ax_resp->data[$AX_SCHEMA_EMAIL][0];   $firstName = $ax_resp->data[$AX_SCHEMA_FIRSTNAME][0];   $lastName = $ax_resp->data[$AX_SCHEMA_LASTTNAME][0];   // Map to user in DB    $user = fetch_user($openid, $email, $firstName,       $lastName); }  Tuesday, July 6, 2010
  23. Single Sign On - Code Manifest <ApplicationManifest>   <Name>Sassy Voice</Name>   <Description>Voice Mail &amp; Messaging</Description>   <Admin>     <Link rel="setup"> http://voice.saasyapp.com/setup.php?domain=${DOMAIN_NAME}     </Link>   </Admin>     <Extension id="navlink" type="link">     <Name>SaasyVoice</Name>     <Url> http://voice.saasyapp.com/login.php?domain=${DOMAIN_NAME}     </Url>    </Extension>   <Extension id="realm" type="openIdRealm">     <Url>http://voice.saasyapp.com/</Url>   </Extension> </ApplicationManifest> Tuesday, July 6, 2010
  24. Single Sign On What we learned about users Claimed ID: http://example.com/openid?id=12345 Email: bob@example.com* First Name: Bob Last Name: Dobbs * Be sure to confirm or white-list trusted providers Tuesday, July 6, 2010
  25. How it's done - Data Access Tuesday, July 6, 2010
  26. 2-Legged OAuth Delegating data access <ApplicationManifest>   ...    <Scope id="userFeed">     <Url>https://apps-apis.google...     <Reason>To get a list of user...   </Scope>   <Scope id="contactsFeed">     <Url>https://www.google.com...      <Reason>To display names of...    </Scope>   <Scope id="docsFeed">      <Url>https://docs.google.com...      <Reason>To export a call log...    </Scope>  </ApplicationManifest> Tuesday, July 6, 2010
  27. Data Access - Code Step 1 of 2 - Initialize the client function getOauthClient() {   $options = array(     'consumerKey' => $CONSUMER_KEY,     'consumerSecret' => $CONSUMER_SECRET,     'signatureMethod' => 'HMAC-SHA1',     'requestScheme' =>Zend_Oauth::REQUEST_SCHEME_HEADER,     'version' => '1.0');   $consumer = new Zend_Oauth_Consumer($options);   $token = new Zend_Oauth_Token_Access();   $httpClient = $token->getHttpClient($options);   return $httpClient; } Tuesday, July 6, 2010
  28. Data Access - Code Step 2 of 2 - Fetching Users with Provisioning API // Initialize client $userClient = new Zend_Gdata_Gapps(getOauthClient()); // Query feed for current user's domain $userQuery = new Zend_Gdata_Gapps_UserQuery(   getCurrentUserDomain()); $usersFeed = $userClient->getUserFeed($userQuery); // Extract data from user feed $users = array(); foreach ($usersFeed as $userEntry) {   $login = $userEntry->getLogin();   $name = $userEntry->getName();   $users[] = array(    'username' => $login->getUsername(),    'firstName' => $name->getGivenName(),    'lastName' => $name->getFamilyName(),    'admin' => $login->getAdmin()); } Tuesday, July 6, 2010
  29. Integration Recap With not a lot of code we added: • Single Sign On with OpenID • Quicker setup with Attribute Exchange, Provisioning API • Integrated user data with Contacts API • Uploading spreadsheets with Docs API Tuesday, July 6, 2010
  30. Gadgets - your real estate in Google Apps Tuesday, July 6, 2010
  31. Gadgets • Many types of gadgets o Gmail sidebar o Calendar sidebar o Sites o Spreadsheets • Two new types of gadgets available this week! o Gmail contextual! o Wave! • All use the OpenSocial gadgets specification Tuesday, July 6, 2010
  32. Gmail contextual gadgets • Detect e-mail content via regular expressions • Display and collect actionable business information directly in Gmail, below each message Tuesday, July 6, 2010
  33. Gmail contextual gadgets Tuesday, July 6, 2010
  34. Gmail contextual gadgets Tuesday, July 6, 2010
  35. Calendar sidebar gadgets • Can detect: o currently-displayed date range o currently-selected event title attendees date/time Tuesday, July 6, 2010
  36. Calendar sidebar gadgets Tuesday, July 6, 2010
  37. Calendar sidebar gadgets Tuesday, July 6, 2010
  38. Sell Tuesday, July 6, 2010
  39. Complete Manifest <ApplicationManifest xmlns="http://schemas.google.com/ ApplicationManifest/2009"> <Name>SaaSy Voice</Name>   <Description>SaaSy Voice</Description>   <Support>     <Link rel="setup" href="https://.../setup.php?domain=${DOMAIN_NAME}"/ >     <Link rel="manage" href="https://.../config.php?domain=$ {DOMAIN_NAME}"/>     <Link rel="deletion-policy" href="http://.../deletion-policy.php"/>     <Link rel="support" href="http://.../support.php"/>   </Support> Tuesday, July 6, 2010
  40. Complete Manifest (continued)   <!-- Link in universal navigation -->    <Extension id="oneBarLink" type="link">      <Name>SaaSy Voice</Name>     <Url>http://.../index.php?domain=${DOMAIN_NAME}</Url>     <Scope ref="provisioningFeed"/>     <Scope ref="contactsFeed"/>      <Scope ref="docsFeed"/>   </Extension>    <!-- Declare our openid.realm -->    <Extension id="realm" type="openIdRealm">     <Url>http://voice.saasyapp.com/marketplace/</Url>   </Extension> Tuesday, July 6, 2010
  41. Complete Manifest (continued)   <Scope id="provisioningFeed">     <Url>https://apps-apis.google.com/a/feeds/user/#readonly</Url>      <Reason>To get a list of users to provision accounts</Reason>   </Scope>   <Scope id="contactsFeed">      <Url>https://www.google.com/m8/feeds/</Url>     <Reason>To display names of people who called</Reason>   </Scope>   <Scope id="docsFeed">     <Url>https://docs.google.com/feeds/</Url>      <Reason>To export a call log</Reason>   </Scope> </ApplicationManifest> Tuesday, July 6, 2010
  42. Listing in the Google Apps Marketplace Tuesday, July 6, 2010
  43. Listing in the Google Apps Marketplace Tuesday, July 6, 2010
  44. Reach 25 million Google Apps Users "We've seen a very meaningful increase in high quality, non-paid lead flow directly attributable to Google Apps customers...Our customers cite Smartsheet's tight integration with Google's Data APIs as a key factor in their decision to purchase." Brent Frei, Smartsheet Tuesday, July 6, 2010
  45. Roadmap Tuesday, July 6, 2010
  46. Roadmap - General • Recent Launches o Gmail contextual gadgets o OAuth support for Gmail's IMAP and SMTP Tuesday, July 6, 2010
  47. Roadmap - Billing • Goals o Simplified invoicing & payments for customers o Consistent experience across apps • Planned 2nd-half of 2010 o Required for installable apps after transition period o Revenue sharing starts once APIs adopted • Features o Unified billing for apps purchased in marketplace o Support for a variety of pricing & licensing models o APIs for customization and up-selling Tuesday, July 6, 2010
  48. Learn More Tuesday, July 6, 2010
  49. Resources • Business and Marketing o http://developer.googleapps.com/marketplace • Technical and Code o http://code.google.com/googleapps/marketplace • Shopping! o http://www.google.com/appsmarketplace • Don't have Google Apps? o http://www.google.com/a Tuesday, July 6, 2010
  50. Build and Buy all your enterprise cloud apps... Buy from others Buy from Google Build your own Google Apps Google Apps Google App Engine Marketplace for Business for Business Google Apps Platform Enterprise Firewall 47 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010
  51. Google App Engine Tuesday, July 6, 2010
  52. Leveraging Google's Leadership in Cloud Computing • Massive data center operations • Purpose built hardware • Multi tenant software platform at Internet scale 49 Tuesday, July 6, 2010
  53. Google App Engine - Easy to build - Easy to maintain - Easy to scale 50 Tuesday, July 6, 2010
  54. Cloud development in a box • SDK & “The Cloud” • Hardware • Networking • Operating system • Application runtime Java, Python o • Static file serving • Services • Fault tolerance • Load balancing 51 Tuesday, July 6, 2010
  55. App Engine Services Memcache Datastore URL Fetch Mail XMPP Task Queue Images Blobstore User Service 52 Tuesday, July 6, 2010
  56. Always free to get started ~5M pageviews/month • 6.5 CPU hrs/day • 1 GB storage • 650K URL Fetch calls/day • 2,000 recipients emailed • 1 GB/day bandwidth • 100,000 tasks enqueued • 650K XMPP messages/day 53 Tuesday, July 6, 2010
  57. Purchase additional resources * * free monthly quota of ~5 million page views still in full effect 54 Tuesday, July 6, 2010
  58. By the numbers 250,000+ Developers 100,000+ Apps 0.5B+ daily Pageviews 55 Tuesday, July 6, 2010
  59. App Engine 56 Tuesday, July 6, 2010
  60. Socialwok 57 Tuesday, July 6, 2010
  61. Social networking at scale 58 Tuesday, July 6, 2010
  62. Social networking at scale >62M Users 3.6M DAUs on Facebook 1.9M DAUs on MySpace Orkut, Bebo, Hi5, Friendster, Hyves, Ning, … 58 Tuesday, July 6, 2010
  63. Chillingo Crystal 59 Gaming meets Social Zombie Dash Angry Birds LITE Underground Meltdown Cogs Mission Deep Sea Speed Forge Guerilla Bob Ravensword: Angry Birds Extreme The Fallen King Tuesday, July 6, 2010
  64. gigy Socialize 60 Tuesday, July 6, 2010
  65. gigy Socialize - traffic 61 Tuesday, July 6, 2010
  66. gigy Socialize - traffic 61 Tuesday, July 6, 2010
  67. App Engine 62 Tuesday, July 6, 2010
  68. Google App Engine for Business Tuesday, July 6, 2010
  69. Build your Enterprise Apps on Google • Easy to Build - Java standards • Easy to Deploy - push-button deployment • Easy to Scale - from small apps to millions of users 64 Tuesday, July 6, 2010
  70. Google App Engine for Business • Centralized administration - controls • Reliability and support - SLA, Premium support • Secure by default - only your users • Pricing that makes sense - pay only for what you use • Enterprise features - hosted SQL, SSL on your domain 65 Tuesday, July 6, 2010
  71. Understanding the Cloud Computing Landscape SaaS PaaS IaaS 66 Source: Gartner AADI Summit Dec 2009 Tuesday, July 6, 2010
  72. Google's Cloud Offerings 1. Our Apps 2. 3rd party Apps: Google Apps Marketplace SaaS 3. ________ PaaS Google App Engine Google Storage for Devs IaaS Machine Learning BigQuery 67 Tuesday, July 6, 2010
  73. Google's Cloud Offerings Your Apps 1. Our Apps 2. 3rd party Apps: Google Apps Marketplace SaaS 3. ________ PaaS Google App Engine Google Storage for Devs IaaS Machine Learning BigQuery 67 Tuesday, July 6, 2010
  74. Domain Console 68 Tuesday, July 6, 2010
  75. Domain Console Like the regular admin console Designed to manage enterprises with a portfolio of apps • Keep track of all apps in a domain • Access Control: view apps, deploy • Global Settings: apply to all apps in the domain • Billing rolling up to single account • DNS configuration done only once: *.ext.example.com • All apps by default for logged in users from domain 69 Tuesday, July 6, 2010
  76. Google Apps Integration • SSO/SSO delegation • APIs for most Google Apps for integration   70 Tuesday, July 6, 2010
  77. Federate your on-premise data 71 Tuesday, July 6, 2010
  78. Secure Data Connector (SDC) 72 Tuesday, July 6, 2010
  79. Using Secure Data Connector 73 Tuesday, July 6, 2010
  80. Using Secure Data Connector Installation - Determine access rules - Configure and install SDC 73 Tuesday, July 6, 2010
  81. Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel 73 Tuesday, July 6, 2010
  82. Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel Serving - User request sent to App Engine - User authenticated - App makes request through tunnel - SDC performs access checks - Results returned 73 Tuesday, July 6, 2010
  83. App Engine for Business Pricing Intranet apps: Each app costs $8 / active user / month Capped at $1,000 / month (i.e. users above 125 are free) Apps are auth-restricted to domain users Development is free Overage charges on Background Analysis/Storage Non intranet apps (external/public/ISV apps): Pricing TBD Postpaid (i.e. billed at the end of month) 74 Tuesday, July 6, 2010
  84. App Engine for Business Support and SLA Paid Support Email based 1000$/month 1h response time on operational issues 8h on development issues SLA 99.9% uptime Service credits from 10% to 100% refund of monthly bill 75 Tuesday, July 6, 2010
  85. Google Developer Qualification Chrome Gadgets Search App Engine JS Maps API KML 3D Extensions 76 Tuesday, July 6, 2010
  86. Two years in review Apr 2008 Python launch May 2008 Memcache, Images API Jul 2008 Logs export Aug 2008 Batch write/delete Oct 2008 HTTPS support Dec 2008 Status dashboard, quota details Feb 2009 Billing, larger files Apr 2009 Java launch, DB import, cron support, SDC May 2009 Key-only queries Jun 2009 Task queues Aug 2009 Kindless queries Sep 2009 XMPP Oct 2009 Incoming email Dec 2009 Blobstore Feb 2010 Datastore cursors, Appstats Mar 2010 Read policies, IPv6 May 2010 App Engine for Business 77 Tuesday, July 6, 2010
  87. An evolving platform 78 Tuesday, July 6, 2010
  88. App Engine Roadmap Improved monitoring/alerting Background servers SSL for your domain Control datastore availability vs. latency trade-offs Datastore dump and restore facility Mapping operations across datasets Raise request/response size limits for some APIs Reserved instances Built-in support for OAuth & OpenID Channel API 79 Tuesday, July 6, 2010
  89. Google Storage Tuesday, July 6, 2010
  90. Google Storage • Cloud-based binary object store o Structured as buckets and objects o Many buckets, many objects, large objects • You control your data o Private, shared, or public o Get your data back out at any time • For developers o RESTful API o Many SDKs + tools o Integration with other Google services Tuesday, July 6, 2010
  91. Google Storage Benefits High Performance and Scalability backed by Google infrastructure Flexible Authentication & Sharing Models Get Started Fast with Google & 3rd Party Utilities Tuesday, July 6, 2010
  92. Demo: Get Started In < 1 Minute • click emailed invitation link • read & accept Terms Of Service • start using GS Manager Tuesday, July 6, 2010
  93. Google services using Google Storage Partner Reporting Data Liberation Haiti Relief Imagery Partner Reporting Google Google BigQuery Prediction API Tuesday, July 6, 2010
  94. Some current users Tuesday, July 6, 2010
  95. Technical Overview Tuesday, July 6, 2010
  96. Google Storage Overview • Fast, scalable, highly available object store o Objects of any type and practically any size o Lots of objects, lots of buckets o All data replicated to multiple US data centers o Read-your-writes data consistency • Easy, flexible authentication and sharing o Key-based authentication o Authenticated downloads from a web browser o Sharing with individuals and groups • Google products and 3rd party tools/services o Compatible with many available tools and libraries o Getting started toolkit Tuesday, July 6, 2010
  97. API Concepts • RESTful API o Verbs: GET, PUT, POST, HEAD, DELETE o Resources: identified by URI • Buckets o Flat containers • Objects o Any type, practically any size • Access Control for Google Accounts o Google Groups • Two Ways to Authenticate Requests o Sign request using access keys o Web browser login Tuesday, July 6, 2010
  98. Sample Signed Request PUT /mybucket/My/Long/Object/Name HTTP/1.1 Host: commondatastorage.googleapis.com:443 Accept-Encoding: identity Date: Sat, 08 May 2010 19:04:21 GMT Content-Length: 28 Content-Type: text/plain Authorization:GOOG1 GOOG4622809698762217:J+y3mj5GThfI6Ed1MqLi7JpCq5Y=   This is my object's content. Tuesday, July 6, 2010
  99. Sharing and ACLs • Data can be private or shared • Bucket ACL determines: o who can list objects (READ) o who can create / delete objects (WRITE) o who can read / write bucket ACL (FULL_CONTROL) • Object ACL determines: o who can read objects (READ) o who can read / write object ACL (FULL_CONTROL) Tuesday, July 6, 2010
  100. Read-Your-Writes Consistency • Once a write succeeds, all future reads will see a snapshot that includes that write... no matter what replica they talk to • Once any reader sees a result (even if the write previously appeared to fail) then all future readers will see a snapshot that includes the write Tuesday, July 6, 2010
  101. Interoperability Gives You Choice • Data Liberation o You shouldn't be locked in by your choice of storage provider • Choice of tools o You should be able to use the same tools to manage your data, regardless of where you keep it Tuesday, July 6, 2010
  102. Demo: Using Google Storage Tuesday, July 6, 2010
  103. Computing in the Cloud: App Engine Sample os.environ['BOTO_CONFIG'] = 'boto.cfg' from boto import storage_uri # <other imports omitted> class MainPage(webapp.RequestHandler):   def get(self):     self.response.out.write('<html><body>')     uri = storage_uri('gs://pub/shakespeare/rose.txt')     poem = uri.get_contents_as_string()     self.response.out.write('<pre>' + poem + '</pre>')     self.response.out.write('</body></html>') def main():   application = webapp.WSGIApplication([('/', MainPage)])   run_wsgi_app(application) if __name__ == "__main__":   main() Tuesday, July 6, 2010
  104. Pricing and Availability • Pay as you go pricing • Storage - $0.17/GB/month • Network o Upload data to Google  $0.10/GB o Download data from Google  $0.15/GB for Americas and EMEA  $0.30/GB for APAC • Requests o PUT, POST, LIST - $0.01 per 1,000 Requests o GET, HEAD - $0.01 per 10,000 Requests • Free storage (up to 100GB) during preview period o No SLA • http://code.google.com/apis/storage Tuesday, July 6, 2010
  105. What's Coming Up • Service Level Agreement • Support • Available to Premium Apps Customers • Technical Features: o Group support in ACLs o Resumable uploads o Additional regions Tuesday, July 6, 2010
  106. Google BigQuery and Prediction APIs Tuesday, July 6, 2010
  107. Overview • Big Data - Challenging and Important • Google has tools for deep data analysis • Now you can use these tools Tuesday, July 6, 2010
  108. Overview • Big Data - Challenging and Important • Google has tools for deep data analysis • Now you can use these tools • Announcing two new APIs to get more from your data: 1.BigQuery 2.Prediction API Tuesday, July 6, 2010
  109. Benefits • Built on Google technology • Scalability • Security • Sharing • Easy integration with Google App Engine, Google Spreadsheets, .... Tuesday, July 6, 2010
  110. Using Your Data with BigQuery & Prediction API 1. Upload Upload your data Your Data to Google Storage Import to tables BigQuery 2. Process Train a model Prediction API Run queries Your Apps 3. Act Make predictions Tuesday, July 6, 2010
  111. BigQuery Interactive Analysis of Big Data Tuesday, July 6, 2010
  112. Big Data is Challenging Starts with Scale Tuesday, July 6, 2010
  113. Many Use Cases ... Interactive Tools Trends Spam Detection Web Network Dashboards Optimization Tuesday, July 6, 2010
  114. Demo: Analyzing M-Lab An open platform for advanced network research http://www.measurementlab.net/ Tuesday, July 6, 2010
  115. Demo: Exploring M-Lab Tuesday, July 6, 2010
  116. Key Capabilities of BigQuery • Scalable: Billions of rows • Fast: Response in seconds • Simple: Queries in SQL • Web Service o REST o JSON-RPC o Google App Scripts Tuesday, July 6, 2010
  117. Using Your Data with BigQuery Upload to Google Storage 1. Upload Import data into a BigQuery Table 2. Import - No need to define indices, keys, etc.. Execute queries via APIs 3. Query - No provisioning machines or resources Tuesday, July 6, 2010
  118. Writing Queries Compact subset of SQL o SELECT ... FROM ... WHERE ... GROUP BY ... ORDER BY ... LIMIT ...; Common functions o Math, String, Time, ... Statistical approximations o TOP o COUNT DISTINCT Tuesday, July 6, 2010
  119. API in a Minute GET /bigquery/v1/tables/{table name} GET /bigquery/v1/query?q={query} Sample JSON Reply: {   "results": {     "fields": { [       {"id":"COUNT(*)","type":"uint64"}, ... ]     },     "rows": [       {"f":[{"v":"2949"}, ...]},       {"f":[{"v":"5387"}, ...]}, ... ]   } } Also supports JSON-RPC Tuesday, July 6, 2010
  120. Security and Privacy Standard Google Authentication • Client Login • OAuth • AuthSub HTTPS support • protects your credentials • protects your data Use Google Storage for Developers to manage access Tuesday, July 6, 2010
  121. Large Corpus Analysis Wikimedia Revision History Wikimedia Revision history data from: http://download.wikimedia.org/enwiki/latest/ enwiki-latest-pages-meta-history.xml.7z Tuesday, July 6, 2010
  122. Using BigQuery Shell Python DB API 2.0 + B. Clapper's sqlcmd http://www.clapper.org/software/python/sqlcmd/ Tuesday, July 6, 2010
  123. BigQuery from a Spreadsheet Tuesday, July 6, 2010
  124. BigQuery from a Spreadsheet Tuesday, July 6, 2010
  125. BigQuery Recap • Interactive analysis of very large data sets • Simple SQL query language • APIs enable a variety of use cases Tuesday, July 6, 2010
  126. Google Prediction API Machine learning as a web service Tuesday, July 6, 2010
  127. Prediction API 101 • Google's sophisticated machine learning algorithms • Available as an on-demand RESTful HTTP web service • Train a model offline/asynchronously • Predict results in real-time "Tous pour un, un pour tous, Prediction API "french" c'est notre devise." Tuesday, July 6, 2010
  128. How does it work? "english" The quick brown fox jumped over the The Prediction API lazy dog. finds relevant "english" To err is human, but to really foul things features in the up you need a computer. sample data during "spanish" No hay mal que por bien no venga. training. "spanish" La tercera es la vencida. The Prediction API later searches for ? To be or not to be, that is the question. those features during prediction. ? La fe mueve montañas. Tuesday, July 6, 2010
  129. A virtually endless number of applications... Customer Transaction Species Message Diagnostics Sentiment Risk Identification Routing Churn Legal Docket Suspicious Work Roster Inappropriate Prediction Classification Activity Assignment Content Recommend Political Uplift Email Career Products Bias Marketing Filtering Counselling ... and many more ... Tuesday, July 6, 2010
  130. Three simple steps to use the Prediction API Upload your Use the API, gsutil or any 1. Upload training data to compatible utility to upload Google Storage your data to Google Storage Build a model prediction/v1/train/{} 2. Train from your data POST : a training request prediction/v1/query/{} Make new GET : model info 3. Predict predictions POST : a prediction request Tuesday, July 6, 2010
  131. Prediction API Demo Automatically categorize and respond to emails by language • Customer: ACME Corp, a multinational organization • Goal: Respond to customer emails in their language • Data: Many emails, tagged with their languages • Outcome: Predict language and respond accordingly Tuesday, July 6, 2010
  132. Step 1: Upload Upload your training data to Google Storage • Training data: outputs and input features • Data format: comma separated value format (CSV) $ head -n 2 ${data} "english","To err is human, but to really ..." "spanish","No hay mal que por bien no venga." Upload to Google Storage $ gsutil cp ${data} gs://io10/${data} Tuesday, July 6, 2010
  133. Step 2: Train Create a new model by training on data To train a model: POST prediction/v1/train/${data} Training runs asynchronously. To see if it has finished: GET prediction/v1/query/${data} {"data": {     "resource": {       "data": "${data}",        "modelinfo": "estimated accuracy: ${acc}"}}} Tuesday, July 6, 2010
  134. Step 3: Predict Apply the trained model to make predictions on new data POST prediction/v1/query/${data} { data : {    "instance" : {     "input" : { "text" : [       "J'aime X! C'est le meilleur" ]}}}} Tuesday, July 6, 2010
  135. Step 3: Predict Apply the trained model to make predictions on new data POST prediction/v1/query/${data} { data : {    "instance" : {     "input" : { "text" : [       "J'aime X! C'est le meilleur" ]}     "output" :     {"output_label" : "french"}}}} Tuesday, July 6, 2010
  136. Step 3: Predict Apply the trained model to make predictions on new data import httplib header = {"Content-Type" : "application/json"}#...put new data in JSON format in params variable conn = httplib.HTTPConnection("www.googleapis.com")conn.request("P OST",   "/prediction/v1/query/${data}",  params, header)print conn.getresponse() Tuesday, July 6, 2010
  137. Prediction API Capabilities Data • Input Features: numeric or unstructured text • Output: up to 100s of discrete categories Training • Many machine learning techniques • Automatically selected • Performed asynchronously Access from many platforms: • Web app from Google App Engine • Apps Script (e.g. from Google Spreadsheet) • Desktop app Tuesday, July 6, 2010
  138. Prediction API and BigQuery Demo: Tagger Input Data: http://delic.io.us/chanezon – 6000 urls, 14000 tags in 6 years Analyze my delicious tags – use delicious API to get all tagged urls – cleanup data, resize (100Mb limit) – PUT data in Google storage – Define table – analyze Predict how I would tag a technology article – input is tag,url,text – send new url and text – get predicted tag Tuesday, July 6, 2010
  139. Get the BigQuery & Prediction APIs • Preview, opened to a limited number of developers • You need a Google Storage for Developers account • To request access and get more information, go to: o http://code.google.com/apis/bigquery o http://code.google.com/apis/prediction Tuesday, July 6, 2010
  140. Acknowledgement Thanks to many Googlers from their slides, most available at the Google IO 2010 website 131 Tuesday, July 6, 2010
  141. Thank you Read more http://code.google.com/appengine/ Contact info Patrick Chanezon Developer Advocate chanezon@google.com http://twitter.com/chanezon Questions ? 132 Tuesday, July 6, 2010
  142. Tuesday, July 6, 2010

×