Google Cloud for Developers - Devfest Manila

Google Cloud for Developers Apps, Apps Marketplace, App Engine for Business, Storage, Prediction, Bigquery Patrick Chanezon Developer Advocate #devfest Manila chanezon@google.com http://twitter.com/chanezon July 6 2010 2 Tuesday, July 6, 2010

Tuesday, July 6, 2010

The benefits of Cloud Computing Economics Pay for only what you use TCO OPEX vs CAPEX Operations Day to day: no maintenance Fighting fires: no Pagers Elasticity Focus on your Business 3 Tuesday, July 6, 2010

Build and Buy all your enterprise cloud apps... Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010

Build and Buy all your enterprise cloud apps... Buy from Google Google Apps for Business Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010

Build and Buy all your enterprise cloud apps... Buy from others Buy from Google Google Apps Google Apps Marketplace for Business Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010

Build and Buy all your enterprise cloud apps... Buy from others Buy from Google Build your own Google Apps Google Apps Google App Engine Marketplace for Business for Business Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010

Customers want more Apps Business in the cloud Google Apps 5 Tuesday, July 6, 2010

Google Apps Marketplace Tuesday, July 6, 2010

Marketplace Overview Tuesday, July 6, 2010

Google Apps is Focused on Messaging & Collaboration Gives every employee powerful messaging and collaboration tools without the usual IT hassle and cost Tuesday, July 6, 2010

These customers want more great cloud apps... Messaging Document Management Collaboration Productivity Accounting & Finance Project Management Admin Tools Sales & Marketing Calendaring / Meetings Security & Compliance Customer Management Workflow Tuesday, July 6, 2010

Steps to sell your app to Google Apps customers 1. Build your app: - with any tools and hosting provider you want 2. Integrate your app: - add Single Sign On using OpenID (required) - access over a dozen integration points from Calendar, Contacts, Docs, etc. using OAuth (optional) 3. Sell your app: - to 2M+ businesses & 25M+ users - through Google Apps resellers - to wherever Google Apps goes - One-time fee of $100, 20% rev share starting 2H'10 Tuesday, July 6, 2010

Everything in the Cloud Tuesday, July 6, 2010

Integrating with Google Apps & the Marketplace Manifest Tuesday, July 6, 2010

Demo Google Apps Marketplace Tuesday, July 6, 2010

How it's done - Single Sign On Tuesday, July 6, 2010

Single Sign On OpenID with Google Apps bob@foo.com Welcome Bob! *********** Sounds complicated, but not hard in practice! Tuesday, July 6, 2010

Single Sign On OpenID Libraries Language Libraries Java OpenID4Java, Step2 .NET DotNetOpenAuth php-openid, PHP php-openid-apps-discovery ruby-openid, Ruby ruby-openid-apps-discovery Any RPX, Ping Identity Tuesday, July 6, 2010

Single Sign On - Code Step 1 of 3 - Initialize library function getOpenIDConsumer() { // Initialize client, storing associations // in memcache $store = new Auth_OpenID_MemcachedStore($MEMCACHE); $consumer = new Auth_OpenID_Consumer($store); // Enable Google Apps support GApps_OpenID_EnableDiscovery($consumer); return $consumer; } Tuesday, July 6, 2010

Single Sign On - Code Step 2 of 3 - Make the request // Create an auth request to the user's domain $auth_request = getOpenIDConsumer()->begin($domain); // Request email address & name during login $ax = new Auth_OpenID_AX_FetchRequest; $attr = Auth_OpenID_AX_AttrInfo::make( $AX_SCHEMA_EMAIL, 1, 1, 'email'); $attr = Auth_OpenID_AX_AttrInfo::make( $AX_SCHEMA_FIRSTNAME, 1, 1, 'ﬁrst'); $attr = Auth_OpenID_AX_AttrInfo::make( $AX_SCHEMA_LASTNAME, 1, 1, 'last'); $auth_request->addExtension($ax); // Render Javascript/form to post request $form_html = $auth_request->htmlMarkup($REALM, $RETURN_TO, false, array('id' => 'openid_message')); print $form_html; Tuesday, July 6, 2010

Single Sign On - Code Step 3 of 3 - Handle response // Parse the response from identity provider $response = getOpenIDConsumer()->complete($RETURN_TO); if ($response->status == Auth_OpenID_SUCCESS) { // Extract data from response $openid = $response->getDisplayIdentifier(); $ax = new Auth_OpenID_AX_FetchResponse(); $ax_resp = $ax->fromSuccessResponse($response); $email = $ax_resp->data[$AX_SCHEMA_EMAIL][0]; $firstName = $ax_resp->data[$AX_SCHEMA_FIRSTNAME][0]; $lastName = $ax_resp->data[$AX_SCHEMA_LASTTNAME][0]; // Map to user in DB $user = fetch_user($openid, $email, $firstName, $lastName); } Tuesday, July 6, 2010

Single Sign On - Code Manifest <ApplicationManifest> <Name>Sassy Voice</Name> <Description>Voice Mail & Messaging</Description> <Admin> <Link rel="setup"> http://voice.saasyapp.com/setup.php?domain=${DOMAIN_NAME} </Link> </Admin> <Extension id="navlink" type="link"> <Name>SaasyVoice</Name> <Url> http://voice.saasyapp.com/login.php?domain=${DOMAIN_NAME} </Url> </Extension> <Extension id="realm" type="openIdRealm"> <Url>http://voice.saasyapp.com/</Url> </Extension> </ApplicationManifest> Tuesday, July 6, 2010

Single Sign On What we learned about users Claimed ID: http://example.com/openid?id=12345 Email: bob@example.com* First Name: Bob Last Name: Dobbs * Be sure to confirm or white-list trusted providers Tuesday, July 6, 2010

How it's done - Data Access Tuesday, July 6, 2010

2-Legged OAuth Delegating data access <ApplicationManifest> ... <Scope id="userFeed"> <Url>https://apps-apis.google... <Reason>To get a list of user... </Scope> <Scope id="contactsFeed"> <Url>https://www.google.com... <Reason>To display names of... </Scope> <Scope id="docsFeed"> <Url>https://docs.google.com... <Reason>To export a call log... </Scope> </ApplicationManifest> Tuesday, July 6, 2010

Data Access - Code Step 1 of 2 - Initialize the client function getOauthClient() { $options = array( 'consumerKey' => $CONSUMER_KEY, 'consumerSecret' => $CONSUMER_SECRET, 'signatureMethod' => 'HMAC-SHA1', 'requestScheme' =>Zend_Oauth::REQUEST_SCHEME_HEADER, 'version' => '1.0'); $consumer = new Zend_Oauth_Consumer($options); $token = new Zend_Oauth_Token_Access(); $httpClient = $token->getHttpClient($options); return $httpClient; } Tuesday, July 6, 2010

Data Access - Code Step 2 of 2 - Fetching Users with Provisioning API // Initialize client $userClient = new Zend_Gdata_Gapps(getOauthClient()); // Query feed for current user's domain $userQuery = new Zend_Gdata_Gapps_UserQuery( getCurrentUserDomain()); $usersFeed = $userClient->getUserFeed($userQuery); // Extract data from user feed $users = array(); foreach ($usersFeed as $userEntry) { $login = $userEntry->getLogin(); $name = $userEntry->getName(); $users[] = array( 'username' => $login->getUsername(), 'ﬁrstName' => $name->getGivenName(), 'lastName' => $name->getFamilyName(), 'admin' => $login->getAdmin()); } Tuesday, July 6, 2010

Integration Recap With not a lot of code we added: • Single Sign On with OpenID • Quicker setup with Attribute Exchange, Provisioning API • Integrated user data with Contacts API • Uploading spreadsheets with Docs API Tuesday, July 6, 2010

Gadgets - your real estate in Google Apps Tuesday, July 6, 2010

Gadgets • Many types of gadgets o Gmail sidebar o Calendar sidebar o Sites o Spreadsheets • Two new types of gadgets available this week! o Gmail contextual! o Wave! • All use the OpenSocial gadgets specification Tuesday, July 6, 2010

Gmail contextual gadgets • Detect e-mail content via regular expressions • Display and collect actionable business information directly in Gmail, below each message Tuesday, July 6, 2010

Gmail contextual gadgets Tuesday, July 6, 2010

Calendar sidebar gadgets • Can detect: o currently-displayed date range o currently-selected event title attendees date/time Tuesday, July 6, 2010

Calendar sidebar gadgets Tuesday, July 6, 2010

Sell Tuesday, July 6, 2010

Complete Manifest <ApplicationManifest xmlns="http://schemas.google.com/ ApplicationManifest/2009"> <Name>SaaSy Voice</Name> <Description>SaaSy Voice</Description> <Support> <Link rel="setup" href="https://.../setup.php?domain=${DOMAIN_NAME}"/ > <Link rel="manage" href="https://.../conﬁg.php?domain=$ {DOMAIN_NAME}"/> <Link rel="deletion-policy" href="http://.../deletion-policy.php"/> <Link rel="support" href="http://.../support.php"/> </Support> Tuesday, July 6, 2010

Complete Manifest (continued)  <Extension id="oneBarLink" type="link"> <Name>SaaSy Voice</Name> <Url>http://.../index.php?domain=${DOMAIN_NAME}</Url> <Scope ref="provisioningFeed"/> <Scope ref="contactsFeed"/> <Scope ref="docsFeed"/> </Extension>  <Extension id="realm" type="openIdRealm"> <Url>http://voice.saasyapp.com/marketplace/</Url> </Extension> Tuesday, July 6, 2010

Complete Manifest (continued) <Scope id="provisioningFeed"> <Url>https://apps-apis.google.com/a/feeds/user/#readonly</Url> <Reason>To get a list of users to provision accounts</Reason> </Scope> <Scope id="contactsFeed"> <Url>https://www.google.com/m8/feeds/</Url> <Reason>To display names of people who called</Reason> </Scope> <Scope id="docsFeed"> <Url>https://docs.google.com/feeds/</Url> <Reason>To export a call log</Reason> </Scope> </ApplicationManifest> Tuesday, July 6, 2010

Listing in the Google Apps Marketplace Tuesday, July 6, 2010

Reach 25 million Google Apps Users "We've seen a very meaningful increase in high quality, non-paid lead flow directly attributable to Google Apps customers...Our customers cite Smartsheet's tight integration with Google's Data APIs as a key factor in their decision to purchase." Brent Frei, Smartsheet Tuesday, July 6, 2010

Roadmap Tuesday, July 6, 2010

Roadmap - General • Recent Launches o Gmail contextual gadgets o OAuth support for Gmail's IMAP and SMTP Tuesday, July 6, 2010

Roadmap - Billing • Goals o Simplified invoicing & payments for customers o Consistent experience across apps • Planned 2nd-half of 2010 o Required for installable apps after transition period o Revenue sharing starts once APIs adopted • Features o Unified billing for apps purchased in marketplace o Support for a variety of pricing & licensing models o APIs for customization and up-selling Tuesday, July 6, 2010

Learn More Tuesday, July 6, 2010

Resources • Business and Marketing o http://developer.googleapps.com/marketplace • Technical and Code o http://code.google.com/googleapps/marketplace • Shopping! o http://www.google.com/appsmarketplace • Don't have Google Apps? o http://www.google.com/a Tuesday, July 6, 2010

Build and Buy all your enterprise cloud apps... Buy from others Buy from Google Build your own Google Apps Google Apps Google App Engine Marketplace for Business for Business Google Apps Platform Enterprise Firewall 47 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010

Google App Engine Tuesday, July 6, 2010

Leveraging Google's Leadership in Cloud Computing • Massive data center operations • Purpose built hardware • Multi tenant software platform at Internet scale 49 Tuesday, July 6, 2010

Google App Engine - Easy to build - Easy to maintain - Easy to scale 50 Tuesday, July 6, 2010

Cloud development in a box • SDK & “The Cloud” • Hardware • Networking • Operating system • Application runtime Java, Python o • Static file serving • Services • Fault tolerance • Load balancing 51 Tuesday, July 6, 2010

App Engine Services Memcache Datastore URL Fetch Mail XMPP Task Queue Images Blobstore User Service 52 Tuesday, July 6, 2010

Always free to get started ~5M pageviews/month • 6.5 CPU hrs/day • 1 GB storage • 650K URL Fetch calls/day • 2,000 recipients emailed • 1 GB/day bandwidth • 100,000 tasks enqueued • 650K XMPP messages/day 53 Tuesday, July 6, 2010

Purchase additional resources * * free monthly quota of ~5 million page views still in full effect 54 Tuesday, July 6, 2010

By the numbers 250,000+ Developers 100,000+ Apps 0.5B+ daily Pageviews 55 Tuesday, July 6, 2010

App Engine 56 Tuesday, July 6, 2010

Socialwok 57 Tuesday, July 6, 2010

Social networking at scale 58 Tuesday, July 6, 2010

Social networking at scale >62M Users 3.6M DAUs on Facebook 1.9M DAUs on MySpace Orkut, Bebo, Hi5, Friendster, Hyves, Ning, … 58 Tuesday, July 6, 2010

Chillingo Crystal 59 Gaming meets Social Zombie Dash Angry Birds LITE Underground Meltdown Cogs Mission Deep Sea Speed Forge Guerilla Bob Ravensword: Angry Birds Extreme The Fallen King Tuesday, July 6, 2010

gigy Socialize 60 Tuesday, July 6, 2010

gigy Socialize - traffic 61 Tuesday, July 6, 2010

App Engine 62 Tuesday, July 6, 2010

Google App Engine for Business Tuesday, July 6, 2010

Build your Enterprise Apps on Google • Easy to Build - Java standards • Easy to Deploy - push-button deployment • Easy to Scale - from small apps to millions of users 64 Tuesday, July 6, 2010

Google App Engine for Business • Centralized administration - controls • Reliability and support - SLA, Premium support • Secure by default - only your users • Pricing that makes sense - pay only for what you use • Enterprise features - hosted SQL, SSL on your domain 65 Tuesday, July 6, 2010

Understanding the Cloud Computing Landscape SaaS PaaS IaaS 66 Source: Gartner AADI Summit Dec 2009 Tuesday, July 6, 2010

Google's Cloud Offerings 1. Our Apps 2. 3rd party Apps: Google Apps Marketplace SaaS 3. ________ PaaS Google App Engine Google Storage for Devs IaaS Machine Learning BigQuery 67 Tuesday, July 6, 2010

Google's Cloud Offerings Your Apps 1. Our Apps 2. 3rd party Apps: Google Apps Marketplace SaaS 3. ________ PaaS Google App Engine Google Storage for Devs IaaS Machine Learning BigQuery 67 Tuesday, July 6, 2010

Domain Console 68 Tuesday, July 6, 2010

Domain Console Like the regular admin console Designed to manage enterprises with a portfolio of apps • Keep track of all apps in a domain • Access Control: view apps, deploy • Global Settings: apply to all apps in the domain • Billing rolling up to single account • DNS configuration done only once: *.ext.example.com • All apps by default for logged in users from domain 69 Tuesday, July 6, 2010

Google Apps Integration • SSO/SSO delegation • APIs for most Google Apps for integration 70 Tuesday, July 6, 2010

Federate your on-premise data 71 Tuesday, July 6, 2010

Secure Data Connector (SDC) 72 Tuesday, July 6, 2010

Using Secure Data Connector 73 Tuesday, July 6, 2010

Using Secure Data Connector Installation - Determine access rules - Configure and install SDC 73 Tuesday, July 6, 2010

Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel 73 Tuesday, July 6, 2010

Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel Serving - User request sent to App Engine - User authenticated - App makes request through tunnel - SDC performs access checks - Results returned 73 Tuesday, July 6, 2010

App Engine for Business Pricing Intranet apps: Each app costs $8 / active user / month Capped at $1,000 / month (i.e. users above 125 are free) Apps are auth-restricted to domain users Development is free Overage charges on Background Analysis/Storage Non intranet apps (external/public/ISV apps): Pricing TBD Postpaid (i.e. billed at the end of month) 74 Tuesday, July 6, 2010

App Engine for Business Support and SLA Paid Support Email based 1000$/month 1h response time on operational issues 8h on development issues SLA 99.9% uptime Service credits from 10% to 100% refund of monthly bill 75 Tuesday, July 6, 2010

Google Developer Qualification Chrome Gadgets Search App Engine JS Maps API KML 3D Extensions 76 Tuesday, July 6, 2010

Two years in review Apr 2008 Python launch May 2008 Memcache, Images API Jul 2008 Logs export Aug 2008 Batch write/delete Oct 2008 HTTPS support Dec 2008 Status dashboard, quota details Feb 2009 Billing, larger files Apr 2009 Java launch, DB import, cron support, SDC May 2009 Key-only queries Jun 2009 Task queues Aug 2009 Kindless queries Sep 2009 XMPP Oct 2009 Incoming email Dec 2009 Blobstore Feb 2010 Datastore cursors, Appstats Mar 2010 Read policies, IPv6 May 2010 App Engine for Business 77 Tuesday, July 6, 2010

An evolving platform 78 Tuesday, July 6, 2010

App Engine Roadmap Improved monitoring/alerting Background servers SSL for your domain Control datastore availability vs. latency trade-offs Datastore dump and restore facility Mapping operations across datasets Raise request/response size limits for some APIs Reserved instances Built-in support for OAuth & OpenID Channel API 79 Tuesday, July 6, 2010

Google Storage Tuesday, July 6, 2010

Google Storage • Cloud-based binary object store o Structured as buckets and objects o Many buckets, many objects, large objects • You control your data o Private, shared, or public o Get your data back out at any time • For developers o RESTful API o Many SDKs + tools o Integration with other Google services Tuesday, July 6, 2010

Google Storage Benefits High Performance and Scalability backed by Google infrastructure Flexible Authentication & Sharing Models Get Started Fast with Google & 3rd Party Utilities Tuesday, July 6, 2010

Demo: Get Started In < 1 Minute • click emailed invitation link • read & accept Terms Of Service • start using GS Manager Tuesday, July 6, 2010

Google services using Google Storage Partner Reporting Data Liberation Haiti Relief Imagery Partner Reporting Google Google BigQuery Prediction API Tuesday, July 6, 2010

Some current users Tuesday, July 6, 2010

Technical Overview Tuesday, July 6, 2010

Google Storage Overview • Fast, scalable, highly available object store o Objects of any type and practically any size o Lots of objects, lots of buckets o All data replicated to multiple US data centers o Read-your-writes data consistency • Easy, flexible authentication and sharing o Key-based authentication o Authenticated downloads from a web browser o Sharing with individuals and groups • Google products and 3rd party tools/services o Compatible with many available tools and libraries o Getting started toolkit Tuesday, July 6, 2010

API Concepts • RESTful API o Verbs: GET, PUT, POST, HEAD, DELETE o Resources: identified by URI • Buckets o Flat containers • Objects o Any type, practically any size • Access Control for Google Accounts o Google Groups • Two Ways to Authenticate Requests o Sign request using access keys o Web browser login Tuesday, July 6, 2010

Sample Signed Request PUT /mybucket/My/Long/Object/Name HTTP/1.1 Host: commondatastorage.googleapis.com:443 Accept-Encoding: identity Date: Sat, 08 May 2010 19:04:21 GMT Content-Length: 28 Content-Type: text/plain Authorization:GOOG1 GOOG4622809698762217:J+y3mj5GThfI6Ed1MqLi7JpCq5Y= This is my object's content. Tuesday, July 6, 2010

Sharing and ACLs • Data can be private or shared • Bucket ACL determines: o who can list objects (READ) o who can create / delete objects (WRITE) o who can read / write bucket ACL (FULL_CONTROL) • Object ACL determines: o who can read objects (READ) o who can read / write object ACL (FULL_CONTROL) Tuesday, July 6, 2010

Read-Your-Writes Consistency • Once a write succeeds, all future reads will see a snapshot that includes that write... no matter what replica they talk to • Once any reader sees a result (even if the write previously appeared to fail) then all future readers will see a snapshot that includes the write Tuesday, July 6, 2010

Interoperability Gives You Choice • Data Liberation o You shouldn't be locked in by your choice of storage provider • Choice of tools o You should be able to use the same tools to manage your data, regardless of where you keep it Tuesday, July 6, 2010

Demo: Using Google Storage Tuesday, July 6, 2010

Computing in the Cloud: App Engine Sample os.environ['BOTO_CONFIG'] = 'boto.cfg' from boto import storage_uri # <other imports omitted> class MainPage(webapp.RequestHandler): def get(self): self.response.out.write('<html><body>') uri = storage_uri('gs://pub/shakespeare/rose.txt') poem = uri.get_contents_as_string() self.response.out.write('<pre>' + poem + '</pre>') self.response.out.write('</body></html>') def main(): application = webapp.WSGIApplication([('/', MainPage)]) run_wsgi_app(application) if __name__ == "__main__": main() Tuesday, July 6, 2010

Pricing and Availability • Pay as you go pricing • Storage - $0.17/GB/month • Network o Upload data to Google  $0.10/GB o Download data from Google  $0.15/GB for Americas and EMEA  $0.30/GB for APAC • Requests o PUT, POST, LIST - $0.01 per 1,000 Requests o GET, HEAD - $0.01 per 10,000 Requests • Free storage (up to 100GB) during preview period o No SLA • http://code.google.com/apis/storage Tuesday, July 6, 2010

What's Coming Up • Service Level Agreement • Support • Available to Premium Apps Customers • Technical Features: o Group support in ACLs o Resumable uploads o Additional regions Tuesday, July 6, 2010

Google BigQuery and Prediction APIs Tuesday, July 6, 2010

Overview • Big Data - Challenging and Important • Google has tools for deep data analysis • Now you can use these tools Tuesday, July 6, 2010

Overview • Big Data - Challenging and Important • Google has tools for deep data analysis • Now you can use these tools • Announcing two new APIs to get more from your data: 1.BigQuery 2.Prediction API Tuesday, July 6, 2010

Benefits • Built on Google technology • Scalability • Security • Sharing • Easy integration with Google App Engine, Google Spreadsheets, .... Tuesday, July 6, 2010

Using Your Data with BigQuery & Prediction API 1. Upload Upload your data Your Data to Google Storage Import to tables BigQuery 2. Process Train a model Prediction API Run queries Your Apps 3. Act Make predictions Tuesday, July 6, 2010

BigQuery Interactive Analysis of Big Data Tuesday, July 6, 2010

Big Data is Challenging Starts with Scale Tuesday, July 6, 2010

Many Use Cases ... Interactive Tools Trends Spam Detection Web Network Dashboards Optimization Tuesday, July 6, 2010

Demo: Analyzing M-Lab An open platform for advanced network research http://www.measurementlab.net/ Tuesday, July 6, 2010

Demo: Exploring M-Lab Tuesday, July 6, 2010

Key Capabilities of BigQuery • Scalable: Billions of rows • Fast: Response in seconds • Simple: Queries in SQL • Web Service o REST o JSON-RPC o Google App Scripts Tuesday, July 6, 2010

Using Your Data with BigQuery Upload to Google Storage 1. Upload Import data into a BigQuery Table 2. Import - No need to define indices, keys, etc.. Execute queries via APIs 3. Query - No provisioning machines or resources Tuesday, July 6, 2010

Writing Queries Compact subset of SQL o SELECT ... FROM ... WHERE ... GROUP BY ... ORDER BY ... LIMIT ...; Common functions o Math, String, Time, ... Statistical approximations o TOP o COUNT DISTINCT Tuesday, July 6, 2010

API in a Minute GET /bigquery/v1/tables/{table name} GET /bigquery/v1/query?q={query} Sample JSON Reply: { "results": { "ﬁelds": { [ {"id":"COUNT(*)","type":"uint64"}, ... ] }, "rows": [ {"f":[{"v":"2949"}, ...]}, {"f":[{"v":"5387"}, ...]}, ... ] } } Also supports JSON-RPC Tuesday, July 6, 2010

Security and Privacy Standard Google Authentication • Client Login • OAuth • AuthSub HTTPS support • protects your credentials • protects your data Use Google Storage for Developers to manage access Tuesday, July 6, 2010

Large Corpus Analysis Wikimedia Revision History Wikimedia Revision history data from: http://download.wikimedia.org/enwiki/latest/ enwiki-latest-pages-meta-history.xml.7z Tuesday, July 6, 2010

Using BigQuery Shell Python DB API 2.0 + B. Clapper's sqlcmd http://www.clapper.org/software/python/sqlcmd/ Tuesday, July 6, 2010

BigQuery from a Spreadsheet Tuesday, July 6, 2010

BigQuery Recap • Interactive analysis of very large data sets • Simple SQL query language • APIs enable a variety of use cases Tuesday, July 6, 2010

Google Prediction API Machine learning as a web service Tuesday, July 6, 2010

Prediction API 101 • Google's sophisticated machine learning algorithms • Available as an on-demand RESTful HTTP web service • Train a model offline/asynchronously • Predict results in real-time "Tous pour un, un pour tous, Prediction API "french" c'est notre devise." Tuesday, July 6, 2010

How does it work? "english" The quick brown fox jumped over the The Prediction API lazy dog. finds relevant "english" To err is human, but to really foul things features in the up you need a computer. sample data during "spanish" No hay mal que por bien no venga. training. "spanish" La tercera es la vencida. The Prediction API later searches for ? To be or not to be, that is the question. those features during prediction. ? La fe mueve montañas. Tuesday, July 6, 2010

A virtually endless number of applications... Customer Transaction Species Message Diagnostics Sentiment Risk Identification Routing Churn Legal Docket Suspicious Work Roster Inappropriate Prediction Classification Activity Assignment Content Recommend Political Uplift Email Career Products Bias Marketing Filtering Counselling ... and many more ... Tuesday, July 6, 2010

Three simple steps to use the Prediction API Upload your Use the API, gsutil or any 1. Upload training data to compatible utility to upload Google Storage your data to Google Storage Build a model prediction/v1/train/{} 2. Train from your data POST : a training request prediction/v1/query/{} Make new GET : model info 3. Predict predictions POST : a prediction request Tuesday, July 6, 2010

Prediction API Demo Automatically categorize and respond to emails by language • Customer: ACME Corp, a multinational organization • Goal: Respond to customer emails in their language • Data: Many emails, tagged with their languages • Outcome: Predict language and respond accordingly Tuesday, July 6, 2010

Step 1: Upload Upload your training data to Google Storage • Training data: outputs and input features • Data format: comma separated value format (CSV) $ head -n 2 ${data} "english","To err is human, but to really ..." "spanish","No hay mal que por bien no venga." Upload to Google Storage $ gsutil cp ${data} gs://io10/${data} Tuesday, July 6, 2010

Step 2: Train Create a new model by training on data To train a model: POST prediction/v1/train/${data} Training runs asynchronously. To see if it has finished: GET prediction/v1/query/${data} {"data": { "resource": { "data": "${data}", "modelinfo": "estimated accuracy: ${acc}"}}} Tuesday, July 6, 2010

Step 3: Predict Apply the trained model to make predictions on new data POST prediction/v1/query/${data} { data : { "instance" : { "input" : { "text" : [ "J'aime X! C'est le meilleur" ]}}}} Tuesday, July 6, 2010

Step 3: Predict Apply the trained model to make predictions on new data POST prediction/v1/query/${data} { data : { "instance" : { "input" : { "text" : [ "J'aime X! C'est le meilleur" ]} "output" : {"output_label" : "french"}}}} Tuesday, July 6, 2010

Step 3: Predict Apply the trained model to make predictions on new data import httplib header = {"Content-Type" : "application/json"}#...put new data in JSON format in params variable conn = httplib.HTTPConnection("www.googleapis.com")conn.request("P OST", "/prediction/v1/query/${data}", params, header)print conn.getresponse() Tuesday, July 6, 2010

Prediction API Capabilities Data • Input Features: numeric or unstructured text • Output: up to 100s of discrete categories Training • Many machine learning techniques • Automatically selected • Performed asynchronously Access from many platforms: • Web app from Google App Engine • Apps Script (e.g. from Google Spreadsheet) • Desktop app Tuesday, July 6, 2010

Prediction API and BigQuery Demo: Tagger Input Data: http://delic.io.us/chanezon – 6000 urls, 14000 tags in 6 years Analyze my delicious tags – use delicious API to get all tagged urls – cleanup data, resize (100Mb limit) – PUT data in Google storage – Define table – analyze Predict how I would tag a technology article – input is tag,url,text – send new url and text – get predicted tag Tuesday, July 6, 2010

Get the BigQuery & Prediction APIs • Preview, opened to a limited number of developers • You need a Google Storage for Developers account • To request access and get more information, go to: o http://code.google.com/apis/bigquery o http://code.google.com/apis/prediction Tuesday, July 6, 2010

Acknowledgement Thanks to many Googlers from their slides, most available at the Google IO 2010 website 131 Tuesday, July 6, 2010

Thank you Read more http://code.google.com/appengine/ Contact info Patrick Chanezon Developer Advocate chanezon@google.com http://twitter.com/chanezon Questions ? 132 Tuesday, July 6, 2010

Tuesday, July 6, 2010

http://webtechman.com	3
http://www.slideshare.net	1
http://mpelembe.blogware.com	1
http://blog.lightyoruichi.com	1
http://www.mefeedia.com	1

Google Cloud for Developers - Devfest Manila

by Patrick Chanezon on Jul 06, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 7

Statistics

Google Cloud for Developers - Devfest Manila — Presentation Transcript