Google cloud Platform
Your SlideShare is downloading.
×
![Computing in the Cloud: App Engine Sample
os.environ['BOTO_CONFIG'] = 'boto.cfg'
from boto import storage_...](https://image.slidesharecdn.com/googlecloudfordevelopers-devfestmanila-100706105846-phpapp01/75/google-cloud-for-developers-devfest-manila-103-2048.jpg?cb=1665176381)
Check these out next
Recommended
More Related Content
Slideshows for you (20)
Viewers also liked (20)
Similar to Google Cloud for Developers - Devfest Manila (20)
More from Patrick Chanezon (20)
Recently uploaded (20)
Google Cloud for Developers - Devfest Manila
- Google Cloud for Developers Apps, Apps Marketplace, App Engine for Business, Storage, Prediction, Bigquery Patrick Chanezon Developer Advocate #devfest Manila chanezon@google.com http://twitter.com/chanezon July 6 2010 2 Tuesday, July 6, 2010
- Tuesday, July 6, 2010
- The benefits of Cloud Computing Economics Pay for only what you use TCO OPEX vs CAPEX Operations Day to day: no maintenance Fighting fires: no Pagers Elasticity Focus on your Business 3 Tuesday, July 6, 2010
- Build and Buy all your enterprise cloud apps... Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010
- Build and Buy all your enterprise cloud apps... Buy from Google Google Apps for Business Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010
- Build and Buy all your enterprise cloud apps... Buy from others Buy from Google Google Apps Google Apps Marketplace for Business Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010
- Build and Buy all your enterprise cloud apps... Buy from others Buy from Google Build your own Google Apps Google Apps Google App Engine Marketplace for Business for Business Google Apps Platform Enterprise Firewall 4 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010
- Customers want more Apps Business in the cloud Google Apps 5 Tuesday, July 6, 2010
- Google Apps Marketplace Tuesday, July 6, 2010
- Marketplace Overview Tuesday, July 6, 2010
- Google Apps is Focused on Messaging & Collaboration Gives every employee powerful messaging and collaboration tools without the usual IT hassle and cost Tuesday, July 6, 2010
- These customers want more great cloud apps... Messaging Document Management Collaboration Productivity Accounting & Finance Project Management Admin Tools Sales & Marketing Calendaring / Meetings Security & Compliance Customer Management Workflow Tuesday, July 6, 2010
- Steps to sell your app to Google Apps customers 1. Build your app: - with any tools and hosting provider you want 2. Integrate your app: - add Single Sign On using OpenID (required) - access over a dozen integration points from Calendar, Contacts, Docs, etc. using OAuth (optional) 3. Sell your app: - to 2M+ businesses & 25M+ users - through Google Apps resellers - to wherever Google Apps goes - One-time fee of $100, 20% rev share starting 2H'10 Tuesday, July 6, 2010
- Everything in the Cloud Tuesday, July 6, 2010
- Integrating with Google Apps & the Marketplace Manifest Tuesday, July 6, 2010
- Demo Google Apps Marketplace Tuesday, July 6, 2010
- How it's done - Single Sign On Tuesday, July 6, 2010
- Single Sign On OpenID with Google Apps bob@foo.com Welcome Bob! *********** Sounds complicated, but not hard in practice! Tuesday, July 6, 2010
- Single Sign On OpenID Libraries Language Libraries Java OpenID4Java, Step2 .NET DotNetOpenAuth php-openid, PHP php-openid-apps-discovery ruby-openid, Ruby ruby-openid-apps-discovery Any RPX, Ping Identity Tuesday, July 6, 2010
- Single Sign On - Code Step 1 of 3 - Initialize library function getOpenIDConsumer() { // Initialize client, storing associations // in memcache $store = new Auth_OpenID_MemcachedStore($MEMCACHE); $consumer = new Auth_OpenID_Consumer($store); // Enable Google Apps support GApps_OpenID_EnableDiscovery($consumer); return $consumer; } Tuesday, July 6, 2010
- Single Sign On - Code Step 2 of 3 - Make the request // Create an auth request to the user's domain $auth_request = getOpenIDConsumer()->begin($domain); // Request email address & name during login $ax = new Auth_OpenID_AX_FetchRequest; $attr = Auth_OpenID_AX_AttrInfo::make( $AX_SCHEMA_EMAIL, 1, 1, 'email'); $attr = Auth_OpenID_AX_AttrInfo::make( $AX_SCHEMA_FIRSTNAME, 1, 1, 'first'); $attr = Auth_OpenID_AX_AttrInfo::make( $AX_SCHEMA_LASTNAME, 1, 1, 'last'); $auth_request->addExtension($ax); // Render Javascript/form to post request $form_html = $auth_request->htmlMarkup($REALM, $RETURN_TO, false, array('id' => 'openid_message')); print $form_html; Tuesday, July 6, 2010
- Single Sign On - Code Step 3 of 3 - Handle response // Parse the response from identity provider $response = getOpenIDConsumer()->complete($RETURN_TO); if ($response->status == Auth_OpenID_SUCCESS) { // Extract data from response $openid = $response->getDisplayIdentifier(); $ax = new Auth_OpenID_AX_FetchResponse(); $ax_resp = $ax->fromSuccessResponse($response); $email = $ax_resp->data[$AX_SCHEMA_EMAIL][0]; $firstName = $ax_resp->data[$AX_SCHEMA_FIRSTNAME][0]; $lastName = $ax_resp->data[$AX_SCHEMA_LASTTNAME][0]; // Map to user in DB $user = fetch_user($openid, $email, $firstName, $lastName); } Tuesday, July 6, 2010
- Single Sign On - Code Manifest <ApplicationManifest> <Name>Sassy Voice</Name> <Description>Voice Mail & Messaging</Description> <Admin> <Link rel="setup"> http://voice.saasyapp.com/setup.php?domain=${DOMAIN_NAME} </Link> </Admin> <Extension id="navlink" type="link"> <Name>SaasyVoice</Name> <Url> http://voice.saasyapp.com/login.php?domain=${DOMAIN_NAME} </Url> </Extension> <Extension id="realm" type="openIdRealm"> <Url>http://voice.saasyapp.com/</Url> </Extension> </ApplicationManifest> Tuesday, July 6, 2010
- Single Sign On What we learned about users Claimed ID: http://example.com/openid?id=12345 Email: bob@example.com* First Name: Bob Last Name: Dobbs * Be sure to confirm or white-list trusted providers Tuesday, July 6, 2010
- How it's done - Data Access Tuesday, July 6, 2010
- 2-Legged OAuth Delegating data access <ApplicationManifest> ... <Scope id="userFeed"> <Url>https://apps-apis.google... <Reason>To get a list of user... </Scope> <Scope id="contactsFeed"> <Url>https://www.google.com... <Reason>To display names of... </Scope> <Scope id="docsFeed"> <Url>https://docs.google.com... <Reason>To export a call log... </Scope> </ApplicationManifest> Tuesday, July 6, 2010
- Data Access - Code Step 1 of 2 - Initialize the client function getOauthClient() { $options = array( 'consumerKey' => $CONSUMER_KEY, 'consumerSecret' => $CONSUMER_SECRET, 'signatureMethod' => 'HMAC-SHA1', 'requestScheme' =>Zend_Oauth::REQUEST_SCHEME_HEADER, 'version' => '1.0'); $consumer = new Zend_Oauth_Consumer($options); $token = new Zend_Oauth_Token_Access(); $httpClient = $token->getHttpClient($options); return $httpClient; } Tuesday, July 6, 2010
- Data Access - Code Step 2 of 2 - Fetching Users with Provisioning API // Initialize client $userClient = new Zend_Gdata_Gapps(getOauthClient()); // Query feed for current user's domain $userQuery = new Zend_Gdata_Gapps_UserQuery( getCurrentUserDomain()); $usersFeed = $userClient->getUserFeed($userQuery); // Extract data from user feed $users = array(); foreach ($usersFeed as $userEntry) { $login = $userEntry->getLogin(); $name = $userEntry->getName(); $users[] = array( 'username' => $login->getUsername(), 'firstName' => $name->getGivenName(), 'lastName' => $name->getFamilyName(), 'admin' => $login->getAdmin()); } Tuesday, July 6, 2010
- Integration Recap With not a lot of code we added: • Single Sign On with OpenID • Quicker setup with Attribute Exchange, Provisioning API • Integrated user data with Contacts API • Uploading spreadsheets with Docs API Tuesday, July 6, 2010
- Gadgets - your real estate in Google Apps Tuesday, July 6, 2010
- Gadgets • Many types of gadgets o Gmail sidebar o Calendar sidebar o Sites o Spreadsheets • Two new types of gadgets available this week! o Gmail contextual! o Wave! • All use the OpenSocial gadgets specification Tuesday, July 6, 2010
- Gmail contextual gadgets • Detect e-mail content via regular expressions • Display and collect actionable business information directly in Gmail, below each message Tuesday, July 6, 2010
- Gmail contextual gadgets Tuesday, July 6, 2010
- Gmail contextual gadgets Tuesday, July 6, 2010
- Calendar sidebar gadgets • Can detect: o currently-displayed date range o currently-selected event title attendees date/time Tuesday, July 6, 2010
- Calendar sidebar gadgets Tuesday, July 6, 2010
- Calendar sidebar gadgets Tuesday, July 6, 2010
- Sell Tuesday, July 6, 2010
- Complete Manifest <ApplicationManifest xmlns="http://schemas.google.com/ ApplicationManifest/2009"> <Name>SaaSy Voice</Name> <Description>SaaSy Voice</Description> <Support> <Link rel="setup" href="https://.../setup.php?domain=${DOMAIN_NAME}"/ > <Link rel="manage" href="https://.../config.php?domain=$ {DOMAIN_NAME}"/> <Link rel="deletion-policy" href="http://.../deletion-policy.php"/> <Link rel="support" href="http://.../support.php"/> </Support> Tuesday, July 6, 2010
- Complete Manifest (continued) <!-- Link in universal navigation --> <Extension id="oneBarLink" type="link"> <Name>SaaSy Voice</Name> <Url>http://.../index.php?domain=${DOMAIN_NAME}</Url> <Scope ref="provisioningFeed"/> <Scope ref="contactsFeed"/> <Scope ref="docsFeed"/> </Extension> <!-- Declare our openid.realm --> <Extension id="realm" type="openIdRealm"> <Url>http://voice.saasyapp.com/marketplace/</Url> </Extension> Tuesday, July 6, 2010
- Complete Manifest (continued) <Scope id="provisioningFeed"> <Url>https://apps-apis.google.com/a/feeds/user/#readonly</Url> <Reason>To get a list of users to provision accounts</Reason> </Scope> <Scope id="contactsFeed"> <Url>https://www.google.com/m8/feeds/</Url> <Reason>To display names of people who called</Reason> </Scope> <Scope id="docsFeed"> <Url>https://docs.google.com/feeds/</Url> <Reason>To export a call log</Reason> </Scope> </ApplicationManifest> Tuesday, July 6, 2010
- Listing in the Google Apps Marketplace Tuesday, July 6, 2010
- Listing in the Google Apps Marketplace Tuesday, July 6, 2010
- Reach 25 million Google Apps Users "We've seen a very meaningful increase in high quality, non-paid lead flow directly attributable to Google Apps customers...Our customers cite Smartsheet's tight integration with Google's Data APIs as a key factor in their decision to purchase." Brent Frei, Smartsheet Tuesday, July 6, 2010
- Roadmap Tuesday, July 6, 2010
- Roadmap - General • Recent Launches o Gmail contextual gadgets o OAuth support for Gmail's IMAP and SMTP Tuesday, July 6, 2010
- Roadmap - Billing • Goals o Simplified invoicing & payments for customers o Consistent experience across apps • Planned 2nd-half of 2010 o Required for installable apps after transition period o Revenue sharing starts once APIs adopted • Features o Unified billing for apps purchased in marketplace o Support for a variety of pricing & licensing models o APIs for customization and up-selling Tuesday, July 6, 2010
- Learn More Tuesday, July 6, 2010
- Resources • Business and Marketing o http://developer.googleapps.com/marketplace • Technical and Code o http://code.google.com/googleapps/marketplace • Shopping! o http://www.google.com/appsmarketplace • Don't have Google Apps? o http://www.google.com/a Tuesday, July 6, 2010
- Build and Buy all your enterprise cloud apps... Buy from others Buy from Google Build your own Google Apps Google Apps Google App Engine Marketplace for Business for Business Google Apps Platform Enterprise Firewall 47 Enterprise Data Authentication Enterprise Services User Management Tuesday, July 6, 2010
- Google App Engine Tuesday, July 6, 2010
- Leveraging Google's Leadership in Cloud Computing • Massive data center operations • Purpose built hardware • Multi tenant software platform at Internet scale 49 Tuesday, July 6, 2010
- Google App Engine - Easy to build - Easy to maintain - Easy to scale 50 Tuesday, July 6, 2010
- Cloud development in a box • SDK & “The Cloud” • Hardware • Networking • Operating system • Application runtime Java, Python o • Static file serving • Services • Fault tolerance • Load balancing 51 Tuesday, July 6, 2010
- App Engine Services Memcache Datastore URL Fetch Mail XMPP Task Queue Images Blobstore User Service 52 Tuesday, July 6, 2010
- Always free to get started ~5M pageviews/month • 6.5 CPU hrs/day • 1 GB storage • 650K URL Fetch calls/day • 2,000 recipients emailed • 1 GB/day bandwidth • 100,000 tasks enqueued • 650K XMPP messages/day 53 Tuesday, July 6, 2010
- Purchase additional resources * * free monthly quota of ~5 million page views still in full effect 54 Tuesday, July 6, 2010
- By the numbers 250,000+ Developers 100,000+ Apps 0.5B+ daily Pageviews 55 Tuesday, July 6, 2010
- App Engine 56 Tuesday, July 6, 2010
- Socialwok 57 Tuesday, July 6, 2010
- Social networking at scale 58 Tuesday, July 6, 2010
- Social networking at scale >62M Users 3.6M DAUs on Facebook 1.9M DAUs on MySpace Orkut, Bebo, Hi5, Friendster, Hyves, Ning, … 58 Tuesday, July 6, 2010
- Chillingo Crystal 59 Gaming meets Social Zombie Dash Angry Birds LITE Underground Meltdown Cogs Mission Deep Sea Speed Forge Guerilla Bob Ravensword: Angry Birds Extreme The Fallen King Tuesday, July 6, 2010
- gigy Socialize 60 Tuesday, July 6, 2010
- gigy Socialize - traffic 61 Tuesday, July 6, 2010
- gigy Socialize - traffic 61 Tuesday, July 6, 2010
- App Engine 62 Tuesday, July 6, 2010
- Google App Engine for Business Tuesday, July 6, 2010
- Build your Enterprise Apps on Google • Easy to Build - Java standards • Easy to Deploy - push-button deployment • Easy to Scale - from small apps to millions of users 64 Tuesday, July 6, 2010
- Google App Engine for Business • Centralized administration - controls • Reliability and support - SLA, Premium support • Secure by default - only your users • Pricing that makes sense - pay only for what you use • Enterprise features - hosted SQL, SSL on your domain 65 Tuesday, July 6, 2010
- Understanding the Cloud Computing Landscape SaaS PaaS IaaS 66 Source: Gartner AADI Summit Dec 2009 Tuesday, July 6, 2010
- Google's Cloud Offerings 1. Our Apps 2. 3rd party Apps: Google Apps Marketplace SaaS 3. ________ PaaS Google App Engine Google Storage for Devs IaaS Machine Learning BigQuery 67 Tuesday, July 6, 2010
- Google's Cloud Offerings Your Apps 1. Our Apps 2. 3rd party Apps: Google Apps Marketplace SaaS 3. ________ PaaS Google App Engine Google Storage for Devs IaaS Machine Learning BigQuery 67 Tuesday, July 6, 2010
- Domain Console 68 Tuesday, July 6, 2010
- Domain Console Like the regular admin console Designed to manage enterprises with a portfolio of apps • Keep track of all apps in a domain • Access Control: view apps, deploy • Global Settings: apply to all apps in the domain • Billing rolling up to single account • DNS configuration done only once: *.ext.example.com • All apps by default for logged in users from domain 69 Tuesday, July 6, 2010
- Google Apps Integration • SSO/SSO delegation • APIs for most Google Apps for integration 70 Tuesday, July 6, 2010
- Federate your on-premise data 71 Tuesday, July 6, 2010
- Secure Data Connector (SDC) 72 Tuesday, July 6, 2010
- Using Secure Data Connector 73 Tuesday, July 6, 2010
- Using Secure Data Connector Installation - Determine access rules - Configure and install SDC 73 Tuesday, July 6, 2010
- Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel 73 Tuesday, July 6, 2010
- Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel Serving - User request sent to App Engine - User authenticated - App makes request through tunnel - SDC performs access checks - Results returned 73 Tuesday, July 6, 2010
- App Engine for Business Pricing Intranet apps: Each app costs $8 / active user / month Capped at $1,000 / month (i.e. users above 125 are free) Apps are auth-restricted to domain users Development is free Overage charges on Background Analysis/Storage Non intranet apps (external/public/ISV apps): Pricing TBD Postpaid (i.e. billed at the end of month) 74 Tuesday, July 6, 2010
- App Engine for Business Support and SLA Paid Support Email based 1000$/month 1h response time on operational issues 8h on development issues SLA 99.9% uptime Service credits from 10% to 100% refund of monthly bill 75 Tuesday, July 6, 2010
- Google Developer Qualification Chrome Gadgets Search App Engine JS Maps API KML 3D Extensions 76 Tuesday, July 6, 2010
- Two years in review Apr 2008 Python launch May 2008 Memcache, Images API Jul 2008 Logs export Aug 2008 Batch write/delete Oct 2008 HTTPS support Dec 2008 Status dashboard, quota details Feb 2009 Billing, larger files Apr 2009 Java launch, DB import, cron support, SDC May 2009 Key-only queries Jun 2009 Task queues Aug 2009 Kindless queries Sep 2009 XMPP Oct 2009 Incoming email Dec 2009 Blobstore Feb 2010 Datastore cursors, Appstats Mar 2010 Read policies, IPv6 May 2010 App Engine for Business 77 Tuesday, July 6, 2010
- An evolving platform 78 Tuesday, July 6, 2010
- App Engine Roadmap Improved monitoring/alerting Background servers SSL for your domain Control datastore availability vs. latency trade-offs Datastore dump and restore facility Mapping operations across datasets Raise request/response size limits for some APIs Reserved instances Built-in support for OAuth & OpenID Channel API 79 Tuesday, July 6, 2010
- Google Storage Tuesday, July 6, 2010
- Google Storage • Cloud-based binary object store o Structured as buckets and objects o Many buckets, many objects, large objects • You control your data o Private, shared, or public o Get your data back out at any time • For developers o RESTful API o Many SDKs + tools o Integration with other Google services Tuesday, July 6, 2010
- Google Storage Benefits High Performance and Scalability backed by Google infrastructure Flexible Authentication & Sharing Models Get Started Fast with Google & 3rd Party Utilities Tuesday, July 6, 2010
- Demo: Get Started In < 1 Minute • click emailed invitation link • read & accept Terms Of Service • start using GS Manager Tuesday, July 6, 2010
- Google services using Google Storage Partner Reporting Data Liberation Haiti Relief Imagery Partner Reporting Google Google BigQuery Prediction API Tuesday, July 6, 2010
- Some current users Tuesday, July 6, 2010
- Technical Overview Tuesday, July 6, 2010
- Google Storage Overview • Fast, scalable, highly available object store o Objects of any type and practically any size o Lots of objects, lots of buckets o All data replicated to multiple US data centers o Read-your-writes data consistency • Easy, flexible authentication and sharing o Key-based authentication o Authenticated downloads from a web browser o Sharing with individuals and groups • Google products and 3rd party tools/services o Compatible with many available tools and libraries o Getting started toolkit Tuesday, July 6, 2010
- API Concepts • RESTful API o Verbs: GET, PUT, POST, HEAD, DELETE o Resources: identified by URI • Buckets o Flat containers • Objects o Any type, practically any size • Access Control for Google Accounts o Google Groups • Two Ways to Authenticate Requests o Sign request using access keys o Web browser login Tuesday, July 6, 2010
- Sample Signed Request PUT /mybucket/My/Long/Object/Name HTTP/1.1 Host: commondatastorage.googleapis.com:443 Accept-Encoding: identity Date: Sat, 08 May 2010 19:04:21 GMT Content-Length: 28 Content-Type: text/plain Authorization:GOOG1 GOOG4622809698762217:J+y3mj5GThfI6Ed1MqLi7JpCq5Y= This is my object's content. Tuesday, July 6, 2010
- Sharing and ACLs • Data can be private or shared • Bucket ACL determines: o who can list objects (READ) o who can create / delete objects (WRITE) o who can read / write bucket ACL (FULL_CONTROL) • Object ACL determines: o who can read objects (READ) o who can read / write object ACL (FULL_CONTROL) Tuesday, July 6, 2010
- Read-Your-Writes Consistency • Once a write succeeds, all future reads will see a snapshot that includes that write... no matter what replica they talk to • Once any reader sees a result (even if the write previously appeared to fail) then all future readers will see a snapshot that includes the write Tuesday, July 6, 2010
- Interoperability Gives You Choice • Data Liberation o You shouldn't be locked in by your choice of storage provider • Choice of tools o You should be able to use the same tools to manage your data, regardless of where you keep it Tuesday, July 6, 2010
- Demo: Using Google Storage Tuesday, July 6, 2010
- Computing in the Cloud: App Engine Sample os.environ['BOTO_CONFIG'] = 'boto.cfg' from boto import storage_uri # <other imports omitted> class MainPage(webapp.RequestHandler): def get(self): self.response.out.write('<html><body>') uri = storage_uri('gs://pub/shakespeare/rose.txt') poem = uri.get_contents_as_string() self.response.out.write('<pre>' + poem + '</pre>') self.response.out.write('</body></html>') def main(): application = webapp.WSGIApplication([('/', MainPage)]) run_wsgi_app(application) if __name__ == "__main__": main() Tuesday, July 6, 2010
- Pricing and Availability • Pay as you go pricing • Storage - $0.17/GB/month • Network o Upload data to Google $0.10/GB o Download data from Google $0.15/GB for Americas and EMEA $0.30/GB for APAC • Requests o PUT, POST, LIST - $0.01 per 1,000 Requests o GET, HEAD - $0.01 per 10,000 Requests • Free storage (up to 100GB) during preview period o No SLA • http://code.google.com/apis/storage Tuesday, July 6, 2010
- What's Coming Up • Service Level Agreement • Support • Available to Premium Apps Customers • Technical Features: o Group support in ACLs o Resumable uploads o Additional regions Tuesday, July 6, 2010
- Google BigQuery and Prediction APIs Tuesday, July 6, 2010
- Overview • Big Data - Challenging and Important • Google has tools for deep data analysis • Now you can use these tools Tuesday, July 6, 2010
- Overview • Big Data - Challenging and Important • Google has tools for deep data analysis • Now you can use these tools • Announcing two new APIs to get more from your data: 1.BigQuery 2.Prediction API Tuesday, July 6, 2010
- Benefits • Built on Google technology • Scalability • Security • Sharing • Easy integration with Google App Engine, Google Spreadsheets, .... Tuesday, July 6, 2010
- Using Your Data with BigQuery & Prediction API 1. Upload Upload your data Your Data to Google Storage Import to tables BigQuery 2. Process Train a model Prediction API Run queries Your Apps 3. Act Make predictions Tuesday, July 6, 2010
- BigQuery Interactive Analysis of Big Data Tuesday, July 6, 2010
- Big Data is Challenging Starts with Scale Tuesday, July 6, 2010
- Many Use Cases ... Interactive Tools Trends Spam Detection Web Network Dashboards Optimization Tuesday, July 6, 2010
- Demo: Analyzing M-Lab An open platform for advanced network research http://www.measurementlab.net/ Tuesday, July 6, 2010
- Demo: Exploring M-Lab Tuesday, July 6, 2010
- Key Capabilities of BigQuery • Scalable: Billions of rows • Fast: Response in seconds • Simple: Queries in SQL • Web Service o REST o JSON-RPC o Google App Scripts Tuesday, July 6, 2010
- Using Your Data with BigQuery Upload to Google Storage 1. Upload Import data into a BigQuery Table 2. Import - No need to define indices, keys, etc.. Execute queries via APIs 3. Query - No provisioning machines or resources Tuesday, July 6, 2010
- Writing Queries Compact subset of SQL o SELECT ... FROM ... WHERE ... GROUP BY ... ORDER BY ... LIMIT ...; Common functions o Math, String, Time, ... Statistical approximations o TOP o COUNT DISTINCT Tuesday, July 6, 2010
- API in a Minute GET /bigquery/v1/tables/{table name} GET /bigquery/v1/query?q={query} Sample JSON Reply: { "results": { "fields": { [ {"id":"COUNT(*)","type":"uint64"}, ... ] }, "rows": [ {"f":[{"v":"2949"}, ...]}, {"f":[{"v":"5387"}, ...]}, ... ] } } Also supports JSON-RPC Tuesday, July 6, 2010
- Security and Privacy Standard Google Authentication • Client Login • OAuth • AuthSub HTTPS support • protects your credentials • protects your data Use Google Storage for Developers to manage access Tuesday, July 6, 2010
- Large Corpus Analysis Wikimedia Revision History Wikimedia Revision history data from: http://download.wikimedia.org/enwiki/latest/ enwiki-latest-pages-meta-history.xml.7z Tuesday, July 6, 2010
- Using BigQuery Shell Python DB API 2.0 + B. Clapper's sqlcmd http://www.clapper.org/software/python/sqlcmd/ Tuesday, July 6, 2010
- BigQuery from a Spreadsheet Tuesday, July 6, 2010
- BigQuery from a Spreadsheet Tuesday, July 6, 2010
- BigQuery Recap • Interactive analysis of very large data sets • Simple SQL query language • APIs enable a variety of use cases Tuesday, July 6, 2010
- Google Prediction API Machine learning as a web service Tuesday, July 6, 2010
- Prediction API 101 • Google's sophisticated machine learning algorithms • Available as an on-demand RESTful HTTP web service • Train a model offline/asynchronously • Predict results in real-time "Tous pour un, un pour tous, Prediction API "french" c'est notre devise." Tuesday, July 6, 2010
- How does it work? "english" The quick brown fox jumped over the The Prediction API lazy dog. finds relevant "english" To err is human, but to really foul things features in the up you need a computer. sample data during "spanish" No hay mal que por bien no venga. training. "spanish" La tercera es la vencida. The Prediction API later searches for ? To be or not to be, that is the question. those features during prediction. ? La fe mueve montañas. Tuesday, July 6, 2010
- A virtually endless number of applications... Customer Transaction Species Message Diagnostics Sentiment Risk Identification Routing Churn Legal Docket Suspicious Work Roster Inappropriate Prediction Classification Activity Assignment Content Recommend Political Uplift Email Career Products Bias Marketing Filtering Counselling ... and many more ... Tuesday, July 6, 2010
- Three simple steps to use the Prediction API Upload your Use the API, gsutil or any 1. Upload training data to compatible utility to upload Google Storage your data to Google Storage Build a model prediction/v1/train/{} 2. Train from your data POST : a training request prediction/v1/query/{} Make new GET : model info 3. Predict predictions POST : a prediction request Tuesday, July 6, 2010
- Prediction API Demo Automatically categorize and respond to emails by language • Customer: ACME Corp, a multinational organization • Goal: Respond to customer emails in their language • Data: Many emails, tagged with their languages • Outcome: Predict language and respond accordingly Tuesday, July 6, 2010
- Step 1: Upload Upload your training data to Google Storage • Training data: outputs and input features • Data format: comma separated value format (CSV) $ head -n 2 ${data} "english","To err is human, but to really ..." "spanish","No hay mal que por bien no venga." Upload to Google Storage $ gsutil cp ${data} gs://io10/${data} Tuesday, July 6, 2010
- Step 2: Train Create a new model by training on data To train a model: POST prediction/v1/train/${data} Training runs asynchronously. To see if it has finished: GET prediction/v1/query/${data} {"data": { "resource": { "data": "${data}", "modelinfo": "estimated accuracy: ${acc}"}}} Tuesday, July 6, 2010
- Step 3: Predict Apply the trained model to make predictions on new data POST prediction/v1/query/${data} { data : { "instance" : { "input" : { "text" : [ "J'aime X! C'est le meilleur" ]}}}} Tuesday, July 6, 2010
- Step 3: Predict Apply the trained model to make predictions on new data POST prediction/v1/query/${data} { data : { "instance" : { "input" : { "text" : [ "J'aime X! C'est le meilleur" ]} "output" : {"output_label" : "french"}}}} Tuesday, July 6, 2010
- Step 3: Predict Apply the trained model to make predictions on new data import httplib header = {"Content-Type" : "application/json"}#...put new data in JSON format in params variable conn = httplib.HTTPConnection("www.googleapis.com")conn.request("P OST", "/prediction/v1/query/${data}", params, header)print conn.getresponse() Tuesday, July 6, 2010
- Prediction API Capabilities Data • Input Features: numeric or unstructured text • Output: up to 100s of discrete categories Training • Many machine learning techniques • Automatically selected • Performed asynchronously Access from many platforms: • Web app from Google App Engine • Apps Script (e.g. from Google Spreadsheet) • Desktop app Tuesday, July 6, 2010
- Prediction API and BigQuery Demo: Tagger Input Data: http://delic.io.us/chanezon – 6000 urls, 14000 tags in 6 years Analyze my delicious tags – use delicious API to get all tagged urls – cleanup data, resize (100Mb limit) – PUT data in Google storage – Define table – analyze Predict how I would tag a technology article – input is tag,url,text – send new url and text – get predicted tag Tuesday, July 6, 2010
- Get the BigQuery & Prediction APIs • Preview, opened to a limited number of developers • You need a Google Storage for Developers account • To request access and get more information, go to: o http://code.google.com/apis/bigquery o http://code.google.com/apis/prediction Tuesday, July 6, 2010
- Acknowledgement Thanks to many Googlers from their slides, most available at the Google IO 2010 website 131 Tuesday, July 6, 2010
- Thank you Read more http://code.google.com/appengine/ Contact info Patrick Chanezon Developer Advocate chanezon@google.com http://twitter.com/chanezon Questions ? 132 Tuesday, July 6, 2010
- Tuesday, July 6, 2010
