Building TweetEngine


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Building TweetEngine

  1. 1. Building TweetEngine Ikai Lan Developer Relations, Google App Engine Twitter: @ikai
  2. 2. Goals of this talk Using TweetEngine to explain key concepts of building cloud applications Explaining App Engine as a development environment Whet your appetite!
  3. 3. What is App Engine? App Engine is a platform You build & test your app Then upload your app to Google App Engine runs everything No need to worry about machines, network, storage, scalability, etc.
  4. 4. The king of ease-of-use Extremely rapid development Very low barrier of entry Simple yet robust syntax Rich library of packages/modules App Engine's first language API
  5. 5. Components
  6. 6. Getting TweetEngine Application: Code: Prerequisites: Python 2.5 buildout lxml (for i18n features)
  7. 7. OAuth Secure method for granting permissions to third-party integrations Allows sites to use a trusted identity provider (Google, Twitter) "Valet key" for the internet Image and "valet key" source: (
  8. 8. Standard OAuth flow (User) What does the User see? 1. Link to login with Twitter 2. User is redirected to an http: // URL 3. User logs in, grants access 4. User is redirected back to site
  9. 9. Standard OAuth flow (Server) What does the server do? 1. Acquire request token 2. Generate login URL 3. Redirect or link user to this URL 4. After user has logged in, Twitter will redirect to a callback URL on server 5. Server saves token passed to callback URL
  10. 10. TweetEngine + OAuth OAuth request token saved in TwitterAccount model Saved per Twitter Account handles signing handles proxied Twitter API calls
  11. 11. Logging in to Twitter using OAuth # src/tweetengine/ auth_token = self.request.get("oauth_token") auth_verifier = self.request.get("oauth_verifier") user_info = client.get_user_info(auth_token, auth_verifier=auth_verifier) # Create the twitter account account = model.TwitterAccount. get_or_insert( user_info["username"], oauth_token=user_info["token"], oauth_secret=user_info["secret"], name=user_info["name"], picture=user_info["picture"])
  12. 12. Initializing an OAuth client # src/ OAuthClient.__init__(self, "twitter", consumer_key, consumer_secret, "", "", callback_url)
  13. 13. Twitter API calls - Web handlers user AJAX - Backend calls go to TwitterApiHandler # src/tweetengine/handlers/ for k,v in params.items(): if isinstance(v, unicode): params[k] = v.encode('utf8') # Join all of the params together. params_str = "&".join(["%s=%s" % (encode(k), encode(params[k])) for k in sorted(params)]) # Join the entire message together per the OAuth specification. message = "&".join(["GET" if method == urlfetch.GET else "POST", encode(url), encode(params_str)]) # Create a HMAC-SHA1 signature of the message. key = "%s&%s" % (self.consumer_secret, secret) # Note compulsory "&". signature = hmac(key, message, sha1) digest_base64 = signature.digest().encode("base64").strip() params["oauth_signature"] = digest_base64 return urlencode(params)
  14. 14. OAuth The "de facto" standard for authentication on the web against a third party API TweetEngine's support can be easily generalized to be used on any OAuth supporting site Highly recommend: New sites, do not build identity system, use OAuth or OpenID
  15. 15. i18n - internationalization Problem: maintaining localized versions Localization is more than translations TweetEngine already supports English, German and Italian! Image source: com/photos/natematias/22132919
  16. 16. Workflow 1. Developer marks strings as requiring translation 2. Translators work on .PO files 3. These are compiled into efficient .POT files 4. At runtime, translations are looked up when pages rendered using "Accepts-Header"
  17. 17. What does a .po file look like? #. Default: "Collaborative tweeting" #: tweetengine/templates/ msgid "subline" msgstr "Tweeting collaborativo" #. Default: "Welcome!" #: tweetengine/templates/ msgid "title-welcome" msgstr "Benvenuto!"
  18. 18. Sample template code tweetengine/templates/ <metal:title fill-slot="title" i18n:translate="title-welcome">Welcome!</metal:title> tweetengine/templates/ <h2><a href="/" i18n:translate="subline">Collaborative tweeting</a></h2> Run from TweetEngine root: bin/i18nize
  19. 19. src/tweetengine/handlers/ # Instantiate Chameleon template loader from chameleon.zpt.loader import TemplateLoader from tweetengine import model from import mainmenu tpl_path = os.path.join(os.path.dirname(__file__), "..", "templates") tpl_loader = TemplateLoader(tpl_path) # Later on in the file template_vars['target_language'] = self.request.headers.get('Accept-Language', None) tpl = tpl_loader.load('') template_vars['master'] = tpl.macros['master'] tpl = tpl_loader.load('') template_vars['macros'] = tpl.macros tpl = tpl_loader.load(template_file) self.response.out.write(tpl(**template_vars))
  20. 20. Remember: i18n > translations Resizing UI elements for languages with long/short words Right-to-left languages (Hebrew, Arabic) and layouts Time and date formats Metric/English system Much more ... this just a place to start!
  21. 21. App Engine is also about tooling Image source:
  22. 22. Two App Engine tools AppStats Task Queues Application profile that is Background queueing App Engine aware and worker mechanism Image source:
  23. 23. AppStats Track expensive requests Expose bottlenecks in your code Show you what code is being executed the most for "hotspot optimization" Run in production with minimal overhead
  24. 24. AppStats example
  25. 25. AppStats example (con't)
  26. 26. How does it work? Uses Call Hooks pre‐call hook records start .me, request, stack post‐call hook records end .me, mcycles, response Measuring times: real time using Python’s wall clock API API megacycles using rpc.cpu_usage_mcycles CPU megacycles using quota.get_request_cpu_usage() Get stack contents from Python’s sys._getframe() Recording accumulates in memory object Written to memcache at end of each request
  27. 27. AppStats for TweetEngine Add AppStats as WSGI Middleware: app = recording.appstats_wsgi_middleware(app) Add AppStats GUI to URL mapping (app.yaml): - url: /stats.* script: appstats/ Dev: http://localhost:8080/stats Prod (as admin): That's it!
  28. 28. Task Queues Execute background tasks Throttle control ETA Can be unique! Invoked using a web handler
  29. 29. Standard async worker architecture
  30. 30. Task Queues in App Engine # Creating a task # src/tweetengine/ taskqueue.Task (eta=send_time, name=task_name, url=twitter. ScheduledTweetHandler.URL_PATH).add() # Task Handler # src/tweetengine/handlers/ class ScheduledTweetHandler(webapp.RequestHandler): def post(self): publishApprovedTweets() def publishApprovedTweets(): # Iterate through unsent OutgoingTweets and send them
  31. 31. Task Queues in TweetEngine Used to schedule Tweets ETA parameter used to set an approximate execution time Tasks are named - this allows us to set times to check for OutgoingTweets that should be sent at this time and not duplicate work (we don't have to use cron)
  32. 32. Task Queues Allow developers to build asynchronous workers without hassle of building out queueing/polling system Handles task uniqueness and transactional task creation Can be scheduled, throttled
  33. 33. Summary: TweetEngine technologies 1. OAuth - "Valet key" for internet applications 2. i18n - Using templates and message bundles in web applications 3. AppStats - App Engine request profiling toolkit 4. Task Queues - Background work for App Engine applications
  34. 34. Questions? Application: Code: Twitter: @ikai