Developing TouchActive with GAE and GWT - Google DevFest Barcelona 2012


Published on

This presentation describes our experience developing the TouchActive platform using Google App Engine and Google Web Toolkit.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Developing TouchActive with GAE and GWT - Google DevFest Barcelona 2012

  1. 1. Google App Engine &Google Web Toolkitour experiencedevelopingTouchActiveAndrew MackenzieDaniel RuizBCNTouch
  2. 2. The purpose of this talk● Share our experiences using GAE and GWT● Help guide you if you are considering similar technologies● Provide ideas if you are already developing with these technologies● Provide feedback to Google on things to maintain, improve and renew/drop
  3. 3. Contents ● SolutionOur Project ● PaaS choice ● GAE overview ● GWT overviewGAE + GWT ● GAE + GWT debugging ● TouchActive implementationImplementation ● Addit ● Optimization tools ● Phonegap experimentSummary ● Avoiding lock-in ● What would we do differentlyFeedback ● Feedback to Google
  4. 4. Our Project
  5. 5. TouchActive Platform- what we built Brand A platform to put Using tagged consumers and products that thebrands in touch with consumers were each other interested in Consumer
  6. 6. Mobile apps to read tagged products - NFC, QR, barcode Scalable store in the cloud with product Cloud service and data and data storage synchronization of user data Mobile apps read products tagged with various technologies Get product information, helpremember, organize in lists, etc. Focus on easy and intuitive interaction with no data entry. NFC the best.
  7. 7. Get information about productsfrom tagged objects
  8. 8. Management console for brand- multi-tenant, groups, roles Access control on changes Cloud service and data storage Products findable in open queries from mobile devices
  9. 9. Consumer Web Interface- Lists, Products, History, etc Mobile web version for mobile platforms without a native app Addit "Web Clipper" to capture products from web pages User data synchronized with native mobile app Cloud service and data storage
  10. 10. Social networks, Identity and SharingLeverage existinguser identities for easy sign-up(web and mobile) - OpenId and OAuthSharing of Products, Lists via:● Social Networks● EMail● NFC Tags● NFC peer-to-peer● QR codes● Bump
  11. 11. TouchActive Solution Overview- what we built
  12. 12. TouchActive Solution Overview- what we built Desktop and Mobile IdentityManagement consumer web Providers console Addit Sharing (web and mobile) over net and Cloud backend face-to-face Mobile Apps Tagged products
  13. 13. Context for our choices2+ years ago● Start Small ○ pre-start-up: build prototype ○ uncertain financing (probably small) ○ small team of engineers● Think big, or the possibility of big!● Uncertainties ○ Mobile platforms (especially NFC) ○ Cloud computing platforms
  14. 14. CoolingPower / UPS Security Buildings
  15. 15. LAMP stack on owned Amazon Web Services Google App Engine or hosted HW (AWS) (GAE) small start-up, low Newer options: capital investment. ● OpenStack (Open SW, multiple service Pay as you go, dont providers) buy servers. ● Windows Azure ● Amazon features for auto scaling, etc.unknown/unexpected demand we dont spend people time and money on managing servers, runtimes, etc. Focus on the application. chose GAE Cooling Power / UPS Security But we didnt want to get "locked-in" Buildings to a single service provider
  16. 16. Applications developedwith GWTAdWords AdSense Orkut Wave Health BloggerAnd many more, see:
  17. 17. Google Web ToolkitComplex UI Complex UI with graphical Data representation
  18. 18. GAE + GWT10
  19. 19. Google App Engine Features● Free developer / Pay-as-you go costs● Platform not Infrastructure ○ No need to architect the SW stack for scalability ○ No need to manage the SW stack ○ No machine administration ○ Reasonable set of platform services● Scalability ○ Instances created automatically based on demand ○ Scalable, distributed, datastore ○ Content Distribution Network (CDN) ○ High-performance Image service
  20. 20. Google App EngineTools and LanguagesLanguage choices at the time: Java and Python ○ Now "GO" is also available nowWe chose Java ● Developing J2ME and Android mobile apps in Java at the same timeEclipse IDE ○ Local development server ○ Java debugging of server codeAppstats for server performance analysis (RPC)Management dashboardCommand line tools and ANT tasks for automation
  21. 21. Google App EngineDatastore Datastore Megatable BigTable GFS2 Advantages Drawbacks● Scalable ● Pay-per-use● Time vary with result set ● NoSql & pre-built size, not search set size indexes● Compatible with existing ○ No JOIN operations frameworks ○ No inequality filters on more ○ Java Data Objects (JDO) than one "field" ○ Java Persistence API (JPA) ○ No filtering of data based on● Distributed, replicated subqueries across data centers ○ No Grouping of results ● Write rate limited
  22. 22. Google App EnginePlatform ServicesStandard Implementations Non-Standard● Mail (JMail) ● Cron Tasks● Datastore (JPA/JDO) ● Task Queues● MemCache (JSR107) ● Back-ends● Logging (Java Logging) ● Users (Identity) ● Blob Store ● Image Service ● Chat (XMPP/Jabber) with Presence ● Channels API (Push)
  23. 23. Web Client DevelopmentGoogle Web Toolkit● Develop Javascript AJAX client with Java ○ Strongly, statically, typed language. ○ Many errors caught at compile time ○ Full IDE support (completion, etc) ○ Internationalization (i18n) built-in ○ Generates optimized JavaScript, client bundles, code splitting, etc. ○ Hides browser differences ■ Compiles a "permutation" for each language and browser you support● Integrated Server and Client development project ○ Java across Server and Client ○ Shared classes between Server and Client ○ "Transparent" RPCs between server and client (takes care of
  24. 24. Web Client DevelopmentGoogle Web Toolkit● Deferred binding with "properties" ○ Used for loading script variants based on browser, language, form factor, etc● Reasonable Widget set ○ Has improved a lot ○ Other widget sets available on top (GXT from Sencha, SmartGWT, Vadim, etc)● "DevMode" ○ Java debug experience in IDE for client side code ○ Combined Client/Server/RPC debug experience ○ No-compile, fast turn-around, hot code replace ○ Memory intensive! :-(
  25. 25. Web Client Development Google Web Toolkit imageEncourages reusablemodular code development● Custom widgets● Code in defined "modules" (server/client/shared code, RPCs, html, css, images, etc)● Bundle modules into GWT libraries (.jar)
  26. 26. Web Client Development Google Web ToolkitEase of implementingDesign Patterns● Model-View-Presenter with late-binding of views● Same Presenter, different Views● Detect Form Factor from UA and serve the most appropriate view at run- time
  27. 27. Google Web ToolkitLate-Binding of views with ClientFactory TouchActive.gwt. xml
  28. 28. Google Web ToolkitLate-Binding of views with ClientFactory Desktop Mobile
  29. 29. Google Web ToolkitMVP, Activities & PlacesEvent-Bus across client code forsubscribe/broadcast eventprogrammingApplication Activity and Placemanagement via browser history History Token
  30. 30. Google Web ToolkitUiBinder (XML) for declarative layout With standard or custom widgets
  31. 31. GAE/GWT Debug experience:An example Click Event on Facebook sign in button
  32. 32. GAE/GWT Debug experience:Client debugging
  33. 33. GAE/GWT Debug experience:Server debugging
  34. 34. GAE/GWT Debug experience:Client debugging
  35. 35. Implementation20
  36. 36. Our implementation of TouchActivewith GAE + GWT ● Authentication / Identity management ● Architecture overview ● Data Persistence Layer ● Platform API ● Portability ● CMS Integration ● Addit
  37. 37. Authentication:Own, portable, Identity management● Created own User Object and related to standard external accounts ○ OAuth (FB, Twitter, LikedIn) ○ OpenID (Google, Yahoo, MySpace, ...) ○ Google Apps domains for company users● Username + Password also● Avoid forcing users having to have a Google account● Use OAuth & OpenId on mobile apps transparently for account creation and sign-upAvoided lock-in, didnt use Google App Engine UsersService
  38. 38. Architecture Overview Request ● Headers, cookies, ... Servlet FiltersRequest App LogicContext Platform API Data Persistence Layer Platform Specific Services
  39. 39. Servlet Filters● Flexible, just map filters to servlets with a config file using paths ○ /api ○ /rpc ○ /cms ○ /static● Keeps servlet code clean and modular● Used for: ○ Language selection ○ Desktop/Mobile pages while maintaining constant URL ○ Authentication + assignment user to their Multi-Tenancy namespace ○ Query logging for analytics
  40. 40. Servlet Filters: EntryPoint Filter (The Special One) Request ● Headers, cookies, ... EntryPoint Filter Servlet Filters Request Context● Different types of EntryPoint filters that set different permissions and tokens ○ Initializes the Request Context used in your app and DataManager ○ Use of Thread variables to make available to all code but multi-threaded● Authenticates the user and sets the correct "Namespace" in the request context● Defined the appropriate filter for each request based on Access Control rules in web.xml
  41. 41. Data Persistence LayerRequestContext DataManager I/F to rest of app Data Manager JDO portable library across multiple datastore implementations Google Datastore MySQL Google AppEngine AWS MySQL Server
  42. 42. Data Persistence LayerDataManager Interface DataManager I/F to rest of app● Created an API to manage data objects for reading and writing Data Manager● DataManager layer is used by all app, app is kept 100% portable JDOData Objects● ACL, Protected, Global, etc...● Enforce visibility and Google Datastore MySQL permissions according to filters used on request and object type MySQLMulti-Tenancy Google AppEngine AWS Server● Create data areas in the same datastore● Avoided using Google Datastore Namespaces
  43. 43. Platform Specific Services App LogicDefined a Platform API for platform dependant services GAE Implementation Other Implementation Cron Tasks Cron Tasks Task Queues Task Queues Blobstore Blobstore Image Service Image Service Custom build process Chat (XMPP) Chat (XMPP) outside IDE ● ANT tasks to build, deploy, test, Channels (Push) Channels (Push) release ● Run manually from Command Line or scripts / cron ● Significant effort
  44. 44. Additional Need:Web content beyond the appEditable by non-programmersNo need for application release on each changeLocalized to multiple languagesUse a CMS GWTCMS Content
  45. 45. Integrating a CMSChose to serve content, scripts and RPCs from same domain ○ Deploy CMS to same GAE "appID" as our app ○ This could maybe be avoided with subdomains...Chose Vosao GAE/J based CMS ○ A bit immature at the time....still < 1.0 ○ Uses Apache Velocity (Java based) scripting language ○ Java plugin can integrate tightly with your app and call app code directly from within CMS web pages ○ Has locked us in to GAE at CMS level, but our code is still portable ○ Manual integration of config files, with help from Vosao team it could be automated
  46. 46. Internationalization (i18N)● Client GWT UI ○ Built-in i18N support in GWT, strings in property files ○ Permutation of script compiled for each language and correct language loaded at run time● Server messages ○ We have avoided as much as possible ○ Java Message Bundle + Properties files● CMS content ○ Core feature of CMS. Individual strings or entire pages can be localized with common template
  47. 47. Other Features30
  48. 48. Other features implemented● Chat functionality with server (XMPP) ○ used for testing at the moment● "Presence" detection of users (XMPP)● Push messages to browser using the GAE Channel API ○ GWT Client JSNI Wrapper and EventBus connection● "Addit" Bookmarklet and "web-clipper"
  49. 49. Addit -Objectives● Help users create products in TouchActive by capturing product from any web page● Reduce need for data entry by capturing as much data as possible automatically ○ Use eCommerce/Search meta data when available● Relatively simple and easy to use● Broad usage ○ Support all main web browsers ○ Support as many web sites as possible ○ Desktop, Tablet and Mobile versions
  50. 50. Product meta-datameta and micro-formats used by eCommercesites to provide data to search engines orcrawlers:Various Standards Exist:● Facebook ○ OpenGraph "product"●● GoodRelations● GoogleProducts ○ Google Merchant Center / Google Product Search● hProduct
  51. 51. Google Product Search
  52. 52. Addit Installation Drag and drop bookmarklet from web-page to bookmarks bar or folder Install extension directly ● From Chrome Web Store ● Chrome, Safari, Firefox from web page
  53. 53. User clicks either theAddit Use Bookmarklet or the Extension button (FF, Safari, Chrome)
  54. 54. Addit Detect language from Load localized script from cloud server and injectedScript loading page contents into the page Keeping script out of the bookmarklet or extention allows us to update the script continually, without any re-install or redistribution to users Script NOT from Same- Origin as page and security concerns and barriers to overcome
  55. 55. Addit Fetch user lists and detect currencyFetches users lists from their TouchActiveaccount using a Cross-Domain "JSONP" RPC call Select default currency based on page language, page contents and user settings retrieved with lists
  56. 56. Addit product meta-data parsingParse DOM and detect micro-format product markup elements.Prepares products for user automatically If multiple products detected, a simple product selector UI is shown initially If no product detected,uses page-level markup (meta, OpenGraph) or general page information (Title, etc)
  57. 57. AdditPersonalize and save to account Users take the suggested texts, or customize them. Select image, price, etc. Select one of their own lists they would like to add the product to Add a personal note ... and "Addit" to their TouchActive account Will be automatically synced with their mobile app
  58. 58. AdditDifferent layouts for different form factors Desktop MobileWider, no vertical scrolling, smaller mouse targets Narrower (adjust to Viewport), vertical scrolling, larger touch targets
  59. 59. All Developed in GWT● GWT XSI Cross-Site linker for CORS script loading● Wrote own JSONP library for Cross-Site RPC ○ in Java with some JSNI● GWT "late run-time binding" of scripts for Localization and Form Factor permutations using custom "Property Providers" ○ Detect "best guess" language from host page using multiple methods (poor standards implementations) ○ Detect form factor using a combination of User Agent and display size● Avoid older Layout Widgets that compile down to HTML Tables (Box Model and CSS problems...)● Fast product meta-data parsing from DOM using "querySelectorAll()" and CSS3 selectors
  60. 60. Difficulties - mainly due to injecting ascript into a page you dont control● Unknown, uncontrolled CSS / DOCTYPE / QuirksMode ○ Cant just do a "CSS Reset" ○ CSS properties influence complex layouts ○ DOCTYPES affect browser behavior (a nightmare on IE 7/8/9)● "Difficult" sites ○ Dont follow W3C standard for language selection ○ iFrames/Frames/Frameset based ○ Flash sites! ○ Broken image links● Different implementations of product markup standards ○ Especially prices "," vs "." EUR vs € before/after/middle (1€25)● JavaScript image galleries and tricks ○ Images loaded but not visible ○ Images being pre-cached (not always in DOM as image)
  61. 61. IE and DOCTYPE and QuirksFor masochists only!
  62. 62. Phonegap ExperimentQ Can you take a GWT compiled app (HTML, CSS, JS, Images etc) and create a "native" app for multiple platforms?A Kind-of...● GWT RPCs from Phonegap client to server! ○ Must be able to go cross-domain ○ We wrapped the RPC class so that it remaps the RPC endpoint to default server when running "offline" ○ Version problems expected with serialization in the future!!● Phonegap has problems with relative file paths ○ Processed GWT comiled output to use absolute paths● Prototype produced from existing GWT app ○ Automated with ANT tasks
  63. 63. Optimization40
  64. 64. Optimization Tools AvailableChrome Developer Tools "Audit"PageSpeedAppStats for server GAE internal RPCSpeedTracerSpeedTracer with AppStats
  65. 65. Chrome Audit
  66. 66. Page Speed
  67. 67. GAE AppStatsunderstand AppEngine RPC usage
  68. 68. AppStatsdrill-down to calls, duration and latency
  69. 69. SpeedTracerUnderstanding client side performance
  70. 70. SpeedTracerNetwork resource usage from client Resource Loads ● HTML ● CSS ● JavaScript
  71. 71. SpeedTracer (client) integrated withAppStats (server) RPC request to server Server side trace and timing from AppStats on server
  72. 72. Whats new in GWT2.5● Super Dev Mode (experimental) ○ Replaces DevMode, uses "SourceMaps" in Chrome to enable debugging in language JS was written in (Java, Dart...)● New compiler optimizations ○ Smaller, faster JavaScript code ○ With slightly longer compile times● Elemental ○ Lightweight JavaScript library● Updated ARIA support● UIBinder enhancements● Validation enhancements
  73. 73. GWT Compiled codesize reductions GWT 2.4 100% GWT 2.5 93% GWT 2.5 Closure Compiler 87%
  74. 74. Summary40
  75. 75. Did we avoid provider "Lock-in"?Portable data layer built on JDO● Maybe not the most optimal when running on GAE● Have to work with limitations of JDO/GAE subset● Over time, actually moving the data becomes a problemPlatform API has kept rest of code portable● needs work to get running on additional platformsGAE based CMS has increased it● outside of the code, but still important for the project● decouple by moving to multiple domain approach
  76. 76. Things we would do differently if we started againNew competitors to GAE ● New AWS services bring it closer to PaaS ● OpenStack from multiple Analytics providers ● Analyze more closely and prototype if time permits Phonegap app API and directory structure CMS
  77. 77. Things we would do differently if we started againNew competitors to GAE Analytics ● Avoid programming a lot of that, and falling short Phonegap app ● Expose analytics data and use external tools ● Give business users top- API and directory class tools with flexibility of structure analysis, drill-down etc CMS
  78. 78. Things we would do differently if we started againNew competitors to GAE Analytics Phonegap app ● Design and structure web-app for that from day one API and directory structure CMS
  79. 79. Things we would do differently if we started againNew competitors to GAE Analytics Phonegap app ● Assume you will have toAPI and directory structure version your external API ● Design app structure early on ○ statics, resources, API, CMS CMS, RPC,...
  80. 80. Things we would do differently if we started againNew competitors to GAE Analytics Phonegap app API and directory ● Explore using other CMS structure outside of GAE ● Sub-domains for "Single- CMS Origin" content, RPCs
  81. 81. Possible Future Work● Optimize ○ Using tools to focus our efforts ○ More use of memcache for DS accesses● Explore use JDO3 (Polymorphism)● Prototype alternative CMS ○ Run on sub-domain in parallel● Write platform code for other platforms ○ Test on AWS or OpenStack?● Phonegap app for other platforms ○ Blackberry, Windows Phone...● Investigate Continuous Integration Support ○ Jenkins on GAE
  82. 82. GAE + GWTThe Good● Scalable, without knowing about or managing the stack● Free for developers, our costs have been pretty low● Java across mobile, backend and client● Programming large app "all together" as client-server ○ But avoid that permitting spaghetti, structure APIs and code ○ Hierarchy of reusable modules of client, server and shared code ○ Drop-in GTW libraries with client, server and shared code● Excellent debug experience when it works! ○ including hot code replacement (server + client)● Model-View-Presenter architecture combined with late- binding ○ extend your app to mobile, tablet views more easily● UI Binder for declarative UI layout, code for behavior
  83. 83. GAE + GWTThe BadDevMode problems● Lack of maintenance on browser plug-ins● Memory intensiveYou have to put in effort to avoid "lock-in"JDO● Some people reporting long start-up times (meta-data related) ○ Has worked pretty well for us● Some features missing: e.g. No Async writes
  84. 84. Feedback44
  85. 85. Feedback for GoogleApp Engine● Dont introduce features or ways to use features that increase lock-in. ○ If you do, we will go to "standard" stacks on AWS or OpenStack... ○ e.g. threads, etc● More transparency on Billing and scheduler behaviour● Need better attention to developers in forums ○ Paying customers! ○ Especially around scheduling and billing - invest people hours in this● Channels API ○ Support multiple listeners per page and broadcast ○ Google supported JSNI GWT library
  86. 86. Feedback for GoogleGWT Debugging Dont break the debug experience!● Combined server/RPC/client debug experience in a single IDE is fantastic and unique (?) in developing an AJAX client - cloud server application ○ Make it better! More maintenance of plug-in ○ Support at least on Chrome / WebKit browser ○ Less of a memory hog● GWT 2.5 SuperDevMode does not replace it IMHO ○ list of limitations of seems to outweigh the advantages ○ main advantages (plug-in maintenance) are for Google
  87. 87. Feedback for GoogleGWT - others● Allow client-only code and server-only code in same file ○ E.g. Via @annotations ○ Avoid having to complicate the class hierarchy to have shared/server-only/client-only code for a class● Java Calendar emulation in GWT and allow for shared-code date handling
  88. 88. Thank You! Questions? Andrew Mackenzie Daniel Ruiz Giménez