Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mobile Apps quality - a tale about energy, performance, and users’ perception

Slides of my guest lecture at TUDelft held in Oct 2017.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Login to see the comments

  • Be the first to like this

Mobile Apps quality - a tale about energy, performance, and users’ perception

  1. 1. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM VRIJE UNIVERSITEIT AMSTERDAM Ivano Malavolta Assistant professor Vrije Universiteit Amsterdam Mobile Apps quality - a tale about energy, performance, and users’ perception
  2. 2. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM iOS developer Who’s speaking? 2009 Android developer2010 Cross-platform developer2011 Instructor on mobile apps development for 5 years2012 VRIJE UNIVERSITEIT AMSTERDAM 2014 2016 Experiments at the GREEN LAB Empirical research on hybrid apps 2017
  3. 3. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM The Green Lab Our platform for researching on software – energy efficiency – performance Students measure real software solutions OpenSTF + ADB Serve web pages VMWare manager SSH MEASURES Wifi network Router Experiment orchestrator OpenSTF Web interface Power meters A PLATFORM A COURSE
  4. 4. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Roadmap Questions Deep dive into mobile apps development Who is speaking? Energy consumption (of progressive web apps) User perceptions (of hybrid apps) Performance (of native apps)
  5. 5. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Deep dive into mobile apps development Ivano Malavolta (2016). Web-based hybrid mobile apps: state of the practice and research opportunities. In Proceedings of the International Conference on Mobile Software Engineering and Systems, MOBILESoft '16 Ivano Malavolta (2016). Beyond Native Apps: Web Technologies to the Rescue! (Keynote). In Proceedings of the 1st International Workshop on Mobile Development (Mobile!)
  6. 6. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Let’s burst the first myth http://www.slideshare.net/fling/native-v-hybrid-v-web Mobile apps development IS NOT CHEAP
  7. 7. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM NATIVE APP 01010101010101101010 1010101011011010 010101010101011101 010101010101011010 PLATFORM APIs Native Development strategies
  8. 8. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM • rich user interfaces and heavy graphics • work offline • visibility via stores • slow iteration pace for iOS (stores mediation) • no indexing by search engines • and..... PROS CONS Native apps NATIVE APP 01010101010101101010 1010101011011010 010101010101011101 010101010101011010 PLATFORM APIs
  9. 9. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Fragmentation Obj-C code Swift code XCode Java code C++ code Eclipse C# code C++ code Visual Studio JS code
  10. 10. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM NATIVE APP 01010101010101101010 1010101011011010 010101010101011101 010101010101011010 PLATFORM APIs Native Development strategies BROWSER <html> <head> <script src=” ...” /> </head> <body> ... Web
  11. 11. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM • fast development • simple maintenance and updates • cross-platform • 2-steps access – they are launched via a browser • do not work offline • no background activities – geofencing • poor access to system APIs – push notifications, contacts, etc. Web apps PROS CONS
  12. 12. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM BROWSER <html> <head> <script src=” ...” /> </head> <body> ... NATIVE WRAPPER <html> <head> <script src=” ...” /> </head> <body> ... PLATFORM APIs NATIVE APP 01010101010101101010 1010101011011010 010101010101011101 010101010101011010 PLATFORM APIs Native Web Hybrid Development strategies
  13. 13. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Web-based hybrid mobile apps Single code base Bridge API
  14. 14. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Web-based hybrid apps • cross-platform portability • reuse of existing knowledge of web developers • simpler and less expensive development process • inherit some cons of native apps • restricted access to system APIs • (slight) decrease in performance PROS CONS
  15. 15. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM BROWSER <html> <head> <script src=” ...” /> </head> <body> ... NATIVE WRAPPER <html> <head> <script src=” ...” /> </head> <body> ... PLATFORM APIs NATIVE APP 01010101010101101010 1010101011011010 010101010101011101 010101010101011010 PLATFORM APIs Native Web Hybrid Development strategies Progressive BROWSER <html> <head> <script src=” ...” /> </head> <body> ... Service workers
  16. 16. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM The journey of a PWA 1. The user accesses the website as usual https://goo.gl/KIZydg
  17. 17. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM The journey of a PWA 1. The user accesses the website as usual 2. After the 3rd-4th visit, the website asks to be installed https://goo.gl/KIZydg
  18. 18. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM The journey of a PWA 1. The user accesses the website as usual 2. After the 3rd-4th visit, the website asks to be installed 3. The user can decide to add the app to the home screen https://goo.gl/KIZydg
  19. 19. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM The journey of a PWA 1. The user accesses the website as usual 2. After the 3rd-4th visit, the website asks to be installed 3. The user can decide to add the app to the home screen 4. From now on, the app is top-level, full-screen, and can receive push notifications https://goo.gl/KIZydg
  20. 20. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Components of a PWA HTTPS Web Manifest Service workers { "name": "MyApplication", "short_name": "MyApp", "scope": "./webApp/", "orientation": "portrait", "display": "fullscreen", "background_color": "#fff", "description": "A simple application for testing.", "icons": [{ "src": "images/touch/homescreen48.png", "sizes": "48x48", "type": "image/png" }, ... }], ... }
  21. 21. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Service workers Implemented in JavaScript Multithreading → runs in a separate thread w.r.t. the main thread Used for: • push notifications • background operations • content caching – offline functionality
  22. 22. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Offline-first experience Web App Service worker response response request Backend Offline cache this.addEventListener('fetch', function(event) { event.respondWith( caches.match(event.request).then(function(response) { return response || fetch(event.request); }); ); });
  23. 23. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Energy consumption (of progressive web apps) Ivano Malavolta, Giuseppe Procaccianti, Paul Noorland, Petar Vukmirovic. Assessing the Impact of Service Workers on the Energy Efficiency of Progressive Web Apps. In Proceedings of the International Conference on Mobile Software Engineering and Systems, MOBILESoft 2017,
  24. 24. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM PWAs have been advertised as: • performance boosters • network savers • providers of better UX However… How does the use of service workers impact the energy efficiency of PWAs under different network conditions? How does the use of service workers impact the energy efficiency of PWAs?RQ1 RQ2
  25. 25. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM PWAs selection • Real PWAs from the pwa.rocks1 repository • Pseudo-random selection – no toy examples – data-driven PWAs (aka no videogames) 1 https://pwa.rocks Name Category Total size SW size (loc) Ali Express Shopping 2.1Mb 69 Google I/O 2016 Events 4.2Mb 358 The Washington Post News 4.0Mb 85 Flipkart Shopping 3.8Mb 907 Babe News News 1.2Mb 156 Wiki offline Knowledge 800Kb 1009 The Billings Gazette News 2.1Mb 60
  26. 26. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Experiment design What we measure Energy consumption of the device in Joules Variable name Treatments SW status <on, off> Android device <high-end, low-end> Network condition <2G, WiFi> What we control What we measure
  27. 27. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Experiment design • full 2x2x2 factorial à all possible combinations of treatments • 8 combinations x 7 PWAs x 8 runs à • each run executes a typical usage scenario (10-15 gestures) 448 runs SW status Device Network on low-end 2G on low-end WiFi on high-end 2G on high-end WiFi off low-end 2G off low-end WiFi off high-end 2G off high-end WiFi
  28. 28. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM 7a. HTTP requests as part of the scenario Mobile device Server Measurement infrastructure Orchestration script Monkey runner Trepn profiler Chrome Monkeyrunner 1. HTTP requests impersonating phone 2. HTTP responses (recorded) 9. save collected data 3. Start experiment run 5. start 6. start 7b. HTTP responses (possibly altered) 8. collect data Fiddler proxy Hosted PWA ADB OS 4. start scenario run
  29. 29. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Results - devices Overall energy consumption across devices Low-end High-end High difference across devices
  30. 30. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Results: SW-on VS SW-off Low-end High-end Smaller difference in the high-end device Service workers DO NOT influence the energy consumption of a PWA running on a mobile device
  31. 31. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Results – network conditions High-end + 2G Low-end + 2G Low-end + WiFi High-end + WiFi PWAs consume less energy on WiFi Same device + same network condition à low impact of SWs
  32. 32. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM PWA-specific trends Low-end 2G WiFi High-end Different PWAs à different impact of SWs Same PWA à SWs have a different impact under different conditions - - - - + + + + - +
  33. 33. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM How does each SW look like?
  34. 34. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Results of manual review No specific trend here Name Listened events Caching Obfuscation/ minification Complexity Ali Express P, N ✓ ✓ 12 Google I/O 2016 I, A, F, M ✓ 9 The Washington Post I, A, P, N ✓ 131 Flipkart I, A, F ✓ ✓ 5 Babe News I, A, F, P, N ✓ 16 Wiki offline I, A, F, S, M, N ✓ ✓ 7 The Billings Gazette I, A, F ✓ 194
  35. 35. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Conclusions Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM PWAs have been advertised as: • performance boosters • network savers • providers of better UX However… Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM 7a. HTTP requests as part of the scenario Mobile device Server Measurement infrastructure Orchestration script Monkey runner Trepn profiler Chrome Monkeyrunner 1. HTTP requests impersonating phone 2. HTTP responses (recorded) 9. save collected data 3. Start experiment run 5. start 6. start 7b. HTTP responses (possibly altered) 8. collect data Fiddler proxy Hosted PWA ADB OS 4. start scenario run Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Results: SW-on VS SW-off Low-end High-end Smaller difference in the high-end device Service workers DO NOT influence the energy consumption of a PWA running on a mobile device
  36. 36. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Performance (of native apps) Teerath Das, Massimiliano Di Penta, Ivano Malavolta. A Quantitative and Qualitative Investigation of Performance-Related Commits in Android Apps. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME 2016)
  37. 37. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Goal Quantitative and qualitative characterization of (documented) performance-related commits for Android apps Developer ? App
  38. 38. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Research questions What are the concerns that developers have when dealing with performance issues of Android apps? To what extent developers consider performance issues of Android apps?RQ1 RQ2
  39. 39. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Dataset?
  40. 40. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Dataset
  41. 41. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM How to identify performance-related commits?
  42. 42. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Results (RQ1) Out of 68,025 commits of 2,443 apps, we discovered a total of 457 performance-related commits (0.67%) spread across 180 apps
  43. 43. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Open card sorting – 2 people performed the task, a third one checked the results 1st phase commits tagged based on keywords e.g., read from file system, swipe lag 2nd phase commits clustered into meaningful, labeled groups
  44. 44. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Results (RQ2)
  45. 45. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM User interface • Applied solutions: – use of Android’s recycled views instead of plain list views – render images in slices – ... Problems related to swipe lags, screen layout drawing, lists scrolling responsiveness FIX layouts for better rendering performance
  46. 46. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Code smells • Applied solutions: – use more efficient regular expressions – avoid recurrent computations of constant data – avoid usage of deprecated decryption algorithms – ... Symptoms of poor design and implementation choices fixed string concatenation performance issue
  47. 47. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Memory • Applied solutions: – stopping auxiliary services when available memory gets low – avoiding to load potentially unused data – ... Related to the memory footprint of the app Fixed major memory leak; should improve responsiveness on older devices
  48. 48. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Local database access • Applied solutions: – perform queries in an asynchronous task – add indexes to specific fields – ... Access to local database can be highly inefficient Added indexes for post.blogID to improve performance of several lookups
  49. 49. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM As a developer I can exploit this emerging collective knowledge for improving the performance of MY mobile app
  50. 50. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM User perceptions (of hybrid apps) Ivano Malavolta, Stefano Ruberto, Tommaso Soru, Valerio Terragni. Hybrid Mobile Apps in the Google Play Store: An Exploratory Investigation. 2nd ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft 2015). Ivano Malavolta, Stefano Ruberto, Valerio Terragni, Tommaso Soru. End Users’ Perception of Hybrid Mobile Apps in the Google Play Store. IEEE International Conference on Mobile Services (MS 2015).
  51. 51. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Pros • cross-platform portability • reuse of existing knowledge of web developers • simpler and less expensive development processes Hybrid mobile apps Cons • restricted access to hardware features • decrease in performance • variations on user experience As of today, limited empirical investigations have been performed on hybrid mobile apps Strong debate about benefits and drawbacks
  52. 52. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Research goal Developer End users creates download & use App What is the difference between hybrid and native mobile apps as perceived by end users? Are hybrid mobile apps published in the Google Play Store? What are their main traits?S1 S2
  53. 53. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Data extraction Classified apps (hybrid vs native) Hybrid apps classifier* top-500 most popular free apps for each category of the Google Play Store ~11k app binaries Apps and reviews mining * analysis tool available here: http://github.com/GabMar/ApkCategoryChecker
  54. 54. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Results Data- intensive mobile apps[2] Apps with closer interaction with the Android platform Winners, in line with informal claims[3,4,5]
  55. 55. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Quick test
  56. 56. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Data extraction Classified apps (hybrid vs native) Hybrid apps classifier* Reviews analyzer top-500 most popular free apps for each category of the Google Play Store ~11k app binaries 50 pages (~255) of reviews for each app ~3M user reviews apps scores Apps and reviews mining perceived value: 0.5 users sentiment: 0.6 #reviews: 243 performance: 0.6 bugginess: 0.1 size: 3,456 kb* analysis tool available here: http://github.com/GabMar/ApkCategoryChecker
  57. 57. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Reviews analysis Stopwords removal manually performed by 2 domain experts Single review Single review score polaritypos: 0.8 performancepos: 0.6 polarityneg: 0.1 performanceneg: 0.05 bugginess: 0.2 300 random reviews Keywords extraction Relevant keywords Lemmatization Tf-idf based vectors similarity computation
  58. 58. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Perceived value Average of the ratings as provided by end users 3.35 3.75 Rating = real number in [1, 5] Certain balance, with neglectable differences
  59. 59. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Perceived value Polarity of sentiment of end users where posa = #positive reviews nega = #negative reviews Balance between hybrid and native apps, with some exceptions Non data-intensive or requiring multimedia capabilities
  60. 60. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Perceived value Average review count where Ra ∈ ℕ Native apps have been reviewed in average 6.5 times more than hybrid mobile apps Possible interpretation: hybrid mobile apps are neither perceived as too satisfying nor dissatisfying w.r.t. native ones
  61. 61. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Perceived performance where posa = #positive reviews w.r.t. performance of the app nega = #negative reviews w.r.t. performance of the app Balance between hybrid and native apps, with some exceptions
  62. 62. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Perceived bugginess where buga = #reviews signalling the presence of bugs or failures reviewsa = #reviews of the app The highest unbalance between the two development strategies in our study bugginessa = buga / reviewsa Possible interpretation: absence of full-fledged testing frameworks for hybrid apps, such as those provided by native apps IDEs like Eclipse and Android Studio
  63. 63. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Results – initial download size 6,586 kb4,625 kb In line with the average size of Android apps sizea = file size in kilobytes of the app APK file
  64. 64. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Data extraction Classified apps (hybrid vs native) Hybrid apps classifiers Reviews analyzer top-500 most popular free apps for each category of the Google Play Store ~11k app binaries 50 pages (~255) of reviews for each app ~3M app reviews apps scores Apps and reviews mining perceived value: 0.5 users sentiment: 0.6 #reviews: 243 performance: 0.6 bugginess: 0.1 size: 3,456 kb Apache Cordova (258) and Appcelerator Titanium (116) Developers reuse JavaScript frameworks for the desktop web End users value hybrid and native apps similarly Hybrid may be good for data-intensive apps, whereas it performs poorly when dealing with low-level, platform-specific features Reviews analysis Stopwords removal manually performed by 2 domain experts Single review Single review score polaritypos : 0.8 performancepos : 0.6 polarityneg : 0.1 performanceneg : 0.05 bugginess: 0.2 300 random reviews Keywords extraction Relevant keywords Lemmatization Tf-idf based vectors similarity computation
  65. 65. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Wrap up
  66. 66. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM 7a. HTTP requests as part of the scenario Mobile device Server Measurement infrastructure Orchestration script Monkey runner Trepn profiler Chrome Monkeyrunner 1. HTTP requests impersonating phone 2. HTTP responses (recorded) 9. save collected data 3. Start experiment run 5. start 6. start 7b. HTTP responses (possibly altered) 8. collect data Fiddler proxy Hosted PWA ADB OS 4. start scenario run Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Reviews analysis Stopwords removal manually performed by 2 domain experts Single review Single review score polaritypos: 0.8 performancepos: 0.6 polarityneg: 0.1 performanceneg: 0.05 bugginess: 0.2 300 random reviews Keywords extraction Relevant keywords Lemmatization Tf-idf based vectors similarity computation Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Open card sorting – 2 people performed the task, a third one checked the results 1st phase commits tagged based on keywords e.g., read from file system, swipe lag 2nd phase commits clustered into meaningful, labeled groups Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM BROWSER <html> <head> <script src=” ...” /> </head> <body> ... NATIVE WRAPPER <html> <head> <script src=” ...” /> </head> <body> ... PLATFORM APIs NATIVE APP 01010101010101101010 1010101011011010 010101010101011101 010101010101011010 PLATFORM APIs Native Web Hybrid Development strategies Progressive BROWSER <html> <head> <script src=” ...” /> </head> <body> ... Service workers
  67. 67. Ivano Malavolta VRIJE UNIVERSITEIT AMSTERDAM Contact Ivano Malavolta | Vrije Universiteit Amsterdam iivanoo i.malavolta@vu.nl www.ivanomalavolta.com A few of the contents of this presentation are from co-authors’ slides

×