Smartphone Behavior On A Featurephone Budget


Published on

JavaOne 2009 BoF Presentation

Mobile application features typical in smartphones can also be implemented on mass-market featurephones using Java ME and Web 2.0 back-end services. In this presentation, we explore the multimodality and rich user interface of a search-driven portal application written in Java ME and broadly ported to mass-market featurephones. Multimodality enables the user to search, browse and discover using familiar activities on a mobile device - saying a phrase, entering text and snapping a photo.

Snippets of Java ME code used to implement voice recording, image capture, location awareness and advanced mapping are presented and analyzed.

Published in: Technology, Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • This talk is a “postcard from the industry” – an example of our use of Java ME to implement features into mass-market mobile devices that are traditionally seen only on smartphones.You will learn our that definition of smartphone behavior is an application that integrates with native phone features and with Web 2.0 services.
  • We are going to see 4 feature examples with demonstrations and code samples.
  • Let’s find out a little bit about the audience.If you are a mobile developer, please raise your hand. How about QA? Those who work with operators or publishers?If you have shipped MIDlets in North America, please raise your hand. Europe? Asia? Other parts of the world?Last one – your experience level with Java ME – Beginner? Intermediate? Advanced?
  • Feature set convergence between low-end and high-end devices in the near future.Especially with emergence of new classes of mobile devices (netbooks, MIDs, game consoles, media players), lines are blurring between featurephones and smartphones.Features I will demonstrate tonight show tightened integration between: Java ME and native phone features (contact list, location), Java ME and the Internet (voice search, web search result suggestions, mapping).
  • An asynchronous network layer is essential for implementing advanced networkingThe asynchronous network layer uses one queue of all kinds of network transactions, a cache of recent requests and a listener model for notifying when new information has arrived from the network.
  • Explain concept of PredictionaryPredictionary = user search history (stored in RMS), Medio suggested terms, contacts list on device and web search result suggestions.Some of these features may seem pretty simple from a technological point of view – but their big usability upsides merit their inclusion.The Network Response Cache allows caching of recent network transactions, for re-use,
  • Do the demonstration and then review the sample code.
  • Do the demonstration and then review the sample code
  • Check device specifications and test for supported low-bitratecodecs. No comprehensive industry reference for which codecs are supported on which devices.
  • Assuming you know that the mobile device supports audio capture and recording, this sample code
  • Do the demonstration and then review the sample code
  • This chart describes possible caching behavior of map tiles as the user pans left and right in a map image. Tiles to the left and right of the image persist in the network cache, as well as tiles directly above and below the current map location.Cache design and thresholds depend on device capabilities.Set cache parameters in JAD attributes to allow for easy cache modification.
  • No demo – uncertain of a quick GPS fix here in Moscone Center.
  • Use default criteria for greatest chance of success in LocationProvider.getInstance();OK for simple location queryThis is a synchronous example and a starting point to integrate LBS into a background thread.
  • Of course, many of these advanced techniques are not possible without requesting permission to use protected APIs in Java ME. Here is an example of requesting permissions in the JAD file that are required to run the application.
  • JSR-229 (mobile payments) is another API for smartphone features but requires integration with external payment providers
  • Java ME for Featurephones: MIDP, CLDCJava ME for Smartphones: MIDP, CDCBoth demonstration devices use MIDP 2.1 and CLDC 1.1
  • Smartphone Behavior On A Featurephone Budget

    1. Smartphone Behavior on a Featurephone Budget using Java ME Gail Rahn Frederick Mobile Software Architect Medio Systems
    2. What You Will Learn… > A Definition of Smartphone Behavior > Developing Smartphone Features in Java ME  MIDlet Demos  Code Samples > Real-World Advice about Coding and Porting  Smartphone Features for the Mass-Market  Integrating Java ME and Web 2.0 2
    3. Agenda > Introduction > Smartphone Features in Mass-Market Devices using Java ME  Searchbox Suggestions from Web 2.0 API  Search by Voice  Maps with Pan and Zoom  Search by Mobile Location > Audience Participation and Q & A 3
    4. About the Presenter and the Audience… INTRODUCTION 4
    5. Introduction About the Presenter > Gail Rahn Frederick > Mobile Software Architect at Medio Systems  Mobile search, discovery and content aggregation for mobile operators and publishers  Native Applications and Mobile Web > My Background in Mobile  Native mobile applications on 100+ device models  Java ME applications on 50+ device models  Mobile web sites targeting 500+ device models  Distributed in North America and Europe 5
    6. Introduction Audience Participation > Who are you?  Mobile developers, executives, operators? > This BoF Session is Informal > Audience Participation is Encouraged!  Share your Java ME Expertise and Industry Views  Discussion and Q&A at the End 6
    7. Smartphone behavior on a featurephone budget… HOW TO BUILD SMARTPHONE FEATURES IN JAVA ME 7
    8. Smartphone Features in Java ME Industry Definition of Terms > Featurephone  Low-cost, mass-market mobile phone  Third-party software isolated to Applications menu  Less integrated with main phone UI and native OS  Java ME: MIDP, CLDC > Smartphone  Mobile phone with PC-like functionality  Third-party software is a first-class citizen  Native OS is open development environment  Easy(ier) integration of Apps with OS and Web  Java ME: MIDP, CDC (if device includes a KVM) 8
    9. Smartphone Features in Java ME About the Demo Devices > Mass-Market Java ME Devices  SonyEricsson C905  MIDP 2.1, CLDC 1.1  MSA (JSR-248)  Nokia N96  MIDP 2.1, CLDC 1.1  MSA (JSR-248) 9
    10. Smartphone Features in Java ME About the Application > Mobile Search and Discovery  Multimodal, Search-Driven UX  Search by text and voice  Surfaces Web, Information and Consumables > Ported to Mass-Market Devices  MIDP 2.0+ and CLDC 1.1 > Asynchronous Network Layer  Manages all transactions  Caches recent network requests 10
    11. Smartphone Features in Java ME The Smartphone Features > Predictive Searchbox  Suggestions from Web 2.0 Service > Multimodality  Search by Voice > Maps with Pan, Zoom and Traffic  Search with Mobile Location 11
    12. Smartphone Features in Java ME Search Suggestions > How to Integrate Search Suggestions  User enters search query above some character length threshold  MIDlet requests suggestions from Web 2.0 Service  Web Service responds with suggestions in standard JSON format  MIDlet reads JSON response and displays suggestions UI 12
    13. Search Suggestions Server Side is Web 2.0 Service in JSON > Sample Request URI for Search Suggestions  > MIME type for JSON responses  application/x-suggestions+json > JSON response array contains 4 elements  Query string – the requested search term  Array of completions – list of suggested searches  Array of descriptions – short additional information  Array of URLs – Instant answer for suggested search 13
    14. Search Suggestions Sample Server Response in JSON > Example of Search Suggestions response in JSON format for query “love”: [quot;lovequot;, [quot;YouTube - Love Songquot;, quot;Lovebug - Wikipediaquot;], [quot;;, quot;en.wikipedia.orgquot;], [quot;;, quot;;] ] 14
    15. Search Suggestions Development Approach > Keep the Smarts in the Web Server  MIDlet displays all data in JSON response  Use HTTP 1.1 keep-alive  Minimize connection setup and teardown in Java ME > Read Entire Response as JSON Array  Read Response Components as Arrays or Strings  JSONException thrown for Format Problems 15
    16. Search Suggestions JSON in Java ME > Helpful JSON References  Open source library from  Java ME package is  Sun Developer Network tutorial on reading JSON in Java ME 16
    17. Search Suggestions Sample Java ME Code // The 'response' variable is the JSON response body as a String. try { JSONArray jsonArray = new JSONArray(response); String original = jsonArray.getString(0); JSONArray suggestions = jsonArray.getJSONArray(1); JSONArray descs = jsonArray.getJSONArray(2); JSONArray urls = jsonArray.getJSONArray(3); // Get the values for the first suggestion. String suggestion = suggestions.getString(0); String desc = descs.getString(0); String url = urls.getString(0); } catch (JSONException ex) { // JSON Response is not formatted as expected } 17
    18. Smartphone Features in Java ME Search by Voice > How to Search by Voice  UI Cues to Initiate Voice Search  Hold SEND and Speak  Key Event Starts Audio Recording  Canvas.keyPressed()  User Speaks into Microphone  Key Event Stops Audio Recording  Canvas.keyReleased()  Audio Recording Uploaded to Voice-to-Text or Search Service 18
    19. Search by Voice Development Approach > Requires JSR-135 Mobile Media API with Audio Capture > Use a low-bitrate codec  Minimizes byte size of sound recording > Audio capture in background thread  Progress updates sent periodically to UI thread > Audio capture is byte array  Audio data uploaded to voice-to-text service > Voice-to-text service responds with translation 19
    20. Search by Voice Sample Code for Audio Capture, 1 of 2 // Request audio capture with low-bitrate AMR encoding String playerURL = quot;capture://audio?encoding=amrquot;; // Instantiate player for audio capture Player player = Manager.createPlayer(playerURL); player.realize(); // Get interface for recording audio. RecordControl recCtrl = (RecordControl)player.getControl(quot;RecordControlquot;); // Instantiate byte stream to receive audio data ByteArrayOutputStream outStream = new ByteArrayOutputStream(); recCtrl.setRecordStream(outStream); // Start recording audio recCtrl.startRecord(); this.player.start(); // Notify UI that player started recording. 20
    21. Search by Voice Sample Code for Audio Capture, 2 of 2 // Periodically update UI during audio recording long startTime = System.currentTimeMillis(); While( …max recording duration unmet and user still recording… ) { try { Thread.sleep(PROGRESS_CALLBACK_PERIOD_MS); } catch(InterruptedException ex) {} // ... Update UI for progress of recording } // After audio recording completes, release player resources. recCtrl.commit(); this.player.close(); // Audio data stored in byte array byte[] audioData = outStream.toByteArray(); 21
    22. Smartphone Features in Java ME Maps with Pan and Zoom > How to Implement Maps  User presses keys to pan, zoom and switch map styles  Maps are a Grid of Tiles  Each Tile has an X,Y Position Relative to Map Center > Caching Network Manager is Critical for Performance  Sends Multiple Tile Requests in One Transaction  Caches Recent Tile Responses 22
    23. Maps with Pan and Zoom Development Approach > Maps are a Grid of Tiles  Map tile is an image > Each Tile has an X,Y Position Relative to Map Center  Tile key is (map centroid, map style, tile X, tile Y)  Sent to mapping server to identify requested tile > Asynchronous and Caching Network Manager  Sends Multiple Tile Requests in One Transaction  Caches Recent Tile Responses 23
    24. Maps with Pan and Zoom Design Approach Tiles Currently Displayed in MIDlet Tiles in Network Cache Tiles to be Requested On Demand 24
    25. Smartphone Features in Java ME Search with Mobile Location > How to Search with Mobile Location  User interacts with MIDlet  MIDlet background thread registers for location updates  When location data is available, MIDlet incorporates into server request  Location fix in 5 seconds or 15 minutes  Transparent to User  User can disable LBS in Preferences 25
    26. Search with Mobile Location Development Approach > MIDlet requests location using JSR-179 > Location requests and callbacks in background > Last known location cached and re-used  Within some time threshold > Gotchas?  Phones used in inconvenient locations for LBS  Indoors, high floors, etc.  MIDlet must degrade gracefully and eventually stop trying. 26
    27. Search with Mobile Location Sample Java ME Code LocationProvider m_locProvider; // Use default criteria for greatest chance of success. m_locProvider = LocationProvider.getInstance(null); // Ask for last known location. If provided, may be stale. Location loc = m_locProvider.getLastKnownLocation(); // Use default timeout for location lookup. // This call is blocking – call in dedicated thread loc = m_locProvider.getLocation(-1); // Method is data source: cell ID, GPS, network-assisted, etc. int method = loc.getLocationMethod(); // Get lat/lon co-ordinates and incorporate into search query. double latitude = loc.getQualifiedCoordinates().getLatitude(); double longitude = loc.getQualifiedCoordinates().getLongitude(); 27
    28. Smartphone Features in Java ME Request API Permissions in JAD > Advanced Java ME APIs are Protected > Request API Permissions in the JAD > MIDlet Signing Reduces User Permission Dialogs  Operator, Java Verified and Third-Party Signatures MIDlet-Permissions:,,,, Text formatting added for readability. MIDlet-Permissions is one line in JAD file. 28
    29. Smartphone Features in Java ME Related APIs for Smart Features in Java ME > JSR-256: Mobile Sensor API  Uses General Connection Framework to access sensor hardware on a mobile device  Accelerometer, battery charge state, network field intensity, etc. > JSR-172: Web Services API  Web service client for Java ME with XML Parser > Mobile AJAX in Java ME  Sun Technical Paper  Info and Code Samples @  Streaming ATOM parser, expression language, async I/O library 29
    30. Smartphone Features in Java ME Key Points > MIDlets with Web 2.0 and other smart features are possible in Java ME on mass-market devices.  MSA and MSA Subset and open source libraries. > Asynchronous network manager with caching is critical to supports Web 2.0 functionality. 30
    31. Smartphone Features in Java ME Audience Participation > How do you define featurephone and smartphone?  What is the role of Java ME in each ecosystem? > What are your challenges to implement smartphone features in Java ME?  Permissions, OEM Support for Optional APIs, Fragmentation, etc… 31
    32. Gail Rahn Frederick Medio Systems 32