JavaOne 2009 BoF Presentation
Mobile application features typical in smartphones can also be implemented on mass-market featurephones using Java ME and Web 2.0 back-end services. In this presentation, we explore the multimodality and rich user interface of a search-driven portal application written in Java ME and broadly ported to mass-market featurephones. Multimodality enables the user to search, browse and discover using familiar activities on a mobile device - saying a phrase, entering text and snapping a photo.
Snippets of Java ME code used to implement voice recording, image capture, location awareness and advanced mapping are presented and analyzed.
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Smartphone Behavior On A Featurephone Budget
1. Smartphone Behavior
on a Featurephone Budget
using Java ME
Gail Rahn Frederick
Mobile Software Architect
Medio Systems
2. What You Will Learn…
> A Definition of Smartphone Behavior
> Developing Smartphone Features in Java ME
MIDlet Demos
Code Samples
> Real-World Advice about Coding and Porting
Smartphone Features for the Mass-Market
Integrating Java ME and Web 2.0
2
3. Agenda
> Introduction
> Smartphone Features in Mass-Market Devices
using Java ME
Searchbox Suggestions from Web 2.0 API
Search by Voice
Maps with Pan and Zoom
Search by Mobile Location
> Audience Participation and Q & A
3
5. Introduction
About the Presenter
> Gail Rahn Frederick
> Mobile Software Architect at Medio Systems
Mobile search, discovery and content aggregation
for mobile operators and publishers
Native Applications and Mobile Web
> My Background in Mobile
Native mobile applications on 100+ device models
Java ME applications on 50+ device models
Mobile web sites targeting 500+ device models
Distributed in North America and Europe
5
6. Introduction
Audience Participation
> Who are you?
Mobile developers, executives, operators?
> This BoF Session is Informal
> Audience Participation is Encouraged!
Share your Java ME Expertise and Industry Views
Discussion and Q&A at the End
6
7. Smartphone behavior on a featurephone budget…
HOW TO BUILD
SMARTPHONE FEATURES IN
JAVA ME
7
8. Smartphone Features in Java ME
Industry Definition of Terms
> Featurephone
Low-cost, mass-market mobile phone
Third-party software isolated to Applications menu
Less integrated with main phone UI and native OS
Java ME: MIDP, CLDC
> Smartphone
Mobile phone with PC-like functionality
Third-party software is a first-class citizen
Native OS is open development environment
Easy(ier) integration of Apps with OS and Web
Java ME: MIDP, CDC (if device includes a KVM)
8
9. Smartphone Features in Java ME
About the Demo Devices
> Mass-Market Java ME Devices
SonyEricsson C905
MIDP 2.1, CLDC 1.1
MSA (JSR-248)
Nokia N96
MIDP 2.1, CLDC 1.1
MSA (JSR-248)
9
10. Smartphone Features in Java ME
About the Application
> Mobile Search and Discovery
Multimodal, Search-Driven UX
Search by text and voice
Surfaces Web, Information and
Consumables
> Ported to Mass-Market Devices
MIDP 2.0+ and CLDC 1.1
> Asynchronous Network Layer
Manages all transactions
Caches recent network requests
10
11. Smartphone Features in Java ME
The Smartphone Features
> Predictive Searchbox
Suggestions from Web 2.0 Service
> Multimodality
Search by Voice
> Maps with Pan, Zoom and Traffic
Search with Mobile Location
11
12. Smartphone Features in Java ME
Search Suggestions
> How to Integrate Search
Suggestions
User enters search query above
some character length threshold
MIDlet requests suggestions
from Web 2.0 Service
Web Service responds with
suggestions in standard JSON
format
MIDlet reads JSON response
and displays suggestions UI
12
13. Search Suggestions
Server Side is Web 2.0 Service in JSON
> Sample Request URI for Search Suggestions
http://example.com/suggest?q=love
> MIME type for JSON responses
application/x-suggestions+json
> JSON response array contains 4 elements
Query string – the requested search term
Array of completions – list of suggested searches
Array of descriptions – short additional information
Array of URLs – Instant answer for suggested
search
13
14. Search Suggestions
Sample Server Response in JSON
> Example of Search Suggestions response in
JSON format for query “love”:
[quot;lovequot;,
[quot;YouTube - Love Songquot;,
quot;Lovebug - Wikipediaquot;],
[quot;www.youtube.comquot;,
quot;en.wikipedia.orgquot;],
[quot;http://www.youtube.com/watch?v=MR5xv3pt7KIquot;,
quot;http://en.wikipedia.org/wiki/Lovebugquot;]
]
14
15. Search Suggestions
Development Approach
> Keep the Smarts in the Web Server
MIDlet displays all data in JSON response
Use HTTP 1.1 keep-alive
Minimize connection setup and teardown in Java ME
> Read Entire Response as JSON Array
Read Response Components as Arrays or Strings
JSONException thrown for Format Problems
15
16. Search Suggestions
JSON in Java ME
> Helpful JSON References
Open source library from json.org
Java ME package is org.json.me
Sun Developer Network tutorial on reading JSON in
Java ME
16
17. Search Suggestions
Sample Java ME Code
// The 'response' variable is the JSON response body as a String.
try {
JSONArray jsonArray = new JSONArray(response);
String original = jsonArray.getString(0);
JSONArray suggestions = jsonArray.getJSONArray(1);
JSONArray descs = jsonArray.getJSONArray(2);
JSONArray urls = jsonArray.getJSONArray(3);
// Get the values for the first suggestion.
String suggestion = suggestions.getString(0);
String desc = descs.getString(0);
String url = urls.getString(0);
}
catch (JSONException ex) {
// JSON Response is not formatted as expected
}
17
18. Smartphone Features in Java ME
Search by Voice
> How to Search by Voice
UI Cues to Initiate Voice Search
Hold SEND and Speak
Key Event Starts Audio Recording
Canvas.keyPressed()
User Speaks into Microphone
Key Event Stops Audio Recording
Canvas.keyReleased()
Audio Recording Uploaded to
Voice-to-Text or Search Service
18
19. Search by Voice
Development Approach
> Requires JSR-135 Mobile Media API with
Audio Capture
> Use a low-bitrate codec
Minimizes byte size of sound recording
> Audio capture in background thread
Progress updates sent periodically to UI thread
> Audio capture is byte array
Audio data uploaded to voice-to-text service
> Voice-to-text service responds with translation
19
20. Search by Voice
Sample Code for Audio Capture, 1 of 2
// Request audio capture with low-bitrate AMR encoding
String playerURL = quot;capture://audio?encoding=amrquot;;
// Instantiate player for audio capture
Player player = Manager.createPlayer(playerURL);
player.realize();
// Get interface for recording audio.
RecordControl recCtrl =
(RecordControl)player.getControl(quot;RecordControlquot;);
// Instantiate byte stream to receive audio data
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
recCtrl.setRecordStream(outStream);
// Start recording audio
recCtrl.startRecord();
this.player.start();
// Notify UI that player started recording.
20
21. Search by Voice
Sample Code for Audio Capture, 2 of 2
// Periodically update UI during audio recording
long startTime = System.currentTimeMillis();
While( …max recording duration unmet and user still recording… ) {
try {
Thread.sleep(PROGRESS_CALLBACK_PERIOD_MS);
}
catch(InterruptedException ex) {}
// ... Update UI for progress of recording
}
// After audio recording completes, release player resources.
recCtrl.commit();
this.player.close();
// Audio data stored in byte array
byte[] audioData = outStream.toByteArray();
21
22. Smartphone Features in Java ME
Maps with Pan and Zoom
> How to Implement Maps
User presses keys to pan, zoom
and switch map styles
Maps are a Grid of Tiles
Each Tile has an X,Y Position
Relative to Map Center
> Caching Network Manager is
Critical for Performance
Sends Multiple Tile Requests in
One Transaction
Caches Recent Tile Responses
22
23. Maps with Pan and Zoom
Development Approach
> Maps are a Grid of Tiles
Map tile is an image
> Each Tile has an X,Y Position Relative to Map
Center
Tile key is (map centroid, map style, tile X, tile Y)
Sent to mapping server to identify requested tile
> Asynchronous and Caching Network Manager
Sends Multiple Tile Requests in One Transaction
Caches Recent Tile Responses
23
24. Maps with Pan and Zoom
Design Approach
Tiles Currently Displayed in MIDlet
Tiles in Network Cache
Tiles to be Requested On Demand
24
25. Smartphone Features in Java ME
Search with Mobile Location
> How to Search with Mobile Location
User interacts with MIDlet
MIDlet background thread registers
for location updates
When location data is available,
MIDlet incorporates into server
request
Location fix in 5 seconds or 15 minutes
Transparent to User
User can disable LBS in Preferences
25
26. Search with Mobile Location
Development Approach
> MIDlet requests location using JSR-179
> Location requests and callbacks in background
> Last known location cached and re-used
Within some time threshold
> Gotchas?
Phones used in inconvenient locations for LBS
Indoors, high floors, etc.
MIDlet must degrade gracefully and eventually stop trying.
26
27. Search with Mobile Location
Sample Java ME Code
LocationProvider m_locProvider;
// Use default criteria for greatest chance of success.
m_locProvider = LocationProvider.getInstance(null);
// Ask for last known location. If provided, may be stale.
Location loc = m_locProvider.getLastKnownLocation();
// Use default timeout for location lookup.
// This call is blocking – call in dedicated thread
loc = m_locProvider.getLocation(-1);
// Method is data source: cell ID, GPS, network-assisted, etc.
int method = loc.getLocationMethod();
// Get lat/lon co-ordinates and incorporate into search query.
double latitude = loc.getQualifiedCoordinates().getLatitude();
double longitude = loc.getQualifiedCoordinates().getLongitude();
27
28. Smartphone Features in Java ME
Request API Permissions in JAD
> Advanced Java ME APIs are Protected
> Request API Permissions in the JAD
> MIDlet Signing Reduces User Permission Dialogs
Operator, Java Verified and Third-Party Signatures
MIDlet-Permissions:
javax.microedition.io.Connector.http,
javax.microedition.media.control.RecordControl,
javax.microedition.pim.ContactList.read,
javax.microedition.io.Connector.file.read,
javax.microedition.io.Connector.file.write
Text formatting added for readability. MIDlet-Permissions is one line in JAD file.
28
29. Smartphone Features in Java ME
Related APIs for Smart Features in Java ME
> JSR-256: Mobile Sensor API
Uses General Connection Framework to access
sensor hardware on a mobile device
Accelerometer, battery charge state, network field
intensity, etc.
> JSR-172: Web Services API
Web service client for Java ME with XML Parser
> Mobile AJAX in Java ME
Sun Technical Paper
Info and Code Samples @ meapplicationdevelopers.dev.java.net
Streaming ATOM parser, expression language, async I/O library
29
30. Smartphone Features in Java ME
Key Points
> MIDlets with Web 2.0 and other smart features are
possible in Java ME on mass-market devices.
MSA and MSA Subset and open source libraries.
> Asynchronous network manager with caching is
critical to supports Web 2.0 functionality.
30
31. Smartphone Features in Java ME
Audience Participation
> How do you define featurephone and
smartphone?
What is the role of Java ME in each ecosystem?
> What are your challenges to implement
smartphone features in Java ME?
Permissions, OEM Support for Optional APIs,
Fragmentation, etc…
31
This talk is a “postcard from the industry” – an example of our use of Java ME to implement features into mass-market mobile devices that are traditionally seen only on smartphones.You will learn our that definition of smartphone behavior is an application that integrates with native phone features and with Web 2.0 services.
We are going to see 4 feature examples with demonstrations and code samples.
Let’s find out a little bit about the audience.If you are a mobile developer, please raise your hand. How about QA? Those who work with operators or publishers?If you have shipped MIDlets in North America, please raise your hand. Europe? Asia? Other parts of the world?Last one – your experience level with Java ME – Beginner? Intermediate? Advanced?
Feature set convergence between low-end and high-end devices in the near future.Especially with emergence of new classes of mobile devices (netbooks, MIDs, game consoles, media players), lines are blurring between featurephones and smartphones.Features I will demonstrate tonight show tightened integration between: Java ME and native phone features (contact list, location), Java ME and the Internet (voice search, web search result suggestions, mapping).
An asynchronous network layer is essential for implementing advanced networkingThe asynchronous network layer uses one queue of all kinds of network transactions, a cache of recent requests and a listener model for notifying when new information has arrived from the network.
Explain concept of PredictionaryPredictionary = user search history (stored in RMS), Medio suggested terms, contacts list on device and web search result suggestions.Some of these features may seem pretty simple from a technological point of view – but their big usability upsides merit their inclusion.The Network Response Cache allows caching of recent network transactions, for re-use,
Do the demonstration and then review the sample code.
Do the demonstration and then review the sample code
Check device specifications and test for supported low-bitratecodecs. No comprehensive industry reference for which codecs are supported on which devices.
Assuming you know that the mobile device supports audio capture and recording, this sample code
Do the demonstration and then review the sample code
This chart describes possible caching behavior of map tiles as the user pans left and right in a map image. Tiles to the left and right of the image persist in the network cache, as well as tiles directly above and below the current map location.Cache design and thresholds depend on device capabilities.Set cache parameters in JAD attributes to allow for easy cache modification.
No demo – uncertain of a quick GPS fix here in Moscone Center.
Use default criteria for greatest chance of success in LocationProvider.getInstance();OK for simple location queryThis is a synchronous example and a starting point to integrate LBS into a background thread.
Of course, many of these advanced techniques are not possible without requesting permission to use protected APIs in Java ME. Here is an example of requesting permissions in the JAD file that are required to run the application.
JSR-229 (mobile payments) is another API for smartphone features but requires integration with external payment providers
Java ME for Featurephones: MIDP, CLDCJava ME for Smartphones: MIDP, CDCBoth demonstration devices use MIDP 2.1 and CLDC 1.1