Augmented Smartphone

Augmented-smartphone
using the Android device beyond the display to
interact with the surroundings

DroidCon Italy - Turin - February 2014 - Stefano Sanna - Open Reply

Agenda
•

The company, the team and the speaker 

•

Forget the augmented-reality!

•

Domains: when, where, what 

•

Conclusions

Augmented-Smartphone - Stefano Sanna - DroidCon IT 2014 - Turin

The company
•

Reply is today a leading IT Services Company

•

Founded in 1996 by a group of Italian IT Executives now
operates in Italy, Germany, UK, Benelux, USA and Brasil.

•

+4000 employees

•

+400M€ revenues 2013/Q3 (+12% 2012/Q3) 

•

Open Reply is the company of Reply Group focused on
open source software, multichannel web solutions and
mobile applications

Open Reply and the Android Lab
•

Based in Rome, Open Reply’s Android Lab is a young team
of over 20 engineers 100% focused on Android
development

•

We are specialised in broadcasting, banking and OS
customisation

•

I’m the head of Android Lab 
My adventure on mobile world began in 1999 (PSION 5MX!)
and I did my journey through Symbian, JME, iOS and
Android. I wrote two books about JME and advanced
Android programming.

Android: numbers, please!
•

+1 Billion activations

•

+1 Million apps in the Play Store

•

2008

+50 Billions apps downloaded

2013


Augmented Reality

Credits: http://alexcalic.com/2010/06/16/the-new-commerce-opportunities-in-the-current-wave-of-innovation/


Augmented smartphone? :-)
•

To date, it seems that just proximity and light sensors are
always used to enhance application behaviour

•

More recently, pervasiveness of location services provide
“geo-referencing” for incoming data (web search, weather,
news) 

•

Is there something more we can do to make our
applications more reactive to the environment? Can we
adapt UI and features according to user context?

•

“Yes, we can!”  
It’s time to “augment the smartphone”

Domains

WHEN

WHAT

WHERE


When
•

Time is a key and quite “absolute” context information
about
•

•

Is day time? Is night time? Is today bank holiday?

Overnight
•

Use a white-on-black color schema for layouts

•

Disable sounds

•

Skip “unless” notiﬁcation


Changing behaviour overnight
You may change the layout according to current time:

•

public void onCreate(Bundle savedInstanceData) {

!

super.onCreate(savedInstanceState); 
if (isNight()) {
setContentView(R.layout.main_night);
} else {
setContentView(R.layout.main_day);
}

}

•

May work, but it is annoying for the developer

•

Could be restricted for key features (e.g., text reader)


Changing behaviour overnight
Could be better to work with themes, using a speciﬁc layout
(night vs day) only when strictly needed:

•

public void onCreate(Bundle savedInstanceData) {
super.onCreate(savedInstanceState); 

!

if (isNight()) {
setTheme(R.style.Theme_Night);
} else {
setTheme(R.style.Theme_Day);
} 

!

setContentView(R.layout.main);
}

•

However, you still have to:
•

Manually select other type of resources (e.g., constants, dimensions…)

•

Decide what “isNight()” means…

Let’s Android work for us!
•

Android offers a system-level support for automatically
switching app behaviour according to time

•

The UIMode API let applications react to device plugged on
desktop stand or car dock or app running on a television

UiModeManager uiManager = (UiModeManager)  
context.getSystemService(Context.UI_MODE_SERVICE);
int uiMode = uiManager.getCurrentModeType();

!
// modes: NORMAL, DESK, CAR, APPLIANCE
if (uiManager.getNightMode() == UiModeManager.MODE_NIGHT_YES) {
// change behaviour
}


What
•

Quite hard to determine without explicit user input
•

•

Is he at work? Is he busy? Is he driving a car?

Current your activity has direct impact on
•

Can the app throw notiﬁcations?

•

Are sounds permitted?

•

May the user be disturbed or distracted?


User’s activity recognition
•

Google’s Location API provides an optimised (low-power)
service that periodically detects user’s activity and notiﬁes
updates to registered applications

•

The “DetectedActivity” is given with a certain level of
conﬁdence and can match one the following:
•

Still

•

On foot

•

In vehicle

•

Tilting

•

Unknown

How it works


Initialisation, connection & registration
public class YourActivity implements ConnectionCallbacks, OnConnectionFailedListener {
private ActivityRecognitionClient mRecognitionClient;
private PendingIntent mPendingIntent;

!
protected onCreate(Bundle savedInstanceState) {
mRecognitionClient = new ActivityRecognitionClient(mContext, this, this);
Intent intent = new Intent(mContext, YourIntentService.class);
mPendingIntent = PendingIntent.getService(mContext, 0, intent,
PendingIntent.FLAG_UPDATE_CURRENT);

}

public void onStart() {
// could be elsewhere, driven by user input
mRecognitionClient.connect();
}

public void onConnected(Bundle dataBundle) {
mRecognitionClient.requestActivityUpdates(DETECTION_INTERVAL, mPendingIntent);
mRecognitionClient.disconnect();
}


Event handling
!
public class ActivityRecognitionIntentService extends IntentService {

!
protected void onHandleIntent(Intent intent) {
if (ActivityRecognitionResult.hasResult(intent)) {

!
ActivityRecognitionResult result =
ActivityRecognitionResult.extractResult(intent);

!
DetectedActivity mostProbableActivity = result.getMostProbableActivity(); 

!

int confidence = mostProbableActivity.getConfidence();
int activityType = mostProbableActivity.getType();

!
// now you know what most likely the user is doing!
}
}


Moving deeper
•

Now we know if the user is “in vehicle”, “on foot” or “still” 

•

Can we go further? Can we “guess” if the user is driving his
car or he is a passenger of a public bus? Can we “guess” if
the user can be disturbed when the device is flagged as
“still”? 

•

Probably we can ;-)


Is the user driving his car?
•

Probably yes, if the smartphone has been connected to a
handsfree (not headset!) audio system:

mReceiver = new ConnectionBroadcastReceiver();
IntentFilter filter = new IntentFilter(BluetoothAdapter.ACTION_CONNECTION_STATE_CHANGED);
registerReceiver(mReceiver, filter);

private class ConnectionBroadcastReceiver extends BroadcastReceiver {
public void onReceive(Context context, Intent intent) {
String action = intent.getAction();
if (action.equals(BluetoothAdapter.ACTION_CONNECTION_STATE_CHANGED)) {
Bundle extras = intent.getExtras();
BluetoothDevice device = extras.getParcelable(BluetoothDevice.EXTRA_DEVICE);
int state = extras.getInt(BluetoothAdapter.EXTRA_CONNECTION_STATE);

!
if (state == BluetoothAdapter.STATE_CONNECTED
&& device.getBluetoothClass().getDeviceClass() ==
BluetoothClass.Device.AUDIO_VIDEO_HANDSFREE) {
// here we are!
}


On foot & still: can we speculate more?
•

A device standing horizontally with the screen upwards is
expected to stay on a desktop: the user is close to it,
working a PC o any other activity

•

A device standing horizontally with the back upwards
probably means that the user does not want to be
distracted by any message or signal coming from the
display

•

A device standing vertically is likely to be “dressed” by the
user, in his dress shirt or pants pockets; probably a
vibration is more preferable

Device orientation
Z
•

Device on the desktop,
upwards: normal operation

•

Device on the desktop,
backwards: silent

•

X

Device vertical (Z-axis
orthogonal to gravity):
vibration
Y

Handling device orientation
Don’t forget the
this broadcast
intent is not
dispatched to
your app until the
user launches its
main activity at
least once

Boot (BroadcastReceiver)

Schedule Alarm

YourIntentService

Manage Alarm
(BroadcastReceiver)

CheckOrientationService


Check orientation (1/2)
private SensorManager mSensorManager;
private Sensor mAccelerometer, mMagnetometer;
private float[] mLastAccelerometer = new float[3];
private float[] mLastMagnetometer = new float[3];
private boolean mAccelerometerRead, mMagnetometerRead;
private float[] mRMatrix = new float[9];
private float[] mOrientation = new float[3];

Don’t forget to unregister the listener
when the service is destroyed

protected int startCommand(Intent intent, int flags, int startId) {

!
mSensorManager = (SensorManager) getSystemService(Context.SENSOR_SERVICE);
mAccelerometer = mSensorManager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER);
mMagnetometer = mSensorManager.getDefaultSensor(Sensor.TYPE_MAGNETIC_FIELD);

!
mSensorManager.registerListener(this, mAccelerometer,
SensorManager.SENSOR_DELAY_NORMAL);
mSensorManager.registerListener(this, mMagnetometer,
SensorManager.SENSOR_DELAY_NORMAL);
}


Check orientation (2/2)
public void onSensorChanged(SensorEvent event) {
if (event.sensor == mAccelerometer) {
System.arraycopy(event.values, 0, mLastAccelerometer, 0, event.values.length);
mAccelerometerRead = true;
} else if (event.sensor == mMagnetometer) {
System.arraycopy(event.values, 0, mLastMagnetometer, 0, event.values.length);
mMagnetometerRead = true;
}
if (mLastAccelerometerSet && mLastMagnetometerSet) {
SensorManager.getRotationMatrix(mRMatrix, null, mLastAccelerometer,
mLastMagnetometer);

!
SensorManager.getOrientation(mR, mOrientation);

Now you know the current
device orientation

if (Math.abs(mOrientation[2]) > 2) {
// display down
} else if (Math.abs( mOrientation[2]) < 1) {
// display up
}
stopSelf();
}


Where
•

The core question of mobility: where is the user? 

•

Location drives pre- and post- user input
•

Is position related to action triggered by the user? (e.g., web search)

•

Is position related to something the user will need in minutes?

•

Was previous position critical for the user?


Outdoor: geofence
•

Geofence is a tuple (latitude, longitude, radius) that deﬁnes
a proximity area

•

Thanks to Fused Location Provider, the new Google’s
Location API provides a powerful and power-friendly
framework to get asynchronous notiﬁcation on entering/
exiting to/from arbitrary geofences

•

There is a limit: max 100 geofences per app per device


Initialisation
public class YourActivity implements ConnectionCallbacks, OnConnectionFailedListener,
OnAddGeofencesResultListener {
private LocationClient mLocationClient;
private PendingIntent mPendingIntent;
private Geofence mGeofence;

!
protected onCreate(Bundle savedInstanceState) {
mLocationClient = new LocationClient(mContext, this, this);

!
Intent intent = new Intent(mContext, YourIntentService.class);

!
mPendingIntent = PendingIntent.getService(mContext, 0, intent,
PendingIntent.FLAG_UPDATE_CURRENT);

}

private void createGeofence() {
mGeofence = new Geofence.Builder().setCircularRegion(0, 0, 0) 
.setRequestId(“gf1”)
.setTransitionTypes(Geofence.GEOFENCE_TRANSITION_DWELL)
.setExpirationDuration(Geofence.NEVER_EXPIRE).build();
}


Connection & registration
private void addGeofences() {
// before adding geofences, we have to connect to the client
mLocationClient.connect();
}

public void onConnected(Bundle dataBundle) {

!
mLocationClient.addGeofences(Arrays.asList(new Geofence[] {mGeofence}),
mPendingIntent, this);

!
}

public void onAddGeofencesResult(int statusCode, String[] geofenceRequestIds) {
if (LocationStatusCodes.SUCCESS == statusCode) {
}

// geofences have been registered!!! 

mLocationClient.disconnect();
}


Event handling
public class YourIntentService extends IntentService {

!
protected void onHandleIntent(Intent intent) {
if (!LocationClient.hasError(intent)) {
int transitionType = LocationClient.getGeofenceTransition(intent);

!
if (
(transitionType == Geofence.GEOFENCE_TRANSITION_ENTER)
||
(transitionType == Geofence.GEOFENCE_TRANSITION_EXIT)
) {
List <Geofence> geofences =
LocationClient.getTriggeringGeofences(intent);

!
}
}
}
}


Indoor location & iBeacons
•

Indoor location has been a tricky and complex to
implement, especially from the hardware/infrastructure
perspective

•

The advent of Bluetooth Low  
Energy (BLE) and iBeacons  
model introduced in iOS 7  
make it possible to develop  
and deliver cheap yet  
extremely powerful proximity 
driven services

•

Small (but not so stable…) hardware is available to start
trials and demos (estimote, kontakt…)
Credits: http://www.estimote.com


Bluetooth Low Energy primer
•

Part of Bluetooth 4.0 speciﬁcation devoted to low-power,
low-range, long-lasting M2M communication

•

Provides spontaneous and extremely  
fast advertising

•

Bluetooth Low Energy is natively  
supported by Android 4.3, but it can be  
considered stable only by KitKat 4.4

•

In order to support BLE, a ﬁrmware upgrade 
is not enough: the hardware must provide 
native support for Bluetooth 4.0 Smart Ready

BLE Device Discovery
BluetoothManager manager = (BluetoothManager) getSystemService(Context.BLUETOOTH_SERVICE);
adapter = manager.getAdapter();
boolean started = adapter.startLeScan(mLeScanCallback);

BluetoothAdapter.LeScanCallback mLeScanCallback = new BluetoothAdapter.LeScanCallback() {
@Override
public void onLeScan(final BluetoothDevice device, int rssi, final byte[] scanRecord) {

!
// device: use it for opening connections
// rssi: use it to determine link quality e distance estimation
// scanRecord: use it for connection-less operations

!
}
};


Inside the iBeacon advertising packet
Device type and discoverability (is it a
smartphone or a small sensor?
The iBeacon segment: contains proprietary identiﬁers, a customisable
UUID, a couple of variables and the 2’ complement of TX calibrated power  
(d4=212; 256-212= -44dBm)
02:01:1a 
 

!cmd

Apple ID

Beacon UUID

Major & Minor

1a:ff:4c:00:02:15:b9:40:7f:30:f5:f8:46:6e:af:f9:25:55:6b:57:fe:6d:8b:dd:2c:74:d4
 
 
10:09:42:45:41:43:4f:4e:5f:44:49:5f:50:52:4f:56:41
00:00:00:00:00:00:00:00:00:00:00:00:00:00:00

Friendly name

Device “friendly name”: there is no need to ask for human-readable name,
as in Blueooth Classic, it is “immediately” available

iBeacon and indoor localisation
•

Power loss (aka “path loss”) depends on distance,
frequency and objects between TX and RX
•

Raw: power decreases by the square of distance (signal = 1 / (distance^2))

•

Does not work at 2.4Ghz! :-(

•

Usually a logarithm scale is used (dB for gain/loss, dBm for
power in mW):

•

A good rough formula could be
•

distance = 10 ^ ((27,55 - (20 *LOG10(2400)) - F6)/20)


Conclusion
•

Time, activity and location awareness improve apps ability
to better serve and help the user

•

Android offers a rich set of APIs and features to implement
smart behaviours, enhance usability, save power and time

•

While today “augmenting the smartphone” could be
considered (optional) added value to standard apps/
services, once Android will be much more pervasive
(appliances, automotive, facilities, wearables) this ability will
be a key factor to succeed


Contacts
•

Corporate:
•

•

http://www.gaiareply.com

•

s.sanna AT reply.it

•

•

http://www.reply.eu

info AT reply.it

Personal
•

gerdavax AT gmail.com

•

@gerdavax

•

http://www.gerdavax.it

Augmented Smartphone

More Related Content

Viewers also liked

Similar to Augmented Smartphone

More from Stefano Sanna

Recently uploaded

Augmented Smartphone