(Mobile) Device APIs          James Pearce         @jamespearce     http://tripleodeon.com
What are Device APIs? “Client-side APIs that enable the development of Web Applications that interact with device hardware...
Why are they important?• They’re not...    ...if you’re happy with a web of documents• They are...    ... if you want the ...
HistoryCurrent Work Use Cases
WTAI Make Call<a	  href="wtai://wp/mc;16505551234">	  	  Call	  us</a><a	  href="call.wmls#makeCall()">	  	  Call	  us</a>...
WTAI Call Control<a	  href="calls.wmls#setupCall()">Call	  us!</a><a	  href="calls.wmls#acceptCall()">Brring!</a><a	  href...
WTAI Messaging//	  sms.wmlsextern	  function	  sendMsg()	  {	  	  WTANetText.send("16505551234",	  "WML	  rocks");}extern	...
WTAI Contacts//	  contacts.wmlsextern	  function	  addFriend(i,	  num,	  name)	  {	  	  WTAPhoneBook.write(i,	  num,	  nam...
I may be laboring the point But that was 1998... ... and you still can’t do any of this with a contemporary mobile browser
BONDI“enables web based content to access nativedevice capability, intermediated through arobust, but flexible security fra...
Messagingvar	  sms	  =	  bondi.messaging.createSMS({	  	  to:	  ["16505551234"],	  	  body:	  "Home	  soon"});var	  mms	  ...
Messagingbondi.messaging.sendSMS(	  	  function	  ()	  {},	  //	  success	  	  function	  ()	  {},	  //	  error	  	  sms,	...
Messagingbondi.messaging.subscribeToSMS(	  	  function	  ()	  {},	  	  	  	  	  	  	  	  	  	  	  //	  success	  	  functi...
Contactsbondi.pim.contact.getAddressBooks(	  	  function(books)	  {	  //	  success	  	  	  	  var	  contact	  =	  books[0]...
Telephonybondi.telephonyLog.findLogEntries(	  	  function	  ()	  {},	  //	  success	  	  function	  ()	  {},	  //	  error,...
App Launcherbondi.applauncher.launchApplication(	  	  function	  ()	  {},	  //	  success	  	  function	  ()	  {},	  //	  e...
On one hand...• Security-conscious architecture• Comprehensive API support• Consistent API patterns• Conformance specificat...
On the other...
bondi.*deviceapis.*
WAC 2.0• Accelerometer        • File system• Calendar             • Geolocation• Camera               • Messaging• Contact...
Which is all awesome Except...
Device APIs Working Group “to create client-side APIs that enable the development of Web Applications and Web Widgets that...
W3C Device APIs• Media Capture(HTML)                        • Vibration• Media Capture(gUM)                        • Calen...
Media Capture (HTML)<input	  type="file"	  accept="image/*"	  /><!-­‐-­‐	  	  	  The	  accept	  attribute	  is	  a	  hint....
Brad Lassey, September 2008
Media Capture (HTML)<input	  type="file"	  accept="image/*"	  	  capture="camera"/><!-­‐-­‐	  	  	  Again,	  a	  hint.	  V...
Media Capture (HTML)<input	  type="file"	  accept="image/*"	  	  capture="camera"	  id="photo"/>var	  photo	  =	  document...
MediaFileDatainterface	  MediaFileData	  {	  	  	  	  attribute	  DOMString	  	  	  	  	  codecs;	  	  	  	  attribute	  u...
Media capture (gUM)navigator.getUserMedia(	  	  {	  	  	  	  audio:	  true,	  	  	  	  video:	  false	  	  },	  	  functio...
LocalMediaStreamvar	  recorder	  =	  stream.record();//	  recorder	  is	  a	  MediaStreamRecorderrecorder.getRecordedData(...
Messagingnavigator.sendMessage(	  	  "sms:16505551234?"	  +	  	  	  	  "body=JavaScript%20says%20hi",	  	  	  [],	  	  	  ...
Sensorsnavigator.findSensors().onsuccess	  =	  function()	  {	  	  this.result.forEach(function	  (sensor)	  {	  	  	  	  ...
Sensors• Temperature, ºC• AmbientLight, Lux• AmbientNoise, dbA• MagneticField, uTesla• Proximity, cm• AtmPressure, kP• Rel...
Contactsnavigator.contacts.find(	  	  [	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  //	  fields	  ...
Contactinterface	  Contact	  {	  	  	  	  readonly	  attribute	  DOMString	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  id;...
Cool, but... “The ability to add and update contact information is not a function of the API provided in this specification.”
Holy Contacts Editing WTFvar	  vcard	  =	  	  	  BEGIN:VCARDrn	  +	  	  VERSION:3.0rn	  +	  	  UID:	  +	  contact.id	  +	 ...
WebAPI “We are aiming at providing all the necessary APIs to build a basic HTML5 phone experience within the next 3-6 mont...
WebAPI• Telephony   • Embedded browser• Messaging   • Battery• Contacts    • Device Status• Camera      • Settings• Networ...
Hopefully W3C++• “Our proposal might or might not look like  it for the moment”
Telephony!var	  tcall	  =	  	  navigator.telephony.dial("16505551234");tcall.readyState;	  	  //	  dialing,	  ringing,	  b...
Contacts “We need read/write”
USBvar	  usb	  =	  new	  MozUSBManager();usb.addEventListener("attachdevice",	  	  	  function	  ()	  {});usb.addEventList...
We deserve Device APIs...• As forward-looking as WTAI• As well-structured as BONDI• As comprehensive as WAC• As respectabl...
And we’ve got...
PhoneGap• Accelerometer   • Events• Camera          • File• Capture         • Geolocation• Compass         • Media• Connec...
Use Cases
Useless Demos                https://github.com/jamesgpearce/compios5
Fixing layout bugs, FMLwindow.addEventListener("deviceorientation",	  	  function	  (e)	  {	  	  	  	  orientation	  =	  M...
Camera• Profile pictures• Review sites• Photo sharing• Recognition • Bar-codes • Text • Faces
Filters!$(img).vintage({	  	  vignette:	  {	  	  	  	  black:	  0.8,	  	  	  	  white:	  0.2	  	  },	  	  noise:	  20,	  	...
Augmented reality?                     http://fhtr.org/JSARToolKit/
Contacts & calendaring• Add business details to address book• Web-based messaging• Facial recognition again...• Presence• ...
Messaging• Sharing• Notifications• Offline transport fallback• Frictionless web sign-on
Expect more
I still want... • Audio processing • Bluetooth • RFID & NFC • Both cameras • Memory & power • Native recognition
window.addEventListener(	  	  appointmentdue,	  	  function(appt)	  {	  	  	  	  if	  (device.near(WORK))	  {	  	  	  	  	...
James Pearce    @jamespearcehttp://tripleodeon.com
Mobile Device APIs
Mobile Device APIs
Mobile Device APIs
Mobile Device APIs
Mobile Device APIs
Mobile Device APIs
Mobile Device APIs
Mobile Device APIs
Mobile Device APIs
Upcoming SlideShare
Loading in...5
×

Mobile Device APIs

44,203

Published on

Thoughts on the past, present and future of device APIs for (mostly mobile) web developers. NB: is probably out of date already.

Published in: Technology, Business
2 Comments
32 Likes
Statistics
Notes
No Downloads
Views
Total Views
44,203
On Slideshare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
177
Comments
2
Likes
32
Embeds 0
No embeds

No notes for slide

Mobile Device APIs

  1. 1. (Mobile) Device APIs James Pearce @jamespearce http://tripleodeon.com
  2. 2. What are Device APIs? “Client-side APIs that enable the development of Web Applications that interact with device hardware” ...and other capabilities outside of the browser’s traditional remit
  3. 3. Why are they important?• They’re not... ...if you’re happy with a web of documents• They are... ... if you want the the web to be competitive with native app platforms
  4. 4. HistoryCurrent Work Use Cases
  5. 5. WTAI Make Call<a  href="wtai://wp/mc;16505551234">    Call  us</a><a  href="call.wmls#makeCall()">    Call  us</a>//  call.wmlsextern  function  makeCall()  {    WTAPublic.makeCall("16505551234");} http://www.wapforum.org/what/technical/wtai-30-apr-98.pdf
  6. 6. WTAI Call Control<a  href="calls.wmls#setupCall()">Call  us!</a><a  href="calls.wmls#acceptCall()">Brring!</a><a  href="wtai://cc/sd;1*2*3456">Enter  code</a>//  calls.wmlsextern  function  setupCall()  {    WTACallCont.setup("16505551234",  1);}extern  function  acceptCall()  {    WTACallCont.accept("1",  1);}
  7. 7. WTAI Messaging//  sms.wmlsextern  function  sendMsg()  {    WTANetText.send("16505551234",  "WML  rocks");}extern  function  readMsg(i)  {    var  sms  =  WTANetText.read(i);    return  WTANetText.getFieldValue(        sms,  "body"    );}
  8. 8. WTAI Contacts//  contacts.wmlsextern  function  addFriend(i,  num,  name)  {    WTAPhoneBook.write(i,  num,  name);}//  return  structs  of  contact  detailsextern  function  getFriendById(i)  {    return  WTAPhoneBook.read("i",  i);}extern  function  getFriendByName(name)  {    return  WTAPhoneBook.read("t",  name);}
  9. 9. I may be laboring the point But that was 1998... ... and you still can’t do any of this with a contemporary mobile browser
  10. 10. BONDI“enables web based content to access nativedevice capability, intermediated through arobust, but flexible security framework” http://www.omtp.org/1.11
  11. 11. Messagingvar  sms  =  bondi.messaging.createSMS({    to:  ["16505551234"],    body:  "Home  soon"});var  mms  =  bondi.messaging.createMMS({    to:  ["16505551234"],    subject:  "I  will  be...",    body:  "...home  soon"});
  12. 12. Messagingbondi.messaging.sendSMS(    function  ()  {},  //  success    function  ()  {},  //  error    sms,  true);bondi.messaging.sendMMS(    function  ()  {},  //  success    function  ()  {},  //  error    mms,  true);
  13. 13. Messagingbondi.messaging.subscribeToSMS(    function  ()  {},                      //  success    function  ()  {},                      //  error    function  ()  {                          //  listener        alert("They  replied!");    },  {                                      from:  "16505551234"          //  filter    },    true                                            //  exclusive);
  14. 14. Contactsbondi.pim.contact.getAddressBooks(    function(books)  {  //  success        var  contact  =  books[0].createContact({            name:  James  Pearce,            emails:  [{email:j@tripleodeon.com}],            phoneNumbers:  [{number:16505551234}]        });    },      function  ()  {}        //  error);
  15. 15. Telephonybondi.telephonyLog.findLogEntries(    function  ()  {},  //  success    function  ()  {},  //  error,    [        bondi.telephonyLog.MISSED_CALLS_FOLDER,        bondi.telephonyLog.RECEIVED_CALLS_FOLDER    ],    {        phoneNumber:  "16505551234"    }); * Yes, it’s true: you can’t actually make or receive calls
  16. 16. App Launcherbondi.applauncher.launchApplication(    function  ()  {},  //  success    function  ()  {},  //  error    "tel:16505551234");//  mailto:...//  sms:...//  file:...
  17. 17. On one hand...• Security-conscious architecture• Comprehensive API support• Consistent API patterns• Conformance specifications• Reasonable test coverage http://www.omtp.org/1.11/apis
  18. 18. On the other...
  19. 19. bondi.*deviceapis.*
  20. 20. WAC 2.0• Accelerometer • File system• Calendar • Geolocation• Camera • Messaging• Contacts • Orientation• Core • Tasks• Device interaction • Viewport• Device status • Webview http://specs.wacapps.net/
  21. 21. Which is all awesome Except...
  22. 22. Device APIs Working Group “to create client-side APIs that enable the development of Web Applications and Web Widgets that interact with devices services such as Calendar, Contacts, Camera, etc” Joint efforts with WebRTC Working Group, WebApps Working Group http://www.w3.org/2009/dap/
  23. 23. W3C Device APIs• Media Capture(HTML) • Vibration• Media Capture(gUM) • Calendar• Contact • Permissions• Messaging • Menu• Battery Status • Gallery• Network Information • System info• Sensor • Media Stream
  24. 24. Media Capture (HTML)<input  type="file"  accept="image/*"  /><!-­‐-­‐      The  accept  attribute  is  a  hint.    Specification  suggests  that  it  can  be    used  to  offer  alternative  picker  UIs.-­‐-­‐> Public working draft, http://www.w3.org/TR/2011/WD-html-media-capture-20110414
  25. 25. Brad Lassey, September 2008
  26. 26. Media Capture (HTML)<input  type="file"  accept="image/*"    capture="camera"/><!-­‐-­‐      Again,  a  hint.  Valid  options  are:        camera,  camcorder,  microphone,  filesystem-­‐-­‐>
  27. 27. Media Capture (HTML)<input  type="file"  accept="image/*"    capture="camera"  id="photo"/>var  photo  =  document.getElementById(photo);photo.files[0].getFormatData(    function  (data)  {},  //  success    function  ()  {}            //  error);//  file  is  a  MediaFile,  extending  File//  data  is  a  MediaFileData
  28. 28. MediaFileDatainterface  MediaFileData  {        attribute  DOMString          codecs;        attribute  unsigned  long  bitrate;        attribute  unsigned  long  height;        attribute  unsigned  long  width;        attribute  float                  duration;}; ...seems to be under discussion
  29. 29. Media capture (gUM)navigator.getUserMedia(    {        audio:  true,        video:  false    },    function  (stream)  {},  //  success    function  ()  {}                //  error);//  stream  is  a  LocalMediaStream Editors draft, http://dev.w3.org/2011/webrtc/editor/getusermedia.html
  30. 30. LocalMediaStreamvar  recorder  =  stream.record();//  recorder  is  a  MediaStreamRecorderrecorder.getRecordedData(    function  (file)  {}    //  success);//  file  is  a  FileURL.createObjectURL(stream);//  a  Blob  URI  for,  say  <video  src=...  />    
  31. 31. Messagingnavigator.sendMessage(    "sms:16505551234?"  +        "body=JavaScript%20says%20hi",      [],                          //  attachments  as  File  array    function  ()  {},  //  success    function  ()  {}    //  error);//  also  mms:  and  mailto://  oh...  were  you  looking  for  receipt? Editors draft, http://dev.w3.org/2009/dap/messaging/
  32. 32. Sensorsnavigator.findSensors().onsuccess  =  function()  {    this.result.forEach(function  (sensor)  {        //  ...    });};var  sensor  =  new  SensorConnection(Temperature);sensor.addEventListener(sensordata,  function(e)  {    if(e.data  >  20.0)  {        window.console.log(Hawt);    }});sensor.startWatch({  interval:  2000  }); Editors draft, http://dev.w3.org/2009/dap/system-info/Sensors.html
  33. 33. Sensors• Temperature, ºC• AmbientLight, Lux• AmbientNoise, dbA• MagneticField, uTesla• Proximity, cm• AtmPressure, kP• RelHumidity, %
  34. 34. Contactsnavigator.contacts.find(    [                                              //  fields        name,          emails    ],      function  (contacts)  {},  //  success    function  ()  {},                  //  error    {                                              //  find  options          filter:  James,            multiple:  true    }); Editors draft, http://w3c-test.org/dap/contacts/
  35. 35. Contactinterface  Contact  {        readonly  attribute  DOMString                              id;                          attribute  DOMString?                            displayName;                          attribute  ContactName?                        name;                          attribute  DOMString?                            nickname;                          attribute  ContactField[]?                  phoneNumbers;                          attribute  ContactField[]?                  emails;                          attribute  ContactAddress[]?              addresses;                          attribute  ContactField[]?                  ims;                          attribute  ContactOrganization[]?    organizations;                          attribute  Date?                                      birthday;                          attribute  DOMString?                            note;                          attribute  ContactField[]?                  photos;                          attribute  DOMString[]?                        categories;                          attribute  ContactField[]?                  urls;};
  36. 36. Cool, but... “The ability to add and update contact information is not a function of the API provided in this specification.”
  37. 37. Holy Contacts Editing WTFvar  vcard  =      BEGIN:VCARDrn  +    VERSION:3.0rn  +    UID:  +  contact.id  +  rn  +          N:Pearce;Jamesrn  +    FN:James  Pearcern  +    END:VCARD;document.getElementById(vcard).href  =    data:text/x-­‐vcard;charset=utf-­‐8,  +    encodeURIComponent(vcard);  
  38. 38. WebAPI “We are aiming at providing all the necessary APIs to build a basic HTML5 phone experience within the next 3-6 months” https://wiki.mozilla.org/WebAPI
  39. 39. WebAPI• Telephony • Embedded browser• Messaging • Battery• Contacts • Device Status• Camera • Settings• Network • Mouse Lock• USB • Vibrator
  40. 40. Hopefully W3C++• “Our proposal might or might not look like it for the moment”
  41. 41. Telephony!var  tcall  =    navigator.telephony.dial("16505551234");tcall.readyState;    //  dialing,  ringing,  busy,  connecting,    //  connected,  disconnecting,  disconnected,      //  incomingtcall.onconnected  =  function  ()  {};    //  onbusy,  ondisconnected  etctcall2.answer(); http://www.youtube.com/watch?v=RuIQskGD3u0#t=19s
  42. 42. Contacts “We need read/write”
  43. 43. USBvar  usb  =  new  MozUSBManager();usb.addEventListener("attachdevice",      function  ()  {});usb.addEventListener("detachdevice",      function  ()  {});usb.claimDevice({    deviceClass:  ...,    deviceSubClass:  ...});
  44. 44. We deserve Device APIs...• As forward-looking as WTAI• As well-structured as BONDI• As comprehensive as WAC• As respectable as W3C DAP• As agile as WebAPI
  45. 45. And we’ve got...
  46. 46. PhoneGap• Accelerometer • Events• Camera • File• Capture • Geolocation• Compass • Media• Connection • Notification• Contacts • Storage• Device
  47. 47. Use Cases
  48. 48. Useless Demos https://github.com/jamesgpearce/compios5
  49. 49. Fixing layout bugs, FMLwindow.addEventListener("deviceorientation",    function  (e)  {        orientation  =  Math.abs(w.orientation);        rotation  =  Math.abs(e.gamma);        if  (rotation  >  8  &&  orientation  ===  0)  {            if  (enabled)  {                disableZoom();            }  else  {                restoreZoom();            }        }    },  false); http://vimeo.com/adactio/iosbug https://github.com/scottjehl/iOS-Orientationchange-Fix
  50. 50. Camera• Profile pictures• Review sites• Photo sharing• Recognition • Bar-codes • Text • Faces
  51. 51. Filters!$(img).vintage({    vignette:  {        black:  0.8,        white:  0.2    },    noise:  20,    screen:  {        red:  12,        ...    },    desaturate:  0.05}); https://github.com/rendro/vintageJS
  52. 52. Augmented reality? http://fhtr.org/JSARToolKit/
  53. 53. Contacts & calendaring• Add business details to address book• Web-based messaging• Facial recognition again...• Presence• Context-sensitivity of content
  54. 54. Messaging• Sharing• Notifications• Offline transport fallback• Frictionless web sign-on
  55. 55. Expect more
  56. 56. I still want... • Audio processing • Bluetooth • RFID & NFC • Both cameras • Memory & power • Native recognition
  57. 57. window.addEventListener(    appointmentdue,    function(appt)  {        if  (device.near(WORK))  {            siri.remind(                contacts.get(Peter),                calendar.getDetails(appt)            );        }    },    false); PS: does not work
  58. 58. James Pearce @jamespearcehttp://tripleodeon.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×