Mobile Device APIs

47,463 views
47,075 views

Published on

Thoughts on the past, present and future of device APIs for (mostly mobile) web developers. NB: is probably out of date already.

Published in: Technology, Business

Mobile Device APIs

  1. 1. (Mobile) Device APIs James Pearce @jamespearce http://tripleodeon.com
  2. 2. What are Device APIs? “Client-side APIs that enable the development of Web Applications that interact with device hardware” ...and other capabilities outside of the browser’s traditional remit
  3. 3. Why are they important?• They’re not... ...if you’re happy with a web of documents• They are... ... if you want the the web to be competitive with native app platforms
  4. 4. HistoryCurrent Work Use Cases
  5. 5. WTAI Make Call<a  href="wtai://wp/mc;16505551234">    Call  us</a><a  href="call.wmls#makeCall()">    Call  us</a>//  call.wmlsextern  function  makeCall()  {    WTAPublic.makeCall("16505551234");} http://www.wapforum.org/what/technical/wtai-30-apr-98.pdf
  6. 6. WTAI Call Control<a  href="calls.wmls#setupCall()">Call  us!</a><a  href="calls.wmls#acceptCall()">Brring!</a><a  href="wtai://cc/sd;1*2*3456">Enter  code</a>//  calls.wmlsextern  function  setupCall()  {    WTACallCont.setup("16505551234",  1);}extern  function  acceptCall()  {    WTACallCont.accept("1",  1);}
  7. 7. WTAI Messaging//  sms.wmlsextern  function  sendMsg()  {    WTANetText.send("16505551234",  "WML  rocks");}extern  function  readMsg(i)  {    var  sms  =  WTANetText.read(i);    return  WTANetText.getFieldValue(        sms,  "body"    );}
  8. 8. WTAI Contacts//  contacts.wmlsextern  function  addFriend(i,  num,  name)  {    WTAPhoneBook.write(i,  num,  name);}//  return  structs  of  contact  detailsextern  function  getFriendById(i)  {    return  WTAPhoneBook.read("i",  i);}extern  function  getFriendByName(name)  {    return  WTAPhoneBook.read("t",  name);}
  9. 9. I may be laboring the point But that was 1998... ... and you still can’t do any of this with a contemporary mobile browser
  10. 10. BONDI“enables web based content to access nativedevice capability, intermediated through arobust, but flexible security framework” http://www.omtp.org/1.11
  11. 11. Messagingvar  sms  =  bondi.messaging.createSMS({    to:  ["16505551234"],    body:  "Home  soon"});var  mms  =  bondi.messaging.createMMS({    to:  ["16505551234"],    subject:  "I  will  be...",    body:  "...home  soon"});
  12. 12. Messagingbondi.messaging.sendSMS(    function  ()  {},  //  success    function  ()  {},  //  error    sms,  true);bondi.messaging.sendMMS(    function  ()  {},  //  success    function  ()  {},  //  error    mms,  true);
  13. 13. Messagingbondi.messaging.subscribeToSMS(    function  ()  {},                      //  success    function  ()  {},                      //  error    function  ()  {                          //  listener        alert("They  replied!");    },  {                                      from:  "16505551234"          //  filter    },    true                                            //  exclusive);
  14. 14. Contactsbondi.pim.contact.getAddressBooks(    function(books)  {  //  success        var  contact  =  books[0].createContact({            name:  James  Pearce,            emails:  [{email:j@tripleodeon.com}],            phoneNumbers:  [{number:16505551234}]        });    },      function  ()  {}        //  error);
  15. 15. Telephonybondi.telephonyLog.findLogEntries(    function  ()  {},  //  success    function  ()  {},  //  error,    [        bondi.telephonyLog.MISSED_CALLS_FOLDER,        bondi.telephonyLog.RECEIVED_CALLS_FOLDER    ],    {        phoneNumber:  "16505551234"    }); * Yes, it’s true: you can’t actually make or receive calls
  16. 16. App Launcherbondi.applauncher.launchApplication(    function  ()  {},  //  success    function  ()  {},  //  error    "tel:16505551234");//  mailto:...//  sms:...//  file:...
  17. 17. On one hand...• Security-conscious architecture• Comprehensive API support• Consistent API patterns• Conformance specifications• Reasonable test coverage http://www.omtp.org/1.11/apis
  18. 18. On the other...
  19. 19. bondi.*deviceapis.*
  20. 20. WAC 2.0• Accelerometer • File system• Calendar • Geolocation• Camera • Messaging• Contacts • Orientation• Core • Tasks• Device interaction • Viewport• Device status • Webview http://specs.wacapps.net/
  21. 21. Which is all awesome Except...
  22. 22. Device APIs Working Group “to create client-side APIs that enable the development of Web Applications and Web Widgets that interact with devices services such as Calendar, Contacts, Camera, etc” Joint efforts with WebRTC Working Group, WebApps Working Group http://www.w3.org/2009/dap/
  23. 23. W3C Device APIs• Media Capture(HTML) • Vibration• Media Capture(gUM) • Calendar• Contact • Permissions• Messaging • Menu• Battery Status • Gallery• Network Information • System info• Sensor • Media Stream
  24. 24. Media Capture (HTML)<input  type="file"  accept="image/*"  /><!-­‐-­‐      The  accept  attribute  is  a  hint.    Specification  suggests  that  it  can  be    used  to  offer  alternative  picker  UIs.-­‐-­‐> Public working draft, http://www.w3.org/TR/2011/WD-html-media-capture-20110414
  25. 25. Brad Lassey, September 2008
  26. 26. Media Capture (HTML)<input  type="file"  accept="image/*"    capture="camera"/><!-­‐-­‐      Again,  a  hint.  Valid  options  are:        camera,  camcorder,  microphone,  filesystem-­‐-­‐>
  27. 27. Media Capture (HTML)<input  type="file"  accept="image/*"    capture="camera"  id="photo"/>var  photo  =  document.getElementById(photo);photo.files[0].getFormatData(    function  (data)  {},  //  success    function  ()  {}            //  error);//  file  is  a  MediaFile,  extending  File//  data  is  a  MediaFileData
  28. 28. MediaFileDatainterface  MediaFileData  {        attribute  DOMString          codecs;        attribute  unsigned  long  bitrate;        attribute  unsigned  long  height;        attribute  unsigned  long  width;        attribute  float                  duration;}; ...seems to be under discussion
  29. 29. Media capture (gUM)navigator.getUserMedia(    {        audio:  true,        video:  false    },    function  (stream)  {},  //  success    function  ()  {}                //  error);//  stream  is  a  LocalMediaStream Editors draft, http://dev.w3.org/2011/webrtc/editor/getusermedia.html
  30. 30. LocalMediaStreamvar  recorder  =  stream.record();//  recorder  is  a  MediaStreamRecorderrecorder.getRecordedData(    function  (file)  {}    //  success);//  file  is  a  FileURL.createObjectURL(stream);//  a  Blob  URI  for,  say  <video  src=...  />    
  31. 31. Messagingnavigator.sendMessage(    "sms:16505551234?"  +        "body=JavaScript%20says%20hi",      [],                          //  attachments  as  File  array    function  ()  {},  //  success    function  ()  {}    //  error);//  also  mms:  and  mailto://  oh...  were  you  looking  for  receipt? Editors draft, http://dev.w3.org/2009/dap/messaging/
  32. 32. Sensorsnavigator.findSensors().onsuccess  =  function()  {    this.result.forEach(function  (sensor)  {        //  ...    });};var  sensor  =  new  SensorConnection(Temperature);sensor.addEventListener(sensordata,  function(e)  {    if(e.data  >  20.0)  {        window.console.log(Hawt);    }});sensor.startWatch({  interval:  2000  }); Editors draft, http://dev.w3.org/2009/dap/system-info/Sensors.html
  33. 33. Sensors• Temperature, ºC• AmbientLight, Lux• AmbientNoise, dbA• MagneticField, uTesla• Proximity, cm• AtmPressure, kP• RelHumidity, %
  34. 34. Contactsnavigator.contacts.find(    [                                              //  fields        name,          emails    ],      function  (contacts)  {},  //  success    function  ()  {},                  //  error    {                                              //  find  options          filter:  James,            multiple:  true    }); Editors draft, http://w3c-test.org/dap/contacts/
  35. 35. Contactinterface  Contact  {        readonly  attribute  DOMString                              id;                          attribute  DOMString?                            displayName;                          attribute  ContactName?                        name;                          attribute  DOMString?                            nickname;                          attribute  ContactField[]?                  phoneNumbers;                          attribute  ContactField[]?                  emails;                          attribute  ContactAddress[]?              addresses;                          attribute  ContactField[]?                  ims;                          attribute  ContactOrganization[]?    organizations;                          attribute  Date?                                      birthday;                          attribute  DOMString?                            note;                          attribute  ContactField[]?                  photos;                          attribute  DOMString[]?                        categories;                          attribute  ContactField[]?                  urls;};
  36. 36. Cool, but... “The ability to add and update contact information is not a function of the API provided in this specification.”
  37. 37. Holy Contacts Editing WTFvar  vcard  =      BEGIN:VCARDrn  +    VERSION:3.0rn  +    UID:  +  contact.id  +  rn  +          N:Pearce;Jamesrn  +    FN:James  Pearcern  +    END:VCARD;document.getElementById(vcard).href  =    data:text/x-­‐vcard;charset=utf-­‐8,  +    encodeURIComponent(vcard);  
  38. 38. WebAPI “We are aiming at providing all the necessary APIs to build a basic HTML5 phone experience within the next 3-6 months” https://wiki.mozilla.org/WebAPI
  39. 39. WebAPI• Telephony • Embedded browser• Messaging • Battery• Contacts • Device Status• Camera • Settings• Network • Mouse Lock• USB • Vibrator
  40. 40. Hopefully W3C++• “Our proposal might or might not look like it for the moment”
  41. 41. Telephony!var  tcall  =    navigator.telephony.dial("16505551234");tcall.readyState;    //  dialing,  ringing,  busy,  connecting,    //  connected,  disconnecting,  disconnected,      //  incomingtcall.onconnected  =  function  ()  {};    //  onbusy,  ondisconnected  etctcall2.answer(); http://www.youtube.com/watch?v=RuIQskGD3u0#t=19s
  42. 42. Contacts “We need read/write”
  43. 43. USBvar  usb  =  new  MozUSBManager();usb.addEventListener("attachdevice",      function  ()  {});usb.addEventListener("detachdevice",      function  ()  {});usb.claimDevice({    deviceClass:  ...,    deviceSubClass:  ...});
  44. 44. We deserve Device APIs...• As forward-looking as WTAI• As well-structured as BONDI• As comprehensive as WAC• As respectable as W3C DAP• As agile as WebAPI
  45. 45. And we’ve got...
  46. 46. PhoneGap• Accelerometer • Events• Camera • File• Capture • Geolocation• Compass • Media• Connection • Notification• Contacts • Storage• Device
  47. 47. Use Cases
  48. 48. Useless Demos https://github.com/jamesgpearce/compios5
  49. 49. Fixing layout bugs, FMLwindow.addEventListener("deviceorientation",    function  (e)  {        orientation  =  Math.abs(w.orientation);        rotation  =  Math.abs(e.gamma);        if  (rotation  >  8  &&  orientation  ===  0)  {            if  (enabled)  {                disableZoom();            }  else  {                restoreZoom();            }        }    },  false); http://vimeo.com/adactio/iosbug https://github.com/scottjehl/iOS-Orientationchange-Fix
  50. 50. Camera• Profile pictures• Review sites• Photo sharing• Recognition • Bar-codes • Text • Faces
  51. 51. Filters!$(img).vintage({    vignette:  {        black:  0.8,        white:  0.2    },    noise:  20,    screen:  {        red:  12,        ...    },    desaturate:  0.05}); https://github.com/rendro/vintageJS
  52. 52. Augmented reality? http://fhtr.org/JSARToolKit/
  53. 53. Contacts & calendaring• Add business details to address book• Web-based messaging• Facial recognition again...• Presence• Context-sensitivity of content
  54. 54. Messaging• Sharing• Notifications• Offline transport fallback• Frictionless web sign-on
  55. 55. Expect more
  56. 56. I still want... • Audio processing • Bluetooth • RFID & NFC • Both cameras • Memory & power • Native recognition
  57. 57. window.addEventListener(    appointmentdue,    function(appt)  {        if  (device.near(WORK))  {            siri.remind(                contacts.get(Peter),                calendar.getDetails(appt)            );        }    },    false); PS: does not work
  58. 58. James Pearce @jamespearcehttp://tripleodeon.com

×