Successfully reported this slideshow.

Continuous and Seamless Applications

1

Share

Loading in …3
×
1 of 61
1 of 61

Continuous and Seamless Applications

1

Share

Download to read offline

A presentation I put together to demonstrate seamless integration of mobile, cloud, desktop and NUI applications. This was shown internally at SunGard, as well as for Industry Analyst Groups, Houston Tech Fest, Philly.NET Code Camp.

A presentation I put together to demonstrate seamless integration of mobile, cloud, desktop and NUI applications. This was shown internally at SunGard, as well as for Industry Analyst Groups, Houston Tech Fest, Philly.NET Code Camp.

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Continuous and Seamless Applications

  1. 1. An overview of SGS ATS work on new application experiences Michael Heydt Principal Technologist, SGS ATS CONTINUOUS (SEAMLESS), IMMERSIVE, AND CONTEXT AWARE APPLICATIONS
  2. 2. AGENDA • Overview • Components • Examples • Technologies • Demonstration • Applicability • Next Steps • Q&A
  3. 3. HOW WE GOT HERE • Seamless is a natural evolution / aggregation of all of the following previous work: • Composite Applications • Rich Interfaces • Mobile • Cloud • Cloud / on-premise integration • Natural User Interfaces • ATS is already in discussion with a major energy company to • Assist in building a workspace of the future • With fully seamless / mobile / interactive trading environments • Including room size interactive visuals, and • Community workspaces that adapt to the current person in the environment
  4. 4. WHAT I’M NOT GOING TO COVER • Simply a lot of stuff, so I’m going to only cursory cover: • Kinect and NUI • Cloud technologies • Specifics on programming • But I’ll be more than happy to do follow ups for anyone at a later time • The focus today is on what this is, a few examples, and a demo
  5. 5. MEET YOUR NEW OR SOON TO BE USER • http://www.fastcompany.com/magazine/162/generation-flux-future-of-business • Expects always on access • Ability to work anywhere, any time, on anything • Naturally works with multiple devices • Device convergence is a thing of the past
  6. 6. CONTINUOUS / SEAMLESS • The Continuous Client • http://www.engadget.com/2010/05/26/a-modest- proposal-the-continuous-client/ • When you leave one device, you pick up your session exactly in the same place on the next device you use • “Placeshifting” your computing experience from one device to the next with no break in your work, timelines or conversations. • But this is much more than just the “client”
  7. 7. COMPONENTS OF A CONTINUOUS APPLICATION Operating System Provides the capability to run code on a particular platform Application Dynamically composited: “the streaming application” Services Both in the cloud, as well as on other mobile and ephemeral systems Contextual App knows who, what, when, where and what’s around Rendezvous Ability to dynamically locate other devices utilize their capabilities Immersion The application experience is everywhere and all around the user Multi-modal Not just vision and typing, but gestures, source, voice and haptic Augmentation Applications utilize other devices to extend the experience
  8. 8. THE CONTINUOUS CLIENT • Not necessarily a common code base • More a set of similar services on different devices • That can find each other and augment the users experience • Ideally they can be generated and/or composited “on-the-fly” (mashups anyone?) • “Streaming Client”: where an application is composited and downloaded on the fly to a user based upon their current “context” • In essence the client becomes immersive, moving with the user across location and devices, constantly providing the user with the services that are needed exactly at that moment
  9. 9. CONTEXT AWARE • Applications know who is using them, where they are, what they are running on, and what is nearby to augment services (and what time it is) • Example of context: Attentive phone and Smart Actions
  10. 10. IMMERSIVE • Immersion is the state of consciousness where an immersant’s awareness of physical self is diminished or lost by being surrounded in an engrossing total environment, often artificial • http://en.wikipedia.org/wiki/Immersion_(virtual_reality) • Devices such as Kinect allow interaction away from the keyboard and mouse • Devices such as phones can augment capabilities (as we will see)
  11. 11. AUGMENTED • Augmented reality (AR) is a live, direct or indirect, view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data. • http://en.wikipedia.org/wiki/Augmented_reality • In addition to augmented reality, applications will be able to determine what other devices are nearby and use them to provide a combined and greater experience to the user
  12. 12. MULTI-MODAL • Interaction with the application becomes more than mouse and keyboard • These are antiquated and artificial means of interacting with the computer • The new modes: • GUI (graphical representation of information; does not imply mouse and keyboard) • Natural (gestural interfaces, either on tablets or with a Kinect) • Haptic (feedback given to the user in form or resistance and vibration) • Speech (ability to understand complex grammars for interacting with the system)
  13. 13. THE BRAVE NEW WORLD • These are no longer “new” or “advanced” technologies: • Rich Interfaces • Social communication
  14. 14. EXAMPLES • Fictional, but not so much any more… • Minority Report Shopping Mall • Real • Nsquared Seamless Architecture / Design Application
  15. 15. MINORITY REPORT REAL-TIME PERSONALIZED ADVERTISING • Scenario: • You walk into a public place, and video walls present you with personalized information
  16. 16. ENTERING THE PUBLIC PLACE
  17. 17. INITIAL IDENTIFICATION – MOTION TRACKING
  18. 18. DEVICE TRACKS MOTION
  19. 19. DOES A RETINAL SCAN TO IDENTIFY YOU
  20. 20. AND STARTS TO MAKE PERSONALIZED ADS
  21. 21. JOHN HAS SIMILAR TASTES TO MIKE
  22. 22. HOW REALISTIC IS THIS? • Not very unrealistic actually • Detailed motion can be tracked by Kinect • You can be identified easily by: • Facial recognition (my demo later) • RFID / NFC • QR Code (a new market for t-shirts and hats?)
  23. 23. NSQUARED SEAMLESS DEMO • Demonstration of a seamless application using multiple forms • Surface • Slate • Video wall • Kinect • Cellphone • Similar things will be demonstrated later • http://nsquaredsolutions.com/ • http://www.youtube.com/watch?v=oALIuVb0NJ4
  24. 24. PLACE A PHONE ON THE SURFACE
  25. 25. THEY START SHARING DATA – LIKE CONTACTS
  26. 26. SURFACE AUGMENTS THE PHONE
  27. 27. GESTURES ON PHONE EXTEND ALSO
  28. 28. SELECTING A DOCUMENT
  29. 29. AND THE SURFACE STARTS TO OPEN DATA
  30. 30. SLATE IS USED TO AUGMENT THE SURFACE
  31. 31. GIVING A DIFFERENT “LENS” ON THE DATA
  32. 32. SELECT A ROOM AND IT SHOWS ON THE SLATE
  33. 33. NOW MOVE TO A VIDEO WALL AND KINECT
  34. 34. GESTURE TO GO INTO THE HOUSE
  35. 35. AND YOU ANIMATE IN
  36. 36. LETS INTERACT WITH THE MODEL
  37. 37. USE SLATE TO SELECT A NEW KNOB WITH A FLICK
  38. 38. AND THE MODEL CHANGES
  39. 39. PUT THE BILL OF MATERIALS ON THE SURFACE
  40. 40. AND IT FIGURES OUT WHAT IT IS
  41. 41. GIVES A MAP TO THE LINE ITEMS
  42. 42. LETS SEND IT TO THE CUSTOMER
  43. 43. GET THE CONTACTS FROM THE PHONE AND SEND
  44. 44. NSQUARED – WHAT WAS DEMONSTRATED? Continuous Client Multiple applications working together to complete a task Rendezvous Finding other systems and collaborating Gestures Flicking data from one device to another Location Knowing what devices are nearby and where the user is Immersion Movement through the data Augmentation Multiple examples of devices augmenting each other
  45. 45. THE DEMO - SCENARIO • An arbitrary person sits in front of a computer which recognizes the user and starts communication with the users phone • User can interact with the phone application and request augmentation on the desktop system • Desktop application can retrieve contacts from the phone to send mail
  46. 46. THE TECHNOLOGIES IN THE CONTINUOUS DEMO Kinect Used for vision and voice capture Computer Vision OpenCV/EmGuCV Cloud Services Microsoft Azure, SQL Azure, WCF and REST API’s Phone Windows Phone 7 .NET Common code for phone, desktop and cloud Voice Recognition .NET Speech SDK Location Services GPS on the phone. Spatial data services in the cloud. Rendezvous Microsoft AppFabric Service Bus to locate and communication between mobile systems Near-range wireless UDP communications when on local WiFi (fallback to cloud messaging) Gestures Flick data from phone to desktop
  47. 47. KINECT • Video, Depth and Audio capture • $199! • The demo uses a kinect, but for only video and voice
  48. 48. KINECT POINT CLOUDS AND SKELETONS
  49. 49. THE DEMO – GENERAL OUTLINE • Train the system on your face on your desktop • Training data is stored in the cloud • Phone app sends location updates to the cloud • Cloud does spatial queries to find nearby services and lets them know you are near • When you are near, local systems get the facial data from the cloud • When it sees you, it starts communications with your phone, first by cloud messaging and then direct WiFi if available • Use phone to look at stocks, and “flick” them onto the other system
  50. 50. FACE TRAINING AND RECOGNITION
  51. 51. REMOTE APP – DOESN’T KNOW WHO I AM
  52. 52. THE PHONE APP
  53. 53. THE PHONE APP
  54. 54. REMOTE APP NOW KNOWS AND SEES ME
  55. 55. PHONE APP – STOCK AND FLIP
  56. 56. REMOTE APP GETS MESSAGE FROM PHONE
  57. 57. REMOTE APP DISPLAYS AUGMENTED DATA
  58. 58. NEXT STEPS • Add ability to have mobile phone give real-time orientation, position and movement data • Can be used to manipulate items • Extensions to NUIDOTNET framework formalizing: • Location updates • Service location • Device location via spatial queries • Capabilities exchange • Application compositing and delivery (streaming) • Robust gesture processing (on phone, and with Kinect) • Many more
  59. 59. Q&A • Any questions?
  60. 60. THANKS!

×