Context is King: AR, AI, Salience, and the Constant Next Scenario


Published on

Clark Dodsworth’s AREvent talk, Santa Clara, CA June 3, 2010: "Context is King: AR, AI, Saience, and the Constant Next Scenario" Mostly about smartphone AR as a gateway to context aware computing becoming indispensible.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Context is King: AR, AI, Salience, and the Constant Next Scenario

  1. 1. Context is King<br />AR,-Salience and the<br />Constant Next-Scenario<br />
  2. 2. Process of tool design<br /> Empathic pragmatismBroad-based tool knowledge / pattern recognition / intuitive<br />Perspective“Product as tool as interface” Tool should fit you. <br />WorkUser-centered product strategy, dev, design.<br />
  3. 3. “Ambient Intelligence” strategy Philips, ‘98<br />all photos: Philips<br />© 2002 Elmo Diederiks<br />
  4. 4. Hands-up displays<br />Commercial helmet-mounted displays<br />AR Timeline<br />Efficientpower usage<br />Dedicated tools begin:<br />Commercial aviation<br />1st car HUD:Cutlass Supreme <br />ARToolkitUW HITLab<br />Rear-view <br />mirrors<br />Useful smart- phone AR<br />Sailboat <br />tell-tales<br />US DoD air-combat testing<br />“Augmentedenvironments,” sensor nets <br />1stjetfighter <br />heads-up display<br />Nokia’s MARA <br />Now<br />20??<br />~1900<br />1958<br />1970s<br />1974-8<br />1988<br />1992+<br />1999<br />2006<br />Prehistory,soon after invention of the sail.<br />Horse<br />carriages<br />The goal<br />Tom Caudell &David Mizell<br />“UbiComp”:Mark Weiser<br />Sutherland/Sproull, ‘68: 1st HMDs<br />
  5. 5. AR Phase One current status<br />Smartphone ecosystem.<br />Hardware & software platforms exist.<br />Infrastructure issues: huge. <br />Power use issues: huge.<br />Privacy/security issues: gigantic.<br />Interoperability issues: roadblocks.<br />ButPhase One AR is just part of transition to mobile broadband as our primary platform.<br />
  6. 6. AR Phase Two<br />Smartphone ecosystem: <br />------The platform for<br />)<br />(<br />Pervasive computing<br />Context awareness<br />Semantic Web<br />Ubicomp & sensornets<br />Ambient Intelligence <br />Things that Think<br /> last.<br />
  7. 7. Alfred North Whitehead-(1861-1947)First augmented reality theorist<br />1: “Civilization advances by extending the number of important operations which we can perform without thinking of them.”<br />----------------------------...AR shouldn’t increase cognitive load<br />2: “The art of progress is to preserve order amid change and to preserve change amid order.” ...AR: contextually dynamic data management & display<br />3: We think in generalities, but we live in detail.” <br />---------------------------...AR tasks: managing detail<br />
  8. 8. Next Steps: AR Phase Two<br />“It’s all about long-term, sustaining relationships.” <br /> -Alan Kay (2009), on the computer industry trend toward a service-based model.<br /><ul><li>Phase Two AR: long-term, sustaining,-contextually-dynamic, hyperpersonalized-relationships between users & info services.
  9. 9. The available data streams must be far better managed. </li></ul> ...It will just happen first on smartphones. <br />
  10. 10. AR Phase Two<br />Adaptively deliver info, highly personalized via n dimensions of user states, with high space/time precision, without spiking your cognitive load.<br /> Augmented Context + Information Salience = Value<br />
  11. 11. AR Phase Two<br />Mobility <br /> • Diversity of use cases / user.<br /> • User attention = scarce resource.<br /> Tool should learn the user.<br /> The learning is contextual: context becomes king.<br /> • Collect & analyze user’s paths through location, time, tasks, people, queries, interests, decisions...<br /> • Accretes<br /> • Deliver predictive, scenario-driven support.<br />Result: Adaptive context analysis manages data delivery: What, when, how much, what modality, what priority. <br />-------------------------------------------------------Especially when not-to.<br />
  12. 12. Phase Two<br />Context Is King<br />Realtime, context-driven mobile broadband services.<br /> “The software layer that learns you.”<br /> • Realtime data delivery (live monitoring of processes & states).<br /> • Constant predictive analytics, contextual filtering/salience eval. <br /> • Semantic Web:better,-partly-automated-search.<br /> • Hyperpersonalized database creation, web services.<br />• Digital assistants (Sense Networks, Siri) learn you over time.<br />
  13. 13. Phase Two Software<br />•  Allostatic control of data delivery: only the data needed, when needed. <br />• Constant autodiscovery of data feeds, sensor feeds.<br />• Constant markerless 2D & 3D feature I.D. and object recognition. <br />• Motion analysis + evaluation linked to the object recognition.<br />• Indoor position mapping: multiple methods.<br />• Nuanced gestural interpretation + integration with voice.<br />• Constant audio + visual awareness with flagging & interpretation.<br />
  14. 14. Phase Two Hardware<br /><ul><li>Near-field RFID
  15. 15. Constant, wide-field visual recording + auto-tagging
  16. 16. Constant markerless-object I.D. in 2D-&-3D-+-auto-tagging
  17. 17. Constant audio recording + auto-tagging
  18. 18. Galileo-grade location (+ dead reckoning, incl. gyros)
  19. 19. Ultrasonic& IR-sensors for position sensing, spatialanalysis,modeling, mapping, contextualizing
  20. 20. Laser projector + 3D object geometry awareness: </li></ul>-------------“shape-adaptive projection”-for-object-augmentation<br />
  21. 21. Phase Two Example Market<br />Entertainment<br /> First rule of theme parks:<br /> You bring the fun in with you - your friends-& family. <br /> • The attractions you pay for aren’t the experience.<br /> • Now, a new type of friend.<br />For as long as you want.<br />c. Lynne LaCascia, all rights reserved<br />All images © Chimera Design 2007-2008<br />
  22. 22. Phase Two Example Market<br />Entertainment<br />Any licensed character: a useful buddy, a delivery method on top of context-aware, locational, hyper-personalized services. <br />The character behavior, persona, & animation: Character Skin.<br />
  23. 23. Dubailand AR project ’07-’08 • Convert concept to smartphone AR + lots of hidden sensors & projectors.• Live & scripted personalized entertainment via licensed characters, + services for hotel, leisure, retail, restaurant, transit,-and-follow visitors home.<br />© Chimera Design 2007-2008<br />Marvel Superheroes park<br />Part of Dubailand model:<br />c. Lynne LaCascia, all rights reserved<br />
  24. 24. Dubailand AR project ’07-‘08 <br /><ul><li> Storytelling
  25. 25. Extension of attractions
  26. 26. Personal guide/friend
  27. 27. Contextuallycustomizedexperiences
  28. 28. Interaction with park elements
  29. 29. Wayfinding
  30. 30. Drill-down for information
  31. 31. Social networking: local & global
  32. 32. Database: each user’s interests
  33. 33. Game play on device & with others
  34. 34. Marketing: retail opportunities
  35. 35. Long-Term Relationship: “follow the visitor home”</li></ul>All images © Chimera Design 2007-2008<br />
  36. 36. Phase Two Biological Model - Consciousness<br />Key functions of consciousness:<br /> 1) Constant, integrated analysis & evaluation of surroundings<br /> 2) Dynamically judge salience of each aspect of your current situation (state) in context of your goals/needs<br /> 3) Constantly construct scenarios for next step<br />Parallel function of AR2: assess, select & deliver salient data <br /> • “Contextual evaluation drives contextual augmentation” • “Software ‘awareness’ augments user’s awareness”<br />History + current state = basis of decision-making for next state<br />
  37. 37. Phase Two Biological Model<br />The brain’s Default Mode Network<br /> • One of the hardest-working systems in the brain<br /> • Active only when the brain is notfocused on a particular task<br /> • 2 linked brain areas:<br /> Empathy re: intent of others,-+ your own state awareness<br /> Personal memories + visualization of future scenarios<br />State & intent awareness<br />Scenario modeling<br />
  38. 38. Default Mode Network of the brain<br />Posterior <br />cingulate <br />cortex, with <br />the precuneus:<br />accessing<br />personal <br />memories, <br />visualizing<br />oneself in <br />scenarios.<br />Medial <br />prefrontal <br />cortex:<br />Imagining & <br />modeling-the mindset of others.<br />> Who you are, what’s relevant now, and very-near-future scenarios to choose from<br />Brain photo: Omikron/photoresearchers Network overlay: Olaf Sporns/Indiana Univ. (modified by J. Korenblat) <br />
  39. 39. Phase Two Implementation<br />Brain Default Mode Network: constantly considers what to do next<br />> Observe states/intents, visualize-scenarios/outcomes<br /> > Scenario creation supports each next decision<br />Phase Two mobile broadband services assess, prioritize, & select data support decisions<br /> • Realtime data mgmt. helps preserve-focus<br /> • Inverse Content Management System (CMS) ...the filter part<br /> • Adaptive B2B & B2C service optimization ...the revenue part<br />
  40. 40. Facets of Phase Two AR<br />Efficiency & revenue from contextual salience<br />Location Based-Services<br />Constant personal scenario support<br />Autodiscovery:net services &sensor data<br />Integrated sensor,gesture, & voice<br />Hyper-personalization,relationship mgmt.<br />Predictive analytics<br />
  41. 41. Facets of Phase Two AR<br />Efficiency & revenue from contextual salience<br />Normal Everyday Life<br />Data interchange standards.<br />Artificial intelligence permeates web svcs.<br />Context <br />awareness.<br />User control of personal data.<br />Constant personal scenario support<br />Autodiscovery:net services &sensor data<br />Integrated sensor,gesture, & voice<br />Hyper-personalization,relationship mgmt.<br />Predictive analytics<br />Better interfaces matter.<br />Probabilistic data delivery. <br />Activity awareness & modeling.<br />Adaptive filtering of realtime data.<br />Resource conservation. <br />Manage the info overload.<br />The opposite of advertising.<br />Sensor datastream analytics.<br />Qualcomm<br />My6sense<br />Aloqua, Xiam<br />Your Name Here<br />Canesta, Apple/Siri Softkinetic, Nuance<br />Corventis Pachube<br />Sense Networks<br />© Clark Dodsworth<br />
  42. 42.
  43. 43.<br />end OA logo<br />Contact for .pptx & speaker notes.<br />
  44. 44. Credits<br /><ul><li>First use of term, “context awareness: Schilit, B., Theimer, M. Disseminating Active Map Information to Mobile Hosts. IEEE Network, 8(5) (1994) 22-32
  45. 45. Schilit, Bill.; Adams, Norman; Want, Roy. Context-Aware Computing Applications. 1st International Workshop on Mobile Computing Systems and Applications. (1994) 85-90
  46. 46. Default Mode Network paper: Gusnard, DA, Raichle, ME: “Searching for a baseline: the resting human brain from the perspective of functional imaging.” Nature Reviews Neuroscience, 2001</li></ul>• “iLamps: Geometrically Aware and Self-Configuring Projectors,” Rascar, et al. 2003<br /><ul><li>Commercial aviation HUD image courtesy Todd Lapin, cc
  47. 47. Photo of sail’s tell-tales courtesy Bill Gracey, cc
  48. 48. Modern helmet-mounted see-through display courtesy Rockwell-Collins: "SIM EYE SR100-A”
  49. 49. Theme park illustrations © Chimera Design, 2007-2009
  50. 50. Photo of Dubailand model © Lynne LaCascia, 2009
  51. 51. Photo of Alfred North Whitehead from</li>