Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cognitive Insights drive self-driving Accessibility


Published on

Eric Manser and Will Scott from IBM Research, presentation on "Cognitive Insights Drive Self-driving Accessibility" as part of the Cognitive Systems Institute Speaker Series

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Cognitive Insights drive self-driving Accessibility

  2. 2. Important Factors • Over one billion people, or 15% of the world's population, experience some form of disability (World Health Organization) • According to the United Nations, the number of people aged 60 years or older is projected to grow by 56 percent worldwide by 2030. • 1.6 billion people or 23% of the world’s population is over age 50 • The world is experiencing an explosion in connected devices and cognitive computing capabilities, with over 6.4 billion connected devices and interfaces worldwide • Estimated that 10 million self-driving cars will be on the road by 2020. People with Disabilities New Technologies Aging Population Self-Driving Vehicles
  3. 3. About the Project IBM Research and Watson IoT have announced a new collaboration with Local Motors and the Consumer Technology Association to create the world's most accessible, self-driving vehicle. • This partnership will leverage the power of IBM Watson, the Internet of Things, and new innovations in accessibility technology. • We will be launching a series of workshops with talented people of all ages, backgrounds and abilities, including P-TECH schools, High School STEM, AARP and universities including MIT and Princeton • Creating the most accessible vehicle will: • Allow older adults to age in place and remain independent and self-sufficient in their communities • Help those with cognitive, vision, memory or physical challenges to leverage transportation, improving their independence and quality of life • Provide enhanced access to work opportunities and community services to people of all ages and abilities • Ensure a successful transition to driverless cars for all
  4. 4. Workshops and Hackathons
  5. 5. P-TECH at Carver Workshop Location: Local Motors National Harbor MD Date: February 9, 2017 Audience: PTECH Carver Students Brainstorm Challenge - High Impact Area Recommendations AARP Hatchery Workshop Location: AARP Hatchery, Washington DC Date: February 10, 2017 Audience: Older adults (60+) Recommendations • How would you know what to do in an emergency or dangerous situation (can't hear or see) • How to safely find and get to an Olli stop • How do I get on and off Olli • How do I know where to sit • How do I know where I am at and when to get off; missing stops • How do I interact with other passengers (can't hear, see, speak) • Make Olli On Demand - Olli should come to the door, pickup at exact time and location via easy to use call application • Make Olli Safe – should be able to avoid danger/accidents, ensure safety at stops and onboard; give emergency instructions and call to nearest law enforcement and first responders based upon location • Make Olli Personal - should recognize/scan riders before entering vehicle, know their preferences and selected profile information • Make Olli comfortable - easy to get on/off, ability to accommodate or stow equipment (cane, wheelchair, umbrella, etc..), seating that adjusts for bone/joint issues • Make Olli easy to understand - Information delivery should be clear and delivered in multiple formats - visual and auditory • Make Olli affordable to use
  6. 6. Front Porch Workshops Location: Retirement and assisted living centers in San Diego, Carlsbad and La Jolla, CA Date: February 27-28, 2017 Audience: Older Adults (60+) Recommendations Visual Impairment Workshop Location: IBM Research, Cambridge MA Date: March 7, 2017 Audience: Advocates for Visually Impaired and Transportation (Mass Association for the Blind, Perkins School, Umass, Carroll School, MBTA) Recommendations • Alerts/Notifications - Need multiple interfaces – audio, text, haptic • Develop bluetooth connection for headset coupled with a smartphone/pad app - can provide personalized instructions and experience • Provide an Alexa-like service or app to call for or schedule an Olli • Need secure way to access and utilize vehicle - fob key, biometric or visual recognition • Improve access on/off for wheelchair users and others with assistive devices • Low riser for entrance/exit • Provide storage for walker, canes, packages • Develop a seat that can raise or lower automatically to help with moving between sitting and standing positions; provide options for individual seats • Implement grab bars with certain seats • Accessibility features should be discreet - ramps should raise/lower once user who needs them is detected. • Have Olli recognize rider and implement personalized profile option • Have a personalized narration delivered over smart device/headphones; deliver ability to repeat info • Voice enabled navigation for entrance/exit and seat location • Provide ability to secure wheelchairs and other assistive devices • Some vehicles have built in clamps/restraints - others secure via straps; no standards in place from wheelchair manufacturers, depends on device and vehicle • Provide a pre-made wheelchair for Olli • Provide moveable/retractable seats to accommodate wheelchairs • Provide redundancy for wheelchair accessible entrance and exit • Accommodate service animals
  7. 7. HLAA and Henry Claypool Discussions (Hearing and Mobility Impairment) Date: March 29, 2017 Recommendations: Hackathons MIT ATHACK, Cambridge, MA Date: March 4, 2017 Challenge: Create a system to guide a visually impaired rider to an open seat on a public vehicle Solution: • Overhead camera analyzes interior images to identify open and occupied seats • Rider pairs smartphone to vehicle • Camera communicates with smartphone app • App opens in window the size of the screen, simplifying engagement • App communicates via Bluetooth audio navigation guidance to rider • As a baseline, all information should be available via text – app, exterior of vehicle and interior • Sign language translation highly desirable – sign to Olli, and receive sign language back • Voice activated systems need to integrate with Tcoil loops/assistive devices • Integration with Bluetooth devices desirable • Integrate Haptic (touch) technology for alerts • Provide for ambient noise reduction - phone/listening cone at each seat; have info directed to telecoil/hearing aid • All information – route, destination, updates, safety instructions, alerts – should be provided via multiple interfaces – visual, text, auditory with Tcoil loop assist, sign, haptic • Need to provide automated lock down for wheelchairs • Need to provide space to turn wheelchair around in vehicle
  8. 8. Hackathons HackPrinceton Date: March 31, 2017 Future Workshops & Hackathons Workshops • City of Columbus/OSU – Mobility, TBD • City of Las Vegas, TBD • City of Santa Clara/Palo Alto VA Hackathons • TOM-NYC (Tentative), Brooklyn, NY Events • AAPD May 17, Local Motors National Harbor • M-Enabling Summit, June 13-14, Washington DC • Watson Developer Conferences, 9 locations worldwide • Q2-Q4 Challenge: Win the Best Use of Watson IoT Challenge by creating an innovative Accessibility solution using #AccessibleOlli. Results: • 12 teams built with Watson. 67 teams overall submitted projects. • 500 developers engaged, 50 enabled with Watson IoT platform
  9. 9. User Scenarios and Use Cases
  10. 10. Potential Solution Ideas • Uber-like app for Olli, voice and text interfaces • Personalization is a key component - Olli should be able to recognize your face or your fingerprint, and know about what route you usually take, what your schedule is, interaction preference, language preference, who to contact in an emergency • All interactions on Olli should be multi-modal – voice, text, touch • Have a personal "Olli in your ear" that guides you on your trip - have Olli describe what you are passing, give you reminders and suggestions, and tell you when you have arrived at your stop • Use voice, text and haptic or other technologies to guide people on the vehicle and find open seats; Seats should vibrate to signal a stop • Olli should have a seat that automatically folds away to accommodate a wheelchair user and a place to store walker or other assistive mobility devices. Wheelchairs and other assistive devices should automatically be secured. • Have Olli detect emergency situations either in or outside the vehicle, and call the nearest responders (EMT, police, fire) for help. All emergency instructions should be given Focus on riders with these challenges: • Visual impairments, from low vision to complete blindness • Cognitive issues, ranging from dyslexia to memory loss • Hearing impairments including deafness • Mobility issues requiring assistive devices such as wheelchairs, canes and walkers For each category, use cases will cover • Arranging for a ride on Olli • Navigating to the vehicle • Navigating onto the vehicle • Securing a seat/location on the vehicle • Experiencing the ride and being alerted to destination • Arriving at destination and exiting the vehicle
  11. 11. Arranging for a ride on Olli Navigating to Olli Navigating onto Olli & Securing seat Experiencing ride and destination alert Arriving at destination and exiting vehicle Use Case Template Cognitive “Middleware” Across All Use cases Multi-Modal Interface • Voice • Text • Haptic Personalization • Profile • Language • Interface • Alerts Security • App or biometric access Emergency contacts and process Visual Hearing Mobility Cognitive Directions & Guidance • Location Support • Curb cut-outs • Ramps • Elevators Integration with App or Device
  12. 12. Arranging for a ride on Olli Navigating to Olli Navigating onto Olli & Securing seat Experiencing ride and destination alert Arriving at destination and exiting vehicle Visual Use Case Multi-Modal InterfacePersonalization Security Directions &Guidance “Uber” or “Alexa” –like app, voice activated • Establish secure profile with preferences • Call/arrange for Olli pickup • Get updates on arrival/timetable • Get details on location Using voice-activated smartphone get: • Updates on location • Arrival time • Any route issues/delays • Navigation guidance to pickup site Smartphone app should sync with Olli before boarding and communicate/confirm via Bluetooth w/headset: • Onboard info • Identification of rider • Destination and preferred route • Preferences (language, interface) • Emergency contact • Navigation • Using VR and interior camera, identify open seats • Communicate open seat location to rider via app (voice) • Communicate location of storage area (canes, packages, etc.. Smartphone app to provide Olli in your ear/app” – voice interaction via Bluetooth integration • Pre-notification alert • Arrival alert at destination. • If desired, route narration • Emergency situations • All instructions given via voice • Emergency personnel/first responders alerted that riders with disabilities are on board Delivered via Smartphone via voice • Destination announcement • Reminder and retrieval of stored items • Rider advised of way out and other accessibility features at destination Cognitive middleware integration with voice activated Smartphone app
  13. 13. Looking Towards the Future
  14. 14. Existing Watson Integration with OlliPotential New Technology Areas • Visual Recognition • If trained on wheelchair/walker, automatically lower ramp, fold seat • Detecting someone is at a stop and/or has ordered Olli • Rider recognition • Open/Occupied seats • Emergency situations • Personalization • Rider profile – created by caregiver or individual • Interface and Language choice • Destination options • Reminders • Notification is rider gets off/on at non-profile locations • Haptic • For blind/low vision aid with navigation on/off and with finding seat • Haptic feedback to notify rider they are at destination • Bluetooth integration • Braille reader integration
  15. 15. Watson Analytics and Machine Learning • There are a number of Watson services in use with Olli today – the next step is to incorporate Watson Analytics and domain specific Machine Learning algorithms • Watson Analytics is a cloud-based, smart data discovery service • Guides data exploration • Automates predictive analytics • Machine learning is a form of artificial intelligence (AI) • Provides computers with the ability to learn without being explicitly programmed • Computer programs that can change when exposed to new data
  16. 16. Data Driven Insights • Time spent in transit can be leveraged to yield many data driven insights • Big data and analytics, coupled with cognitive computing approaches can be used • Analysis can be performed to asses general health and well- being of passengers over time, using non-intrusive sensors to monitor: • Weight – via seat mounted sensors • Heart rate – via infrared cameras • Facial expressions (for emotional well-being) – via machine learning and visual recognition • Gait - via machine learning and visual recognition • Cognitive decline – via onboard digital “games” to access mental well-being • This data can be fed to a larger, holistic cognitive solution…
  17. 17. Considering a Holistic Cognitive Solution • A cognitive Knowledge Aggregation solution is embodied by a contextual data fusion engine, that centralizes IoT and System of Record/Engagement data across multiple channels • “Snapshots” of passenger data captured while riding Olli can be applied to this larger, holistic cognitive solution to monitor passenger health and well-being • Over time, Olli-based data, along with data captured from other channels (home, wearables, social, etc.) can be used to predict inflection points in well-being
  18. 18. Questions & Answers