Applying Drools in Assistive Technology


Published on

Talk I gave at JUDCon 2010 in Berlin

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Expert – Rule Engine side, Logic side Flow – advanced process engine, tight integration of processes and rules Fusion – Even Processing , CEP integration into Drools Guvnor - about server-side management, web stuff
  • Expert – Rule Engine side, Logic side Flow – advanced process engine, tight integration of processes and rules Fusion – Even Processing , CEP integration into Drools Guvnor - about server-side management, web stuff
  • Applying Drools in Assistive Technology

    1. 2. Applying JBoss Drools in Assistive Technology
    2. 3. Who is Tihomir? <ul><li>R&D Engineer on Drools
    3. 4. Contributor and committer on Seam </li></ul>
    4. 5. <ul>Agenda </ul><ul><li>Assistive Technology (AT)
    5. 6. Applying Drools in AT
    6. 7. Event Processing
    7. 8. Drools Fusion examples
    8. 9. Vision for the future
    9. 10. Q&A </li></ul>
    10. 11. <ul>Assistive Technology </ul><ul><li>Disability affects 15-20% of every country's population (with 10% being classified with severe motor impairments). Growing trends (aging population, severe stroke).
    11. 12. Categories of disabilities: </li></ul><ul>- Sensoric deficiencies – reduced ability to acquire inputs from the physical world (e.g. blindness, deafness, etc.), - Expression deficiencies – reduced ability to produce output to the physical world (e.g. voice, motion, etc.). </ul><ul><li>Assistive technology (AT) – product, device or equipment (e.g. assistive, adaptive or rehabilitative devices) that helps individuals with disabilities to increase, maintain or improve functional capabilities (AT Act, 1998).
    12. 13. AT products help peoples with disabilities to achieve greater independence, enhance their quality of life and compete in the real world ( inclusive society ). </li></ul>
    13. 14. <ul>AT Current situation </ul><ul><li>Steadily growing market (especially for small businesses, estimation: US$ 12 billion in USA, 30 billion in Japan and 40 billion in Europe)
    14. 15. Modern AT industry sustainable and profitable.
    15. 16. The key problems: </li></ul><ul>- User View – Expensive, not sufficiently flexible and adaptable devices. Slow and not responsive systems, non-transparent, difficult to use, sensitive to disturbances and errors. - Developer View – Demands for highly customized products within a huge target-audience, risky business (user acceptance, competitive pricing). </ul>
    16. 17. <ul>AT research and development </ul><ul><li>Long term AT research (priorities in several countries).
    17. 18. Main focus: respond to the needs and wants of users.
    18. 19. Close relationships to main-stream researches, such as robotics, cognition, AI, IT, sensor technology, real time control etc.
    19. 20. Attractive common research focuses on: </li></ul><ul>- human enhancement, - rehabilitation robotics, - human-robot interaction (HRI), - human-computer interfaces (HCI), - brain-computer interfaces (BCI), - computer vision, etc. </ul>
    20. 21. AT essential distinctive problem <ul><li>Extreme individual differences that characterize abilities and needs of motor-impaired people. </li></ul>
    21. 22. Keys to increase acceptance and use <ul><li>High personalization and customization
    22. 23. Scalability and adaptivity
    23. 24. Improve performance over time
    24. 25. Acquisition/preservation of user experience and knowledge
    25. 26. Improve maintainability and robustness in spite of user and environmental disturbances </li></ul>
    26. 27. <ul>AT example: computer access </ul><ul><li>Computer input devices for motor-impaired people
    27. 28. Mouse cursor control for handicapped individuals (MCC) – main operations: pointing and clicking .
    28. 29. Ca. 250 commercial MCC products available at the market worldwide.
    29. 30. Multitude of MCC concepts: switch-related systems (sip/puff switches), head pointing systems, eye tracking systems, BCI etc.
    30. 31. None of them provide a standardized general system for subjects with severe disabilities. </li></ul>
    31. 32. <ul>MCC AT principles </ul>
    32. 33. <ul>Roadmap to success </ul><ul><li>More complex systems that combine several input channels.
    33. 34. Use of innovative technologies, such as AI approaches, methods and tools (e.g. automated reasoning, first-order logic, fuzzy logic, neural-networks etc.).
    34. 35. Development of standardized HW and SW modules that can be configured for a specific user. </li></ul>
    35. 36. <ul>Example: Improve MCC by using fuzzy logic </ul><ul><li>Convenient simple input device: pneumatic dual-action input switch (sip and puff).
    36. 37. Indirect pointing of target items (scanning) based on Morse codes sequences of dots (sip) and dashes (puff).
    37. 38. Traditional line scanning.
    38. 39. Quite inefficient and tedious, especially for users with limited breathing or learning capabilities. </li></ul>
    39. 40. <ul>Why Fuzzy Logic? </ul><ul><li>Dealing with imperfect information natively </li></ul><ul>IF direction IS positive AND strength IS weak THEN output IS weak_positive </ul>
    40. 41. <ul>Fuzzy scanning and inference algorithms </ul><ul><li>Puff/sip define cursor motion directions.
    41. 42. New input air flow defines velocity of cursor moving.
    42. 43. Fuzzy scanning algorithms: circular scan, selection area scan, object selection, etc </li></ul>
    43. 44. <ul>Fuzzy control outcomes and limitations </ul><ul><li>Improved mouse control. Clear step-forward in comparison to the state of art.
    44. 45. Feasible and promising solution for development of sophisticated and adaptive MMC (HCI) systems.
    45. 46. Constraints: relatively good breathing control is needed. Sensitivity to disturbances.
    46. 47. Multi-channel inputs and action outputs (MIMO) can further improve performance and robustness, however fuzzy logic and control applications become extremely complex.
    47. 48. Processing large number of input events becomes problematic. </li></ul>
    48. 49. <ul>Beyond just fuzzy reasoning </ul><ul><li>Needs: </li><ul><li>production rule system
    49. 50. ability to combine classical predicate logic with fuzzy logic to represent the system as combination of discrete as well as continuous events.
    50. 51. ability to process large number of complex events through multiple input channels.
    51. 52. BPM capabilities
    52. 53. tooling and testing support
    53. 54. ability to embed (parts of) system </li></ul><li>One-stop-shop solution?? </li></ul>
    54. 55. Drools (1) “ A common platform to model and govern the business logic of the enterprise.”
    55. 56. Drools (2) “ A common platform to model and govern the business logic of the enterprise.”
    56. 57. <ul>Drools for AT? (1) </ul><ul><li>Fully integrated logic platform (rule engine, workflow, event processing, etc) that supports easy and frequent change of rules (adaptation to users).
    57. 58. Extensions of the RETE networks using set of pluggable modules.
    58. 59. Drools-Chance (still in development) makes Drools capable of dealing with imperfect information natively. Also adds possibility to configure behavior of the engine to reason with different type of imperfection without changing the production rules.
    59. 60. Tooling support
    60. 61. Ability to test/monitor all parts of AT system.
    61. 62. Ability to run/embed into OSGi environments and future AT devices. </li></ul>
    62. 63. <ul>Drools for AT? (2) </ul><ul><li>Separates applications (AT devices or computer control systems) from conditions which control the application flow.
    63. 64. Changing rules does not require a complete deployment of AT systems.
    64. 65. Provides background to build modular and scalable AT systems, such as integrate additional sensors for a specific user and develop rules, without influencing specific application (e.g. computer or assistive device control).
    65. 66. Rules can be modified by different group of users (technicians, physicians, end-users).
    66. 67. Rule language is simple to learn.
    67. 68. End-users can be involved to resolve conflicts and perform specific actions (improved robustness).
    68. 69. Efficient for frequent change of rules dependent on user experience, health state, specific needs etc. </li></ul>
    69. 70. First steps toward improvement for MCC - Event Processing <ul><li>Any form of computing performing operations on events
    70. 71. Event: </li><ul><li>Real world: occurrence of something that has happened at a particular point in time
    71. 72. Comp. Systems: entity/object that represents this occurrence
    72. 73. 1 RW Event -> ∞ CS Events
    73. 74. Raw vs. Derived Events </li></ul><li>Complex Event: abstraction of a number of related events </li><ul><li>For example a hurricane is an abstraction of a large number of natural eve nts </li></ul></ul>
    74. 75. Main Concepts of Event Processing <ul><li>Event Processing Networks (EPN): </li><ul><li>Producers: introduce/emits events into the processing network (Sensors, Systems, Apps, Business Processes, Human Interaction, etc)
    75. 76. Distribution Channels: provides means of distributing events (typically one-to-many distribution, async)
    76. 77. Consumers: receive and act upon events (Applications, Systems, Data Stores, Business Processes, etc)
    77. 78. Intermediary Processing: rout / filter (raw events), produce (derived events). Consists of a number of building blocks (Agents) </li></ul></ul>Intermediary Processing Event Producers Event Consumers
    78. 79. Event Processing Agents Building Blocks for: Translation, Composition, Aggregation, Enrichment, Splitting, Projection, Pattern Detection, etc. Even Filtering Event Matching Event Derivation time time time
    79. 80. Event Processing Network (1) <ul><li>Network based on the core EP building blocks
    80. 81. Can be described as process flows such as Drools-Flow </li></ul>
    81. 82. Event Processing Network (2) <ul><li>Describing network building blocks with Drools-Flow bring good feature support to EP </li><ul><li>Custom work item handlers
    82. 83. Testing, debugging, monitoring (and self-monitoring/adapting)
    83. 84. Customizable and easily replaceable network parts
    84. 85. Integration with Human tasks </li></ul></ul>public class ErrorEventListener implements ProcessEventListener { public void afterNodeLeft(ProcessNodeLeftEvent event) { ksession .getWorkingMemoryEntryPoint( entryPointName ).insert(event); ... } declare ProcessNodeLeftEvent @role( event ) end rule &quot;Number of Cursor Rotations is above threshold&quot; when Number( nbProcesses : intValue > 5 ) from accumulate ( e: ProcessNodeLeftEvent( nodeInstance.nodeName == &quot;CursorRotationNode&quot; ) over window:size(10s), count(e) ) then drools.getWorkingMemory().getProcessInstance(ATDepartmentNotificationProcess).signalEvent( &quot;notify&quot; , &quot;cursor rotation threshold exceeded&quot; ); end
    85. 86. Event Processing - Context <ul><li>Specifies conditions on a group of events so they can be processed in related ways.
    86. 87. Events may/may not be relevant in certain contexts </li><ul><li>Example: Left-click command can only be issued over clickable objects on screen </li></ul><li>Types of contexts: </li><ul><li>Temporal: consists of time intervals and events within intervals
    87. 88. Spacial: takes account geospatial event info
    88. 89. State-oriented: depends on state of some external entity
    89. 90. Segmentation-oriented: partitions events by their attribute values (for example age >= 35) </li></ul></ul>
    90. 91. Drools Fusion
    91. 92. Drools Fusion – Events (1) <ul><li>Representing Raw Events:
    92. 93. Creating Derived Events: </li></ul>declare Direction @role( event ) @timestamp( timestamp ) @expires( 1m ) end declare Strength @role( event ) @expires( 1m ) end declare DoubleClick @role( event ) direction : Direction strength : Strength delay : int end rule detectDoubleClick when $direction : Direction( ) from entry-point &quot;Raw-Direction-Stream&quot; ... $strength : Strength( ) from entry-point &quot;Raw-Strength-Stream&quot; ... then DoubleClick dc = new DoubleClick($direction, $strength, 0); entryPoints[ &quot;Cursor-Control-Endpoint-Stream&quot; ). insert (dc); end
    93. 94. Drools Fusion – Events (2) <ul><li>Absence of events in proper order:
    94. 95. Negative patterns, waiting for events to happen: </li></ul>declare Click @role( event ) @expires( 1m ) delay : int end declare Drag @role( event ) @expires( 1m ) direction : Direction strength : Strength delay : int end rule invalidDragMotion when $dragEvent : Drag( ) from entry-point &quot;Cursor-Control-Endpoint-Stream&quot; not ( Click( this before $dragEvent ) from entry-point &quot;Cursor-Control-Endpoint-Stream&quot; ) // click starts drag event then # notify user+system of invalid command ... end rule lenientDragMotion when $dragEvent : Drag( ) from entry-point &quot;Cursor-Control-Endpoint-Stream&quot; not ( Click( this before $dragEvent ) from entry-point &quot;Cursor-Control-Endpoint-Stream&quot; ) Click( this after[0s,5s] $dragEvent ) from entry-point &quot;Cursor-Control-Endpoint-Stream&quot; then # allow drag motion... ... end
    95. 96. Drools Fusion – Event Streams <ul><li>Drools compiler gathers all entry-point declarations and exposes them through the session API.
    96. 97. Engine manages all scoping and synchronizations behind the scenes.
    97. 98. Event processing from specific streams in rules: </li></ul>StatefulKnowledgeSession ksession = ... ... WorkingMemoryEntryPoint rawDirectionEntryPoint = ksession.getWorkingMemoryEntryPoint( &quot;Raw-Direction-Stream&quot; ); WorkingMemoryEntryPoint rawStrengthEntryPoint = ksession.getWorkingMemoryEntryPoint( &quot;Raw-Strength-Stream&quot; ); … // Every time we get input from direction sensor do: rawDirectionEntryPoint.insert(direction); // Every time we get input from flow sensor do: rawStrengthEntryPoint.insert(strength); ... rule detectVerticalScrolling When ... $direction : Direction( ) from entry-point &quot;Raw-Direction-Stream&quot; $strength : Strength( ) from entry-point &quot;Raw-Strength-Stream&quot; ... then ...
    98. 99. Drools Fusion - Channels <ul><li>Eliminates runtime awareness of event consumers.
    99. 100. Dynamic registration of consumers in channels possible.
    100. 101. Example: </li><ul><li>a) Register Channel:
    101. 102. b) Channel Implementation:
    102. 103. Push derived events for distribution in rules: </li></ul></ul>StatefulKnowledgeSession ksession = kbase.newStatefulKnowledgeSession(config, null ); ksession.registerChannel( &quot;cursor-control&quot; , new CursorControlChannel()); ... public class CursorControlChannel implements Channel { private List<MouseControlService> controlServices ; @Override public void send(Object command) { for (MouseControlService service : controlServices ) { service.publishCommand(command); } } ... rule detectDoubleClick when ... then DoubleClick dc = new DoubleClick(); ... channels[ &quot;cursor-control&quot; ].send( dc ); end
    103. 104. Drools Fusion – Temporal Event Context (1) <ul><li>Enable Temporal Context by putting engine in “Stream mode”:
    104. 105. Choose a Clock type (realtime or pseudo for testing): </li></ul>... KnowledgeBaseConfiguration config = KnowledgeBaseFactory. newKnowledgeBaseConfiguration (); config.setOption( EventProcessingOption. STREAM ); KnowledgeBase kbase = KnowledgeBaseFactory. newKnowledgeBase (config); ... ... KnowledgeSessionConfiguration config = KnowledgeBaseFactory. newKnowledgeSessionConfiguration (); config.setOption( ClockTypeOption. get ( &quot;realtime&quot; ) ); StatefulKnowledgeSession ksession = kbase.newStatefulKnowledgeSession(config, null ); ...
    105. 106. Drools Fusion – Temporal Event Context (2) <ul><li>Stream Mode Enables: </li><ul><li>Notion of “flow of time”, concept of “now”
    106. 107. Automatic event lifecycle management
    107. 108. Support for sliding windows
    108. 109. Automatic rule delaying on absence of facts </li></ul><li>Reference Clock (Session Clock) implementations: </li></ul>
    109. 110. Drools Fusion – Temporal Event Context (3) <ul><li>Sliding Window support allows reasoning over a window of interest: </li><ul><li>Time
    110. 111. Length </li></ul></ul>rule &quot;Normalize cursor motion speed&quot; when CursorSpeedThreshold( $max : max ) Number( doubleValue > $max) from accumulate ( Strength( $val : value ) over window:length( 200 ), average( $val ) ) then // normalize the cursor speed end rule “Detect excessive Cursor Rotation” when Number( intValue > 15 ) from accumulate ( CursorRotation( ) over window:time(30s), count(e) ) then drools.getWorkingMemory().getProcessInstance(ATDepartmentNotificationProcess).signalEvent( &quot;notify&quot; , &quot;cursor rotation threshold exceeded&quot; ); end
    111. 112. Drools Fusion – Temporal Context Operators <ul><li>13 Temporal Operators </li></ul>
    112. 113. Drools Fusion – Temporal Context Operators
    113. 114. <ul>What next? -AT vision architecture </ul>
    114. 115. Questions? <ul><li>THANKS!
    115. 116. Project:
    116. 117. Reference Manual:
    117. 118. Blog:
    118. 119. Mailing Lists: </li></ul>