The document outlines key concepts in sensor fusion and complex event processing (CEP) relevant to robotics, emphasizing the need for integrating diverse sensory data to improve situational awareness and decision-making. It introduces principles for dynamic world modeling, the predict-match-update cycle for fusing sensor readings, and contrasts service-oriented architecture with event-driven architecture in software development. Additionally, it presents the Esper CEP engine as a robust open-source tool for real-time event processing and provides examples of event patterns and queries using its event processing language.

1. Keith McFarlane keith dot mcfarlane at gmail dot com

2. Agenda Sensor Fusion Concepts Complex Event Processing Concepts Esper CEP Engine EventBot Next Steps

3. Concepts

4. What is sensor fusion? “… combining…sensory data or data derived from sensory data from disparate sources such that the resulting information is in some sense better than would be possible when these sources were used individually.” 1 1 From Wikipedia article on Sensor Fusion Sensor readings (bumps, ultrasonic distance measurements, rotations, etc.) give only limited views of the world When taken together, they can be used to build a model of the world they are sensing Analogous to perception – the lens through which the robot views the world Once perception is realized, the robot can achieve situational awareness Finally, the robot can make predictions based on its model, act according to its goals, and measure the model’s accuracy

5. Why give sensor fusion special attention? All robotics systems require some level of sensor data correlation Requires special-purpose code – even if developed in a general, parameterized way, it is bound to “bake-in” some assumptions The application of open source software to the sensor fusion problem advances all at once No need to “re-invent the wheel” – address new problems instead of solved problems Share fusion advances with a broad community Benefit from the “wisdom of the crowd” Access to cleaner code over time Everyone contributes fixes and features, everyone benefits

6. Principles for Dynamic World Modeling 1 Express real-world primitives as set of properties Observation and model should be expressed in common coordinate system Observation and model should be expressed in common vocabulary Properties should include an explicit representation of uncertainty Primitives should be accompanied by a confidence factor. 1 Crowley et al

7. The Predict-Match-Update Cycle A model for fusing sensor readings into an actionable view of the world Transform Match Observations Update World Model Predict Sensors gather raw data about the world The data are put into some form that is compatible with the model The data are matched against / merged with previous predictions about the world The model is updated with the new data Predictions are reset based on updated model

8. Symbolic vs. Numeric Methods Symbolic fusion entails the development and maintenance of a schema-based world model Properties: attributes of entities in schema Coordinate system: position information should be consistent with world model Vocabulary: observations must to be mapped to schema entities and attributes Precision: attribute value variance may be provided or calculated using variance from predictions Confidence: overall trust in world model may be calculated and represented as a state Numerical sensor fusion methods apply for math-intensive domains Vision-based applications fall into this domain, for example

9. Concepts

10. Enterprise Origins: SOA vs. EDA Service-oriented Architecture (SOA) is a style of enterprise software development and governance Emphasizes discrete, distributed units of stateless software logic over monolithic applications Encourages information hiding Well supported by open source tools (Apache Tomcat, Glassfish) Event-driven Architecture (EDA) is a supplemental approach to enterprise software development and governance Accounts for the real-time nature of many business transactions Encourages discovery of business-relevant situations Not as well supported by open source (or other) tools, but support is growing Complex Event Processing (CEP) technologies apply very well to problems in the EDA domain SOA Tools aren’t always a fit

11. Complex Event Processing The acquisition, transformation, and correlation of streams of events for use in real-time or near-real-time applications Many common business scenarios can be interpreted as CEP problems Authentication: if ten password denial events occur inside of ten seconds, lock the user out Fraud detection : if a credit card is used at locations many miles apart within a short time period, alert the cardholder Most robotics applications could be described as CEP applications Robots acquire events via sensors, timers, web services, mobile phones, etc. Robots transform these into some common internal representation Robots determine what actions to take based on interpretations of event content, timing, coincidence, sequence, etc.

12. Events For our purposes, an event is an immutable historical record of some observed occurrence Historical: by definition, events have already happened Important to track time at the proper resolution Immutable: no changes allowed Generally, it would not be useful to modify events in real time Better to generate new events representing a reinterpretation of event meaning Observed: generated externally or through correlation Simple events represent some single occurrence Often useful to consider a variety of sources In enterprise applications, events generally originate from software services (applications, DB triggers,…)

13. Event conveyance vs. processing Sometimes these concepts become entangled – a frequent point of confusion in business integration circles Conveyance : the medium of event movement Local calls and callbacks – e.g., Java listeners Typical of the Observer design pattern Network calls – REST or SOAP Web Services, Java RMI, CORBA Message queues – JMS-based, MSMQ, IBM MQ Series Processing : the application of logic to a set of event streams Engines: Esper, Coral8, Avaya Event Processor Goal: glean information from the chaos of event traffic

14. Complex Events Complex events represent the detection of a related set of other simple or complex events Often used to identify detected “situations” Event Queue 1 Event Queue 2 Event Queue 3 E1 E1 E1 E1 E1 E2 E2 E2 E2 E3 C1 Detect event count C2 Detect event timing C3 Detect event coincidence

15. Scenario: Detecting Cheap Gas Rule: My Location within one mile of gas station with price under $2.75 and gas tank low Sends real-time changes in gas prices for a set of gas stations Gas Prices Service Event Processor “ Cheap Gas Nearby” Alert New Price New Price New Price New Price New Price New Price New Location New Location New Location New Location New Location Mobile phone GPS Location Service New Location Gas Level SMS

16. Event Pattern Abstraction A complex event that represents, denotes, or summarizes the events of which it is composed Some examples: A “Completed Loan Origination” event might represent the “New Loan Request,” “Credit Verified,” and “Final Approval” events in sequence A “Human in Path” event might be generated by a car’s navigation system for coincident infrared, radar and vision events Event processors generally support the ability to create new event streams based upon a detected set of conditions Streams of events may be cascaded for higher levels of situational understanding Some event processors support the ability to “drill down” to the events that triggered the abstract event

17. Sensor Fusion and CEP Conceptual Relationships Sensor Fusion CEP Collects observations Receives Events Requires normalized observation schema Requires normalized event data Dynamic world modeling Event stream maintenance Matching Event correlation/pattern detection Situational awareness Event pattern abstraction Symbolic fusion Event schema Used to model the world using limited data set and predictive feedback model Used to detect specific situations

19. Esper: Open Source CEP Esper is an open source CEP platform with a rich toolset for processing streaming events Open source version is very capable – all core functions present Event query language Pattern recognizer Support for JavaBeans, Maps, and XML documents High throughput – up to 500,000 events/sec. for a two-processor system 1 Extensible – not difficult to add new input and output event connectors or event types Commercial version available that includes HA features http://docs.codehaus.org/display/ESPER/Esper+performance

20. Esper Container Architecture Events Source: http://www.espertech.com Esper Lightweight CEP Container Event Stream Connectors and Adapters Historical Data Access Layer Container Core Event Query and Pattern Language Executive Event Query and Pattern Language Esper Engines Statements Statements Statements Statements Statements POJOs Output Adapters

21. EPL: Event Processing Language Used to specify the event processor’s behavior when confronted with events or patterns of events SQL-like in appearance – helps SQL programmer’s see analogous relationships May also confuse the issue for those who have trouble “letting go” Rather than tables, EPL statements are run against “streams SQL idea of a row is replaced by the “event” concept It is possible to include data from JDBC data sources to augment event processing For example, lookup tables could be used to retrieve a customer’s current balance based on a customer ID delivered with an event

22. EPL: Familiar Concepts from SQL Clause Description SELECT Indicates an that events should be selected based on a set of criteria. Can select on insert stream, remove stream, or both. FROM Defines the event stream(s) or named window(s) from which events should be selected WHERE Allows filtering of events based on a wide range of criteria GROUP BY Allows stratification of events by a set of characteristics HAVING Filters groups specified by the GROUP BY clause INSERT INTO Creates new events in the given named window, allowing events to cascade through the system

23. An Example EPL Statement The “select *” statement preamble indicates an event retrieval operation inclusive of all event properties “from Withdrawal” names the event stream from which to pull events “amount>=200” filters the events that will be accepted into the stream “win:length(5)” defines the stream as containing the last five matching events inserted select * from BatteryChargeEvent(charge<400).win:length(5)

24. Event Streams Defined by EPL statement FROM clause, and may be one of two types: filter or pattern Filter-based event streams apply constraints to incoming events, limiting stream content Example: select * from DistanceEvent(distance>0).win:length(10) Supports standard comparators (=, !=, >, <, >=, <=) and ranges (between, in, not) Pattern-based event streams look for the specified pattern of events Example: select * from pattern [every left=LeftBumpEvent or every right=RightBumpEvent].win:time(5 sec)

25. Views Used to derive or aggregate data from an event stream May be “staggered” (chained) within a single EPL statement Example: select * from DistanceEvent(distance>0).win:length(10).std:groupby(direction) Some views delivered with Esper: win:length win:length_batch win:time win:time_batch win:time_length_batch win:time_accum win:ext_timed ext:sort_window ext:time_order std:unique std:groupby win:firsttime

27. Hardware El Cheapo Laptop Roo232 USB-to-Serial Cable Roomba Sage Reversed Mac Mini-DIN 8 M2M Cable

28. Roomba Connector Home-grown Java library that speaks Roomba SCI Multi-threaded for better processor utilization Command pattern applied for command encapsulation Allow for flexibility in both queuing and macro command building Commands are queued and processed asynchronously Sensors polled with specified frequency Current default is 10 polls/second Polling results in generation of event classes that are sent to subscribed Java listeners Relevant event parameters bundled in event object

29. The Command Pattern

30. EventBot Data Flow vs. Fusion Model Roomba Events Commands Roomba Connector Event Transformer Esper CEP Event Listeners Transform Match Observations Update World Model Predict World Model

31. Code Walkthrough Example: Governing speed via rotation sensors – a simple sensor fusion example Fuses multiple readings into better information about the environment Uses an average of the last N readings to match speed and adjust power if needed Uses this data to adjust in continuous feedback loop Elements: Connector Coding EPL Statements Listener Coding Runtime

32. Next Steps Transducer Markup Language (TML) adapter Allows wide variety of sensor types to be normalized More complex fusion example More sensors – simple vision? Ultrasonic? RSS event feed RFID WS-Eventing adapter Open source availability of tools

33. email: keith dot mcfarlane at gmail dot com

34. References Links Esper home http://esper.codehaus.org http://www.complexevents.com http://www.espertech.com http://en.wikipedia.org/wiki/Sensor_fusion http://en.wikipedia.org/wiki/Gang_of_Four_(software) Papers Principles and Techniques for Sensor Data Fusion http://www-prima.imag.fr/jlc/papers/SigProc-Fusion.pdf Crowley et al. Overview of CEP Concepts http://complexevents.com/wp-content/uploads/2008/07/overview-of-concepts-of-cep.pdf David Luckham