Your SlideShare is downloading. ×
Open Source Event Processing for Sensor Fusion Applications
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Open Source Event Processing for Sensor Fusion Applications


Published on

This deck is from my Robodev 2008 presentation on sensor fusion using open source technologies. It includes a detailed description of my homebrew sensor fusion Roomba, which I demonstrated at the …

This deck is from my Robodev 2008 presentation on sensor fusion using open source technologies. It includes a detailed description of my homebrew sensor fusion Roomba, which I demonstrated at the conference.

Published in: Technology

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Keith McFarlane keith dot mcfarlane at gmail dot com
  • 2. Agenda
    • Sensor Fusion Concepts
    • Complex Event Processing Concepts
    • Esper CEP Engine
    • EventBot
    • Next Steps
  • 3. Concepts
  • 4. What is sensor fusion?
    • “… combining…sensory data or data derived from sensory data from disparate sources such that the resulting information is in some sense better than would be possible when these sources were used individually.” 1
    1 From Wikipedia article on Sensor Fusion
    • Sensor readings (bumps, ultrasonic distance measurements, rotations, etc.) give only limited views of the world
    • When taken together, they can be used to build a model of the world they are sensing
      • Analogous to perception – the lens through which the robot views the world
    • Once perception is realized, the robot can achieve situational awareness
    • Finally, the robot can make predictions based on its model, act according to its goals, and measure the model’s accuracy
  • 5. Why give sensor fusion special attention?
    • All robotics systems require some level of sensor data correlation
      • Requires special-purpose code – even if developed in a general, parameterized way, it is bound to “bake-in” some assumptions
    • The application of open source software to the sensor fusion problem advances all at once
      • No need to “re-invent the wheel” – address new problems instead of solved problems
      • Share fusion advances with a broad community
        • Benefit from the “wisdom of the crowd”
      • Access to cleaner code over time
        • Everyone contributes fixes and features, everyone benefits
  • 6. Principles for Dynamic World Modeling 1
    • Express real-world primitives as set of properties
    • Observation and model should be expressed in common coordinate system
    • Observation and model should be expressed in common vocabulary
    • Properties should include an explicit representation of uncertainty
    • Primitives should be accompanied by a confidence factor.
    1 Crowley et al
  • 7. The Predict-Match-Update Cycle A model for fusing sensor readings into an actionable view of the world Transform Match Observations Update World Model Predict Sensors gather raw data about the world The data are put into some form that is compatible with the model The data are matched against / merged with previous predictions about the world The model is updated with the new data Predictions are reset based on updated model
  • 8. Symbolic vs. Numeric Methods
    • Symbolic fusion entails the development and maintenance of a schema-based world model
      • Properties: attributes of entities in schema
      • Coordinate system: position information should be consistent with world model
      • Vocabulary: observations must to be mapped to schema entities and attributes
      • Precision: attribute value variance may be provided or calculated using variance from predictions
      • Confidence: overall trust in world model may be calculated and represented as a state
    • Numerical sensor fusion methods apply for math-intensive domains
      • Vision-based applications fall into this domain, for example
  • 9. Concepts
  • 10. Enterprise Origins: SOA vs. EDA
    • Service-oriented Architecture (SOA) is a style of enterprise software development and governance
      • Emphasizes discrete, distributed units of stateless software logic over monolithic applications
      • Encourages information hiding
      • Well supported by open source tools (Apache Tomcat, Glassfish)
    • Event-driven Architecture (EDA) is a supplemental approach to enterprise software development and governance
      • Accounts for the real-time nature of many business transactions
      • Encourages discovery of business-relevant situations
      • Not as well supported by open source (or other) tools, but support is growing
      • Complex Event Processing (CEP) technologies apply very well to problems in the EDA domain
        • SOA Tools aren’t always a fit
  • 11. Complex Event Processing
    • The acquisition, transformation, and correlation of streams of events for use in real-time or near-real-time applications
      • Many common business scenarios can be interpreted as CEP problems
        • Authentication: if ten password denial events occur inside of ten seconds, lock the user out
        • Fraud detection : if a credit card is used at locations many miles apart within a short time period, alert the cardholder
      • Most robotics applications could be described as CEP applications
        • Robots acquire events via sensors, timers, web services, mobile phones, etc.
        • Robots transform these into some common internal representation
        • Robots determine what actions to take based on interpretations of event content, timing, coincidence, sequence, etc.
  • 12. Events
    • For our purposes, an event is an immutable historical record of some observed occurrence
      • Historical: by definition, events have already happened
        • Important to track time at the proper resolution
      • Immutable: no changes allowed
        • Generally, it would not be useful to modify events in real time
        • Better to generate new events representing a reinterpretation of event meaning
      • Observed: generated externally or through correlation
        • Simple events represent some single occurrence
    • Often useful to consider a variety of sources
      • In enterprise applications, events generally originate from software services (applications, DB triggers,…)
  • 13. Event conveyance vs. processing
    • Sometimes these concepts become entangled – a frequent point of confusion in business integration circles
    • Conveyance : the medium of event movement
      • Local calls and callbacks – e.g., Java listeners
        • Typical of the Observer design pattern
      • Network calls – REST or SOAP Web Services, Java RMI, CORBA
      • Message queues – JMS-based, MSMQ, IBM MQ Series
    • Processing : the application of logic to a set of event streams
      • Engines: Esper, Coral8, Avaya Event Processor
      • Goal: glean information from the chaos of event traffic
  • 14. Complex Events
    • Complex events represent the detection of a related set of other simple or complex events
      • Often used to identify detected “situations”
    Event Queue 1 Event Queue 2 Event Queue 3 E1 E1 E1 E1 E1 E2 E2 E2 E2 E3 C1 Detect event count C2 Detect event timing C3 Detect event coincidence
  • 15. Scenario: Detecting Cheap Gas Rule: My Location within one mile of gas station with price under $2.75 and gas tank low Sends real-time changes in gas prices for a set of gas stations Gas Prices Service Event Processor “ Cheap Gas Nearby” Alert New Price New Price New Price New Price New Price New Price New Location New Location New Location New Location New Location Mobile phone GPS Location Service New Location Gas Level SMS
  • 16. Event Pattern Abstraction
    • A complex event that represents, denotes, or summarizes the events of which it is composed
      • Some examples:
        • A “Completed Loan Origination” event might represent the “New Loan Request,” “Credit Verified,” and “Final Approval” events in sequence
        • A “Human in Path” event might be generated by a car’s navigation system for coincident infrared, radar and vision events
      • Event processors generally support the ability to create new event streams based upon a detected set of conditions
        • Streams of events may be cascaded for higher levels of situational understanding
      • Some event processors support the ability to “drill down” to the events that triggered the abstract event
  • 17. Sensor Fusion and CEP Conceptual Relationships Sensor Fusion CEP Collects observations Receives Events Requires normalized observation schema Requires normalized event data Dynamic world modeling Event stream maintenance Matching Event correlation/pattern detection Situational awareness Event pattern abstraction Symbolic fusion Event schema Used to model the world using limited data set and predictive feedback model Used to detect specific situations
  • 18.  
  • 19. Esper: Open Source CEP
    • Esper is an open source CEP platform with a rich toolset for processing streaming events
      • Open source version is very capable – all core functions present
        • Event query language
        • Pattern recognizer
        • Support for JavaBeans, Maps, and XML documents
        • High throughput – up to 500,000 events/sec. for a two-processor system 1
      • Extensible – not difficult to add new input and output event connectors or event types
      • Commercial version available that includes HA features
  • 20. Esper Container Architecture Events Source: Esper Lightweight CEP Container Event Stream Connectors and Adapters Historical Data Access Layer Container Core Event Query and Pattern Language Executive Event Query and Pattern Language Esper Engines Statements Statements Statements Statements Statements POJOs Output Adapters
  • 21. EPL: Event Processing Language
    • Used to specify the event processor’s behavior when confronted with events or patterns of events
      • SQL-like in appearance – helps SQL programmer’s see analogous relationships
        • May also confuse the issue for those who have trouble “letting go”
      • Rather than tables, EPL statements are run against “streams
      • SQL idea of a row is replaced by the “event” concept
      • It is possible to include data from JDBC data sources to augment event processing
        • For example, lookup tables could be used to retrieve a customer’s current balance based on a customer ID delivered with an event
  • 22. EPL: Familiar Concepts from SQL Clause Description SELECT Indicates an that events should be selected based on a set of criteria. Can select on insert stream, remove stream, or both. FROM Defines the event stream(s) or named window(s) from which events should be selected WHERE Allows filtering of events based on a wide range of criteria GROUP BY Allows stratification of events by a set of characteristics HAVING Filters groups specified by the GROUP BY clause INSERT INTO Creates new events in the given named window, allowing events to cascade through the system
  • 23. An Example EPL Statement
    • The “select *” statement preamble indicates an event retrieval operation inclusive of all event properties
    • “from Withdrawal” names the event stream from which to pull events
    • “amount>=200” filters the events that will be accepted into the stream
    • “win:length(5)” defines the stream as containing the last five matching events inserted
    select * from BatteryChargeEvent(charge<400).win:length(5)
  • 24. Event Streams
    • Defined by EPL statement FROM clause, and may be one of two types: filter or pattern
    • Filter-based event streams apply constraints to incoming events, limiting stream content
      • Example:
        • select * from DistanceEvent(distance>0).win:length(10)
        • Supports standard comparators (=, !=, >, <, >=, <=) and ranges (between, in, not)
    • Pattern-based event streams look for the specified pattern of events
      • Example:
        • select * from pattern [every left=LeftBumpEvent or every right=RightBumpEvent].win:time(5 sec)
    • Used to derive or aggregate data from an event stream
    • May be “staggered” (chained) within a single EPL statement
      • Example:
        • select * from DistanceEvent(distance>0).win:length(10).std:groupby(direction)
    • Some views delivered with Esper:
    win:length win:length_batch win:time win:time_batch win:time_length_batch win:time_accum win:ext_timed ext:sort_window ext:time_order std:unique std:groupby win:firsttime
  • 26.  
  • 27. Hardware El Cheapo Laptop Roo232 USB-to-Serial Cable Roomba Sage Reversed Mac Mini-DIN 8 M2M Cable
  • 28. Roomba Connector
    • Home-grown Java library that speaks Roomba SCI
    • Multi-threaded for better processor utilization
    • Command pattern applied for command encapsulation
      • Allow for flexibility in both queuing and macro command building
    • Commands are queued and processed asynchronously
    • Sensors polled with specified frequency
      • Current default is 10 polls/second
    • Polling results in generation of event classes that are sent to subscribed Java listeners
      • Relevant event parameters bundled in event object
  • 29. The Command Pattern
  • 30. EventBot Data Flow vs. Fusion Model Roomba Events Commands Roomba Connector Event Transformer Esper CEP Event Listeners Transform Match Observations Update World Model Predict World Model
  • 31. Code Walkthrough
    • Example: Governing speed via rotation sensors – a simple sensor fusion example
      • Fuses multiple readings into better information about the environment
        • Uses an average of the last N readings to match speed and adjust power if needed
      • Uses this data to adjust in continuous feedback loop
      • Elements:
        • Connector Coding
        • EPL Statements
        • Listener Coding
      • Runtime
  • 32. Next Steps
    • Transducer Markup Language (TML) adapter
      • Allows wide variety of sensor types to be normalized
    • More complex fusion example
      • More sensors – simple vision? Ultrasonic?
      • RSS event feed
      • RFID
      • WS-Eventing adapter
    • Open source availability of tools
  • 33. email: keith dot mcfarlane at gmail dot com
  • 34. References
    • Links
      • Esper home
    • Papers
      • Principles and Techniques for Sensor Data Fusion
        • Crowley et al.
      • Overview of CEP Concepts
        • David Luckham