The document explains complex event processing (CEP), emphasizing its ability to analyze and respond to real-time data streams, often measured in nanoseconds. It highlights various application use cases across industries such as retail and finance, and differentiates between terminology like event stream processing (ESP) and simple event processing (SEP). Additionally, it outlines CEP architecture, event types, processing languages, and key considerations for developing high-performance CEP applications.

1. Complex Event Processing
with Esper
Real-time Event Stream & Complex Event Processing
Ted Won
2011. 11. 12.
http://tedwon.com

3. Complex Event Processing

5. Nanosecond
● CEP 세계의 측정 단위는 nanosecond
● A nanosecond (ns) is one billionth of a second

6. Nanosecond
1 second is...
1,000 ms
1,000,000 us
1,000,000,000 ns

7. Concept of CEP
● 교회에서...
● 벨이 울린다.
● 턱시도를 입은 남자와 꽃을 든 여자가 함께 걷는다.
● 두 사람 위로 꽃 잎이 날리고 푹죽이 터진다.

8. Concept of CEP
CEP 기술은 이러한 Complex한 Event를 참조하여
결혼식이라는 것을 알아 차린다.

9. Imagination Beyond Reality
● Imaginary CEP Use Cases

10. Imagination Beyond Reality
● E-mart에서...
● 고객들이 카트를 끌고 매장을 돌아 다닌다.
● 카트의 이동 경로와 패턴을 알 수 없을까?
● 계산대에서 대기하는 고객들 상태 파악 후 실시간 조치
● 카트 + 멤버십 카드라면 카트에 달린 모니터를 통해
개인화된 실시간 추천/쿠폰 발행
● 카트에 물건을 담으면 자동으로 계산

11. Imagination Beyond Reality
● 전국의 이마트 매장이 130개...
● 엄청난 양의 데이터를 어찌 실시간으로 처리한단 말인가??
● Starbucks 매장에서...
● 파주 프리미엄 아웃렛 매장에서...
● 교통 관리에서
실시간으로 교통 상태/패턴을 인지하고 적절한 조치

12. Imagination Beyond Reality
● Algorithmic Stock-Trading
● Applying CEP to the RHQ alert feature
as a Operational Intelligence (OI) solution

13. What is CEP?

14. What is CEP
● Complex Event Processing
● CEP는 Event 중심의 실시간 대용량 데이터 처리 기술
● CEP는 실시간 대용량 Event 처리 기술
● CEP는 EDA(Event-driven architecture) 기반 시스템
● Mutliple event stream에서 발생하는 패턴 감지 기술

15. What is CEP
● Identifying the most meaningful events from event cloud
● Real-time computation and pattern detection over event streams
● Response가 아닌 Reaction을 하는 시스템
● http://en.wikipedia.org/wiki/Complex_Event_Processing

16. What is CEP
● Real-time Processing
● Intelligence System
● Real-time Business Intelligence
● Event Stream Intelligence

17. What is CEP
● Extreme Transaction Processing
● Publish/Subscribe Model
● Loosley-coupled
● Asynchronous Style
Source: Event Processing in Action

18. Open Source CEP
● EsperTech Esper (Stream-oriented)
● JBoss Drools Fusion (Rule-oriented)

19. CEP Language Style
● Strem-oriented
○ EsperTech Esper
○ MS StreamInsight
● Rule-oriented
○ JBoss Drools Fusion
○ TIBCO TIBCO BusinessEvents

20. What is EDA?

21. What is EDA
● Event Driven Architecture
● EDA는 event의 production, detection, consumption과
reaction 하기위한 구조를 정의한 software architecture
Source: Event Processing in Action

22. What is EDA
● More Loosely-coupled than SOA
● Event 표준 스펙 논의가 OASIS에서
Common Base Events라는 이름으로 진행 중
● Response가 아닌 Reaction을 하는 시스템
● 선처리 후저장 패러다임
● http://en.wikipedia.org/wiki/Event-driven_architecture

23. Classical Request-response Architecture
● Synchronous interactions
● Request에 대한 Response를 하는 구조
● 선저장 후처리 패러다임

24. Event Processing Styles
● SEP - Simple Event Processing
○ Single event based
○ 상태가 없는 Event를 처리하는 케이스
○ JMS(Java Message Service)가 대표적인 예
○ ESB(Enterprise Service Bus)
● ESP - Event Stream Processing
● CEP - Complex Event Processing

25. Event Processing Application
Source: Event Processing in Action

26. ESP vs. CEP
● ESP - Event Stream Processing
○ Monitor streams of event data, analyse those events, and act
upon opportunities
○ Volume weighted average of Google stock over the last
(moving) 30min
● CEP - Complex Event Processing
○ Detecting patterns among events
○ If this Google VWAP increased more than 5% two times
followed by Yahoo! VWAP decreased more than 10% then…

27. SOA vs. EDA
● SOA
○ RPC calls
○ Point-to-Point communication
○ Request and reply
○ Vertical interaction
● EDA
○ Message based
○ Less coupled than SOA
○ Publish and subscribe
○ Horizontal communication between tiers in process chain

28. Point-To-Point

29. Publish-and-Subscribe

30. What is an Event?

31. What is an Event
● Event란 무엇인가?
● 무엇을 Event로 정의 할 수 있는가?

32. What is an Event
● Something that happens
● 실제로 발생한 사건, 일, 메세지
● 상태의 변경

33. What is an Event
● An event is an immutable record of a past occurrence
of an action or state change.
● Event properties capture the useful information for an event.

34. What is an Event
Event란 어떤 것이 변화한 것이다!

35. What is an Event
● 변화한다는 것은 어떠한 상태를 가지고 있다는 의미
● 상태를 가지고 변화하는 모든 것이 Event로 정의될 수 있다.
● 이 세상의 모든 것은 변화한다.
● 변화량을 정량적으로 측정가능하도록 정의해야함
● Event를 추상화하여 표현한 것을 Event type이라함

36. What is an Event
● 상태가 없는 Event를 처리하는 경우도 존재
● SEP(Simple Event Processing)의 경우 존재
● JMS가 하나의 예

37. Event Examples
● Stock tick
● 비밀번호 변경
● 10초마다 실내 온도 측정
● 서비스 응답 시간
● 고객이 들고 다가오는 Starbucks membership cards with RFID
● 고객이 끌고 다니는 E-mart carts with RFID

38. Events vs. Messages
● Event는 때로는 Message로 표현됨
● Message는 단순히 Event의 다른 표현이라는 측면도 존재
● Event의 상태라는 속성이 Message에서는 어떤 내용(contents)이
라는 차이
● EDA에 속한 개념으로써 SEP(Simple Event Processing)의 한 종류
인 JMS와 같은 기술에서 사용하는 전문 용어라는 측면

39. What is Event Type?

40. What is Event Type
● Event Representation Schema
● 개념적인 Event를 실제 프로그래밍적으로 표현하는 방법
● Event의 개념적 표현을 Event Schema라 할 수 있는데
● Event Schema의 표현을 Event Type 으로 한다.
● DBMS의 테이블 스키마와 유사한 개념
● 대표적인 Data Format을 이용하여 정의 할 수 있다.

41. Event Type Data Format
● POJO
● Map
● XML
● JSON
● CSV

42. What is Event Stream?

43. What is Event Stream
● 시간의 순서대로 연속되는 Event
● 시작과 끝이 없는 Event의 연속된 흐름

44. What is Real-time?

45. What is Real-time
Real-time system은 다음의 조건들을 충족해야 합니다
● Low latency
● Regularities in Response time
● Providing predictable performance

46. CEP Usages
● Algorithmic Stock-Trading
● Operational Intelligence(OI) solutions
● Real-time analytics
● Predictive analytics
● Event-driven business process management
● BAM - Business Activity Monitoring
● BI - Business intelligence
● BRMSs - Business rule management systems
● Network and System Management
● MOM- Message-oriented middleware
● Stream computing
● ETL(Extract, Transform, Load)

47. CEP Products

48. Esper

49. EsperTech Esper
● Event Stream and Complex Event Processing platform
● Convergence of ESP and CEP
● Open Source ESP/CEP Engine
● Event Processing Language (EPL)
● Java and .Net

50. EsperTech Esper
● High Throughput, Low Latency
● Highly Available (EsperHA)
● Rich Multi-Window GUI (Enterprise Edition)
● Real-time Displays (Enterprise Edition)
● Enterprise *ilities (Enterprise Edition)

51. EsperTech Esper
● Usability
● Maintainability
● Scalability
● Availability and Reliability
● Extensibility
● Security
● Portability

52. EsperTech Esper
● Technical Support (Enterprise Edition)
● Lightweight & Embeddable
● No Runtime Support
● GNU GPL(General Public License) v2

53. Runtime: Esper + Java EE platform

54. Runtime: Esper + OSGi

55. Esper Documentation
● Esper Documentation Home
○ http://goo.gl/isNb6
● Esper 4.4.0 Reference documentation
○ http://goo.gl/5TXel
● EsperIO 4.4.0 Reference documentation
○ http://goo.gl/76uOe

56. Esper Architecture

57. Esper EDAS

58. CEP Architecture

59. CEP Architecture

60. CEP Architecture

61. Sample CEP Architecture

62. Real-time Log Aggregator & CEP

63. Input / Output Adapters
● Binary File Reader Input
● Binary File Writer Output
● CSV File Reader Input
● CSV File Writer Output
● CSV Socket Reader Input
● CSV Socket Writer Output
● Email (IMAP) Reader Input
● Email Sender
● HA Heartbeat Input
● IP Packet Capture /HTTP
● IP Packet Capture /IRC

64. Input / Output Adapters
● IP Packet Capture /POP3
● IP Packet Capture /SMTP
● IRC Output adapter
● Jabber Output Adapter
● JDBC
● JMS Publish/Subscribe*
● Log Output
● Microsoft Excel RTD*
● Multicast UDP Input and Output*
● Quartz Enterprise Scheduler Input*

65. Input / Output Adapters
● Regular Expression File Reader Input
● Regular Expression Socket Reader Input
● RSS
● Twitter
● Unicast UDP Input Adapter
● XML File Writer Output
● XML over HTTP Writer Output
● Hadoop HDFS Input Adapter
● Hadoop HDFS Output Adapter

66. Esper EPL

67. Event Processing Language
select symbol, symbolDesc from OrderEvent.win:time(30 min)

68. Event Processing Language
● No standard
● SQL-like language
● Data windows
● Pattern matching
● Continuous query

69. Event Processing Language

70. Event Processing Model

71. Event Processing Model

72. Event Processing Model

73. Event Processing Model

74. Event Processing Language
● Event filtering
● Sliding windows and aggregation
● Grouped windows and output rate limiting
● Joins and Outer Joins
● Historical or reference data
● Subquery

75. Event Example: The RFID Domain
● RFID: Radio Frequency Identification
● LocationReport
○ asset id - a unique identifier of the tagged asset
○ x - the x location value
○ y - the y location value
○ zone - derived from x and y
● Use cases
○ “When a given group of assets are not moving together from
zone to zone, then...”
○ “When a given asset stays too long in the same zone, then...”

76. EPL Statement Sample
● If a given set of assets are not moving together from zone to zone,
● then alert
insert into CountZone
select zone, count(*) as cnt
from LocationReport.std:unique('assetId')
where assetId in (1, 2, 3)
group by zone

77. EPL Statement Sample
select Part.zone from pattern [
every Part=CountZone(cnt in (1, 2)) ->
( timer:interval(1 min) and not
CountZone(zone=Part.zone, cnt in (0, 3))
)
]

78. Listener and Engine
// Get engine instance and register statement
EPServiceProvider engine = EPServiceProviderManager.getDefaultProvider();
EPStatement statement = engine.getEPAdministrator().createEQL("...");
// Attach a listener
statement.addListener(new UpdateListener() {
public void update(EventBean[] newEvents, EventBean[] oldEvents) {
// Handle complex event
...
}
});

79. Sending events
// Get the same engine instance
EPServiceProvider engine = EPServiceProviderManager.
getDefaultProvider();
EPRuntime runtimeEngine = engine.getRuntime();
...
LocationReport event = new LocationReport(assetId,x,y,zone);
runtimeEngine.sendEvent(event);

80. Event filtering
// Filter for location report by location rectangle
select * from LR(x in [4:10], y in [6:12])

81. Sliding windows and aggregation
// Count all assets reporting zone 10 in last 30 sec
select count(*) from LR(zone=10).win:time(30 sec)

82. Grouped windows and output rate limiting
// Get X location range of the last 100 events per zone
// every 1 minute
select zone, min(x), max(x)
from LR.std:groupby('zone').win:length(100 events)
output every 1 min

83. Continuous joins
// Fire when any asset enters zone 2 before zone 1
select Zone2.assetId
from
LR(zone=2).win:time(1 day) Zone2
left outer join
LR(zone=1).win:time(1 day) Zone1
on Zone1.assetId = Zone2.assetId
where Zone1.assetId is null

84. Historical or reference data
// Alert when we hit the minimum inventory
// for a given zone
select zone, count(*)
from LR.std:unique(assetId) as lr,
sql:db[select mini from Minimum where zone=${lr.zone}]
having mini < count(*)

85. Subquery
// Notify when the asset id is not known
select * from LR
where assetId not in
(select assetId
from KnownAsset.std:unique('assetId'))

86. Data window views
● win:length
● win:length_batch
● win:time
● win:time_batch
● win:time_length_batch
● win:time_accum
● win:ext_timed
● ext:sort_window
● ext:time_order

87. Data window views
● std:unique
● std:groupwin
● std:lastevent
● std:firstevent
● std:firstunique
● win:firstlength
● win:firsttime

88. Statistics views
● std:size
● stat:uni
● stat:linest
● stat:correl
● stat:weighted_avg

89. CEP Application Development Steps
● Define Event
● Define Event Type
● Define EPL Statements
● Publish EPL Statements
● Subscribe Output Adapter
● Attache Input Adapter

90. CEP Application Development Points
● Runtime 환경 결정
● Resilience를 지원 여부
● Zero-downtime event processing
● Support dynamic EPL Statement updates
● Input/Output Adapter에서 bottleneck이 발생
● 그러므로 Adapter의 성능이 중요
● Event의 타입과 용량에 따라서 성능에 영향
● Input Adapter의 통신 protocol 결정 중요
● CEP Engine Performance Tuning

91. CEP Application Key Considerations
● Low Latency
● High Throughput
● High Availability - Failover
● Zero-downtime

92. Performance
● Throughput
○ events/sec
● Latency
○ us/event
● 성능 측정은 throughput을 고정하고 latency statistics로 측정
● http://docs.codehaus.org/display/ESPER/Esper+performance

93. Latency statistics
● Throughput 995900 (active 0 pending 0)
---Stats - engine (unit: ns)
Avg: 2656 #9936457
0< 5000: 99.78% 99.78% #9914178
5000 < 10000: 0.20% 99.98% #20120
10000 < 15000: 0.01% 99.98% #646
15000 < 20000: 0.00% 99.99% #69
20000 < 25000: 0.00% 99.99% #36
25000 < 50000: 0.00% 99.99% #56
50000 < 100000: 0.00% 99.99% #26
100000 < 500000: 0.00% 99.99% #189
500000 < 1000000: 0.00% 99.99% #2
1000000 < 2500000: 0.00% 99.99% #68

94. Thanks
Q&A

95. Thanks
Q&A

96. Practice!!
http://goo.gl/5ijAu

97. JBoss Community (http://www.jboss.org)
Korea JBoss User Group (http://cafe.naver.com/jbossug)