JavaOne 2008 - TS-6048 - Complex Event Processing at Orbitz

COMPLEX EVENT PROCESSING AT ORBITZ WORLDWIDE Matt O’Keefe, Senior Architect Doug Barth, Technical Lead TS-6048

How to make pager duty suck less

$10.8 Billion in Gross Bookings in 2007

Dozens of Apps, Hundreds of VMs and Thousands of Services

The Need for Abstraction Webapp Travel Business Services Switching Services abstraction Transaction Services Suppliers

Complex Event Processing is…

…about sensing and responding to threats and opportunities in real time.

The Big Picture Event Processors Monitored Apps Operations Center Graphite JMX, ssh ERMA SNMP Portal

ERMA

E xtremely R eusable M onitoring A PI

The Monitor Interface

Using EventMonitors protected void doValidate(RequestContext context, Object formObject, Errors errors) throws Exception { super.doValidate(context, formObject, errors); if (errors.hasErrors()) { EventMonitor validationMonitor = new EventMonitor("ValidationErrors"); validationMonitor.set("errors", errors.getAllErrors()); validationMonitor.fire(); } }

The CompositeMonitor Interface

Using TransactionMonitors public Hotel findHotelById(Long id) { TransactionMonitor monitor = new TransactionMonitor(getClass(), "findHotelById"); monitor.set(“id”, id); try { Hotel hotel = em.find(Hotel.class, id); monitor.succeeded(); return hotel; } catch (RuntimeException e) { monitor.failedDueTo(e); throw e; } finally { monitor.done(); } }

TransactionMonitorTemplate public void cancelBooking(final Long id) { TransactionMonitorTemplate.INSTANCE.doInMonitor( getClass(), "cancelBooking", new TransactionMonitorCallback() { public Object doInMonitor(TransactionMonitor monitor) { monitor.set("id", id); Booking booking = em.find(Booking.class, id); if (booking != null) { em.remove(booking); } return null; } }); }

Interceptors public interface HandlerInterceptor { boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception; void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception; void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception; }

Listeners public interface FlowExecutionListener { public void requestSubmitted(RequestContext context); public void requestProcessed(RequestContext context); public void sessionStarting(RequestContext context, FlowDefinition definition, MutableAttributeMap input); public void sessionCreated(RequestContext context, FlowSession session); public void sessionStarted(RequestContext context, FlowSession session); public void eventSignaled(RequestContext context, Event event); public void stateEntering(RequestContext context, StateDefinition state) throws EnterStateVetoException; public void stateEntered(RequestContext context, StateDefinition previousState, StateDefinition state); public void paused(RequestContext context, ViewSelection selectedView); public void resumed(RequestContext context); public void sessionEnding(RequestContext context, FlowSession session, MutableAttributeMap output); public void sessionEnded(RequestContext context, FlowSession session, AttributeMap output); public void exceptionThrown(RequestContext context, FlowExecutionException exception); }

Annotations @Monitored public interface UnderpantsGnomes { void collectUnderpants(); void ?(); void profit(); }

Spring/AspectJ <aop:config> <aop:aspect id="transactionMonitorActionAspect" ref="transactionMonitorActionAdvice"> <aop:pointcut id="transactionMonitorActionPointcut“ expression=" target(org.springframework.webflow.execution.Action) and args(context) "/> <aop:around pointcut-ref="transactionMonitorActionPointcut“ method=" invoke "/> </aop:aspect> </aop:config> <bean id="transactionMonitorActionAdvice" class= " c.o.webframework.aop.aspectj.TransactionMonitorActionAdvice "/>

Joining Monitors - Across Threads TransactionMonitor monitor = new TransactionMonitor("Profit"); ErmaCallable callable = new ErmaCallable( new CollectUnderpantsCallable() ); try { Future future = executorService.submit(callable); future.get(FUNDING_DURATION, TimeUnit.MILLISECONDS); monitor.addChildMonitor(callable.getMonitor()); monitor.succeeded(); } catch (TimeoutException e) { monitor.failedDueTo(e); throw e; } finally { monitor.done(); }

Joining Monitors - Distributed Services public DispatcherResponse doFilter(DispatcherRequest request, Object resource, FilterChain chain) throws Throwable { TransactionMonitor monitor = new TransactionMonitor(getName(request)); MonitoringEngine mEngine = MonitoringEngine.getInstance(); Map inheritableAttributes = mEngine.getInheritableAttributes() ; Map serializableAttributes = mEngine.makeSerializable(inheritableAttributes); HashMap ermaAttributes = new HashMap(serializableAttributes); request.addParameter( ERMA_FILTER_PARAM_KEY , OLCSerializer.serialize(ermaAttributes) ); DispatcherResponse response = chain.doFilter(request, resource); … return response; }

Distributed Services (cont.) public DispatcherResponse doFilter(DispatcherRequest request, Object resource, FilterChain chain) throws Throwable { … DispatcherResponse response = chain.doFilter(request, resource); Object responseBlob = response.getParameter( ERMA_FILTER_PARAM_KEY ); Monitor responseMonitor = (Monitor) OLCSerializer.deserialize ((byte[]) responseBlob); monitor.addChildMonitor(responseMonitor); Throwable throwable = response.getThrowable(); if (throwable == null) { monitor.succeeded(); } else { monitor.failedDueTo(throwable); } return response; }

The MonitorProcessor Interface public interface MonitorProcessor { public void startup(); public void shutdown(); public void monitorCreated(Monitor monitor); public void monitorStarted(Monitor monitor); public void process(Monitor monitor); }

A MonitorProcessorAdaptor Example public class ResultCodeAnnotatingMonitorProcessor extends MonitorProcessorAdapter { public void process(Monitor monitor) { if (monitor.hasAttribute("failureThrowable")) { Throwable t = (Throwable) monitor.get("failureThrowable"); while (t.getCause() != null) { t = t.getCause(); } monitor.set("resultCode", t.getClass().getName()); } else { monitor.set("resultCode", "success"); } } }

LoggingMonitorProcessor Output process: c.o.monitoring.api.monitor.TransactionMonitor -> blockedCount = 9 -> blockedTime = 17 -> cpuTimeMillis = 1470.0 -> createdAt = Sun Mar 09 16:13:17 CDT 2008 -> endTime = Sun Mar 09 16:13:30 CDT 2008 -> failed = false -> hostname = oberon -> latency = 13180 -> name = httpIn_/shop/airsearch/search/air/pageView_airResults -> remoteIpAddress = 10.222.186.147 -> sessionId = D417C9CC585F82E1 -> threadId = 2a20ec -> validationFailure = false -> vmid = wl -> waitedCount = 10 -> waitedTime = 10414

EventPatternLoggingMonitorProcessor Output wl|AirSearchExecuteAction.search| NoSearchResultsAvailableException wl|jiniOut_ShopService_createResultSet| NoSearchResultsAvailableException tbs-shop|jiniIn_ShopService_createResultSet| NoSearchResultsAvailableException tbs-shop|c.o.t.h.s.ShopServiceImpl.createResultSet.AIR| NoSearchResultsAvailableException tbs-shop|c.o.t.s.SpiShopService.createResultSet.AIR| NoSearchResultsAvailableException tbs-shop|jiniOut_LowFareSearchService_execute| SearchSolutionNotFoundException air-search|jiniIn_LowFareSearchService_execute| SearchSolutionNotFoundException air-search|LowFareSearchRequest| SearchSolutionNotFoundException Follow the trail of Exceptions… don’t bother the on-call engineers for the higher layers… save time by narrowing your log search query!

Event Processing

EventFlow

Expression Language !failed latency > 1000 && failed totalFailed/total sum(int(failed)) stddev(latency) indexof(name, ‘jini’) == 0 strftime(“%Y%m%d, windowTimestamp)

Event Processing Demo StreamBase Studio

Custom Functions public class HelloWorld { public static String sayHello(String name) { return "Hello, “ + name; } }

Custom Operators public void processTuple(int inputPortId, Tuple tuple) throws StreamBaseException { String name = tuple.getString("name"); Tuple output = outputSchema.createTuple(); output.setField("message", "Hello, "+name); sendOutput(OUTPUT_PORT, output); }

Visualization

SNMP

Graphite

Graphite RESTful URLs target=tbs-shop.all.jiniIn.ShopService.createResultSet#all.count

Graphite RESTful URLs - Wildcards target=tbs-shop.all.jiniIn.ShopService. * .count

Graphite RESTful URLs - Functions target= derivative( tbs-shop.all.jiniIn.ShopService.*.count )

Graphite RESTful URLs – Pie Charts target=cpuTime:790&target=waitedTime:4043&target=blockedTime:0&target=other:31

Graphite RESTful URLs – Raw CSV Data target=tbs-shop.all.jiniIn.ShopService.*.count& rawData=true 0 0 0 0 0 0 0 0 0 0 0 0 0 None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None 84 92 90 84 76 81 78 84 89 83 85 81 89 None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None None 81 91 94 82 79 81 79 83 87 84 82 86 85 None None None None None None None None None None None None None 0 0 0 0 0 1 0 0 0 0 0 0 0 None None None None None None None None None None None None None 0 0 0 0 0 0 0 0 0 0 0 0 0 81 91 94 82 79 82 79 83 87 84 82 86 85

Graphite CLI - Dashboards

Monitoring Portal

RSS

Command Line Stream Discovery [mokeefe@egcep03 ~]$ sbc list input-streams input-stream DeleteFromThresholds input-stream GarbageCollectorStats input-stream ListActiveAlarmsInput input-stream MemoryPoolStats input-stream MemoryStats input-stream MonitorInput input-stream MonitoringEngineManager_lifecycle input-stream ReloadLog4j input-stream ThreadStats input-stream ThresholdInput …

Command Line Schema Discovery [mokeefe@egcep03 ~]$ sbc describe MonitorInput <stream name=“…" schema=“…" uuid="..."> <schema name="schema:MonitorInput" uuid=“…"> <field name="name" size="256" type="string"/> <field name="vmid" size="128" type="string"/> <field name="hostname" size="128" type="string"/> <field name="threadId" size="32" type="string"/> <field name="createdAt" type="timestamp"/> <field name="endTime" type="timestamp"/> <field name="latency" type="int"/> <field name="cpuTimeMillis" type="int"/> <field name="failureThrowable" size="16384" type="string"/> <field name="failed" type="bool"/> <field name="resultCode" size="128" type="string"/> <field name="sessionId" size="32" type="string"/> <field name="locale" size="24" type="string"/> <field name="remoteIpAddress" size="15" type="string"/> <field name="posCode" size="4" type="string"/> </schema> </stream>

Command Line Queries [mokeefe@egcep03 ~]$ sbc dequeue MonitorInput --where " sessionId == '570EA1FABEE015B9' and vmid='tbs-shop-13.31' " jiniOut_AirportLookupService_findLocationByIATACode,tbs-shop-13.31,egbsoneg01.prod.o.com,67484c,2008-03-10 15:40:42.489-0500,2008-03-10 15:40:42.489-0500,2008-03-10 15:40:42.502-0500,13,null,null,null,false,null,null,GBP,570EA1FABEE015B9,English (United Kingdom),10.235.1.119,null,EBUK jiniIn_LocationFinderService_findAirports,tbs-shop-13.31,egbsoneg01.prod.o.com,67484c,2008-03-10 15:40:42.480-0500,2008-03-10 15:40:42.480-0500,2008-03-10 15:40:42.503-0500,23,null,null,null,false,null,success,GBP,570EA1FABEE015B9,English (United Kingdom),10.235.1.119,null,EBUK …

Event Pattern Monitoring

Event Pattern Monitoring (cont)

Event Pattern Monitoring Work In Progress Orbitz Worldwide Real-time Clickstream Analysis w/Correlation

Recap Event Processors Monitored Apps Operations Center Graphite JMX, ssh ERMA SNMP Portal

Future Directions

Orbitz Worldwide will open source ERMA

Orbitz Worldwide will open source Graphite

Orbitz Worldwide is considering open sourcing of

orbitz-lib-streambase

Esper integration

For More Information

The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems, David Luckham, Addison Wesley Professional, May 2002, ISBN: 0201727897

http:// www.complexevents.com

ERMA and Graphite: Watch http://forum.complexevents.com/ for an announcement

http:// corp.orbitz.com /careers/

StreamBase: http:// www.streambase.com /

Esper: http://esper.codehaus.org/

Matt O’Keefe, Senior Architect Doug Barth, Technical Lead TS-6048

Appendix: Filters public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws … { TransactionMonitor monitor = new TransactionMonitor(getClass(), "doFilter"); try { filterChain.doFilter(servletRequest, servletResponse); monitor.succeeded(); } catch (IOException e) { monitor.failedDueTo(e); throw e; } catch (ServletException e) { monitor.failedDueTo(e); throw e; } catch (RuntimeException e) { monitor.failedDueTo(e); throw e; } finally { monitor.done(); } }

Appendix: Interceptors public class ERMAInterceptor extends HandlerInterceptorAdapter { public boolean preHandle(…) throws Exception { TransactionMonitor monitor = new TransactionMonitor("httpIn"); monitor.setInheritable(POSCODE, getPosCode()); monitor.setInheritable(LOCALE, getLocale()); monitor.setInheritable(CURRENCY, getCurrency()); monitor.setInheritable(CHANNEL, getChannel()); monitor.setInheritable(SESSION_ID, getSessionId()); monitor.setInheritable(IP_ADDR, getClientAddress()); }

Appendix: Interceptors (cont.) public void postHandle(…) throws Exception { View view = modelAndView.getView(); if (view == null) { String viewName = modelAndView.getViewName(); EventMonitor monitor = new EventMonitor("pageView_" + viewName); monitor.fire(); } else if (RedirectView.class.isAssignableFrom( view.getClass())) { String redirectUrl = extractDispatcherPath(request); EventMonitor monitor = new EventMonitor("redirect_" + redirectUrl); monitor.fire(); } }

Appendix: Interceptors (cont.) public void afterCompletion(…, Exception exception) throws Exception { TransactionMonitor httpInMonitor = (TransactionMonitor) MonitoringEngine.getInstance(). getCompositeMonitorNamed("httpIn"); if (exception == null) { httpInMonitor.succeeded(); } else { httpInMonitor.failedDueTo(exception); } httpInMonitor.set(Monitor.NAME, constructNewMonitorName(httpInMonitor)); httpInMonitor.done(); }

Appendix: Listeners public void stateEntering(RequestContext context, StateDefinition nextState) throws EnterStateVetoException { EventMonitor monitor = new EventMonitor("flowExecution.stateEntering"); StateDefinition currentState = context.getCurrentState(); monitor.set("currentStateId", currentState.getId()); monitor.set("nextStateId", nextState.getId()); monitor.fire(); }