Real-time Event
Processing using
LINQ for Logs &
Traces (TX)
Gert Drapers (#DataDude)
Principle Software Design Engineer
Agenda
•Rx (Reactive Extensions)
•Tx (LINQ to Logs and Traces)
•Case Study
Subtitel
TX (LINQ to Logs and Traces)
•http://tx.codeplex.com/
•TX allows you use LINQ
directly on raw event
sources:
• Ad-hoc quer...
Builds on Reactive Extensions (RX.Net)
http://rx.codeplex.com/
Reactive Essential Interfaces
namespace System {
// Producers (.NET events, WinRT Events, Sensors, APM method etc.)
public...
Event processing at all
scales with Reactive
ExtensionsBart de Smet
Thrusday 17 april 13:15 - 14:30
Demo
Using TX with LINQPad
Event Sources
• Tx.Windows
• Event Tracing for Windows (ETW)
• Performance Counters - interop with the "Performance Data
H...
Type Generation
Playback
•Structured Mode
• Lose analogy is that events of given type are like a "Table“
• playback.GetObservable< T> is s...
Playback Features
•Multiplexed sequences
•Hiding the heterogeneity of the event sources
•Same API for real-time and past h...
Demo
HttpSpew
Case Study XblWatch
•Objective of XblWatch:
• Replace existing rule based SCOM pack monitoring of web
service
• Provide ne...
XblWatch/er
xblwatcher.prod.live
XblWatcher.exe XblWatchCmd.exe
http://*:80
tcp://*:9000
Perf Counters
Site/Service/API
Ma...
Demo
XblWatch
Resources
•Tx (LINQ to Logs and Traces)
•(Rx) Reactive Extensions
•Event processing at all scales with Reactive
Extensions...
Laat ons weten wat u vindt van deze sessie! Vul de evaluatie
in via www.techdaysapp.nl en maak kans op een van de 20
prijz...
Real-time Event Processing using LINQ for Logs & Traces
Upcoming SlideShare
Loading in …5
×

Real-time Event Processing using LINQ for Logs & Traces

923 views
684 views

Published on

Dealing with real time events is hard, but required to gain visibility in the clouds; perform low-latency monitoring (seconds) or troubleshoot large number of machines at the same time. LINQ for Traces (http://tx.codeplex.com/) provides a powerful way to deal with standing queries over real-time data feeds, such as Event Tracing for Windows (ETW) sessions or performance counters, as well as ad-hoc queries over past history in the form of trace and log files. This session will take you through the discovery and exploration of TX by building a real-time event analysis solution for web services. The presentation is based on building the actual monitoring event pipeline for Xbox Live Services.

Published in: Software, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
923
On SlideShare
0
From Embeds
0
Number of Embeds
33
Actions
Shares
0
Downloads
27
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Real-time Event Processing using LINQ for Logs & Traces

  1. 1. Real-time Event Processing using LINQ for Logs & Traces (TX) Gert Drapers (#DataDude) Principle Software Design Engineer
  2. 2. Agenda •Rx (Reactive Extensions) •Tx (LINQ to Logs and Traces) •Case Study Subtitel
  3. 3. TX (LINQ to Logs and Traces) •http://tx.codeplex.com/ •TX allows you use LINQ directly on raw event sources: • Ad-hoc query on past history in trace and log files • Standing queries on real-time feeds, such as Event Tracing for Windows (ETW) sessions
  4. 4. Builds on Reactive Extensions (RX.Net) http://rx.codeplex.com/
  5. 5. Reactive Essential Interfaces namespace System { // Producers (.NET events, WinRT Events, Sensors, APM method etc.) public interface IObservable<out T> { IDisposable Subscribe(IObserver<T> observer); } // Events stream consumers public interface IObserver<in T> { void OnNext(T value); void OnError(Exception error); void OnCompleted(); } }
  6. 6. Event processing at all scales with Reactive ExtensionsBart de Smet Thrusday 17 april 13:15 - 14:30
  7. 7. Demo Using TX with LINQPad
  8. 8. Event Sources • Tx.Windows • Event Tracing for Windows (ETW) • Performance Counters - interop with the "Performance Data Helper" (PDH) native API • Event Logs (.evtx) • IIS W3C text log files • Tx.SqlServer • SQL Server Extended Events (XEvent) is the tracing technology used by the SQL-Server engine. • Extensibility
  9. 9. Type Generation
  10. 10. Playback •Structured Mode • Lose analogy is that events of given type are like a "Table“ • playback.GetObservable< T> is similar to dataContext.GetTable<T> in LINQ to SQL • More precisely, IObservable is a sequence (i.e. order matters) • This is the same distinction as IEnumerable vs.. SQL Table •Timeline Mode • This mode is useful to ask questions that are about the timeline of all events, regardless of type
  11. 11. Playback Features •Multiplexed sequences •Hiding the heterogeneity of the event sources •Same API for real-time and past history •Single-Read to answer multiple queries from file(s) •Occurrence Time Scheduler
  12. 12. Demo HttpSpew
  13. 13. Case Study XblWatch •Objective of XblWatch: • Replace existing rule based SCOM pack monitoring of web service • Provide near-realtime monitoring • Create Synthetic Performance Counters for success and failure rates per API and service • Publish Synthetic Performance Counters to integrate with existing monitoring and alerting system
  14. 14. XblWatch/er xblwatcher.prod.live XblWatcher.exe XblWatchCmd.exe http://*:80 tcp://*:9000 Perf Counters Site/Service/API Machine 1 XblWatch.exe http.sys ETW trace Perf Counters Machine/Service/API Machine 2 XblWatch.exe http.sys ETW trace Perf Counters Machine/Service/API Machine 3 XblWatch.exe http.sys ETW trace Perf Counter Machine/Service/API Machine N XblWatch.exe http.sys ETW trace Perf Counters Machine/Service/API
  15. 15. Demo XblWatch
  16. 16. Resources •Tx (LINQ to Logs and Traces) •(Rx) Reactive Extensions •Event processing at all scales with Reactive Extensions Bart de Smet Thrusday 17 april 13:15 - 14:30
  17. 17. Laat ons weten wat u vindt van deze sessie! Vul de evaluatie in via www.techdaysapp.nl en maak kans op een van de 20 prijzen*. Prijswinnaars worden bekend gemaakt via Twitter (#TechDaysNL). Gebruik hiervoor de code op uw badge. Let us know how you feel about this session! Give your feedback via www.techdaysapp.nl and possibly win one of the 20 prices*. Winners will be announced via Twitter (#TechDaysNL). Use your personal code on your badge. * Over de uitslag kan niet worden gecorrespondeerd, prijzen zijn voorbeelden – All results are final, prices are examples

×