Your systems. Working as one. 
Reactive Stream Processingusing DDSand Rx 
www.rti.com 
Sumant Tambe, Ph.D. 
Senior Softwar...
Outline 
•Reactive Systems 
•Stream Processing 
•Overview of Reactive Extensions 
•Overview of DDS 
•Streaming Shapes Demo...
Systems Everyone Wants to Build 
•Event-Driven:Modular, pipelined, asynchronous 
•Elasic:Scales easily up/down with load a...
Stream Processing 
•Stream Processing is the term used to describe an architectural style that operate on a continuous seq...
Shape of an application 
o/p 
Where 
Once 
CombineLatest 
Select 
Scan 
Merge 
Raw Data 
i/p
*nix command line (pipes and filter) 
$ ls-1 | grep“search” | grep“research” 
research 
$ 
10/10/2014 Real-Time Innovation...
Reactive Extensions 
•The Reactive Extensions (Rx) is a library for composing asynchronous and event-based programs using ...
10/10/2014 Real-Time Innovations, Inc. 8 
Data Distribution Service (DDS) 
A Reactive Middleware
DDS: Standards-based Integration Infrastructure forCritical Applications 
© 2009 Real-Time Innovations, Inc. 
StreamingDat...
Systems that interact with the Real World 
•Must adapt to changing environment 
•Cannot stop processing the information 
•...
Real-Time Data Distribution 
10/10/2014 © 2012 RTI • COMPANY 11
RPC over DDS 
2014 
DDS 
Security 
2012 
Family of Specifications 
© 2009 Real-Time Innovations, Inc. COMPANY 
12 
DDS 
Im...
Evolution of DataBus 
Data-centricity basics
Everyday Example: Schedule Meeting via Emails 
Alternative Process #1 (message-centric): 
1.Email: “Meeting Monday at 10:0...
Everyday Example: Schedule Meeting Using a Calendar 
Alternative Process #2: 
1.Calendar: (add meeting Monday at 10:00) 
2...
10/10/2014 © 2012 RTI • COMPANY 16 
DDS Communication Model
DDS for Distribution, Rx for Processing 
10/10/2014 Real-Time Innovations, Inc. 17 
DR 
DR 
DR 
Observable 
Observer 
DW 
...
Rx4DDS.NET = DDS + Rx 
10/10/2014 © 2012 RTI • COMPANY 18 
•DDS wrapper for Rx.NET 
•In C# 
•Anything that produces data i...
DDS and Rx: A Great Match 
10/10/2014 © 2012 RTI • COMPANY 19 
DDS Concept 
Rx Concept/Type/Operator 
Topic of type T 
An ...
DDS and Rx: A Great Match 
10/10/2014 © 2012 RTI • COMPANY 20 
DDS Concept 
Rx Concept/Type/Operation 
QueryConditions 
Io...
Living Demo in C# 
10/10/2014 Real-Time Innovations, Inc. 21
Background Code 
10/12/2014 Real-Time Innovations, Inc. 22 
publicclassShapeType: ICopyable<ShapeType> { publicstringcolor...
Flower Demo 
10/12/2014 Real-Time Innovations, Inc. 23 
IDisposableflower(DDS.DomainParticipantparticipant) { inta = 30, b...
Simple Square to Triangle Transformation 
10/12/2014 Real-Time Innovations, Inc. 24 
IDisposableforward_verbose(DDS.Domain...
Swap Square’s x and y and Propagate Stream Dispose Event 
10/12/2014 Real-Time Innovations, Inc. 25 
IDisposableswap(DDS.D...
Aggregator (Square + Circle = Triangle!) 
10/12/2014 Real-Time Innovations, Inc. 26 
IDisposableaggregator(DDS.DomainParti...
Square/Circle Correlatorusing LINQ 
10/12/2014 Real-Time Innovations, Inc. 27 
IDisposableselectmany_correlator(DDS.Domain...
Thank You! 
•Rx4DDS.NET 
–https://github.com/rticommunity/rticonnextdds-reactive 
•Data Distribution Service 
–http://port...
Upcoming SlideShare
Loading in …5
×

Reactive Stream Processing Using DDS and Rx

1,505 views

Published on

In this presentation you will see why Reactive Extensions (Rx) is a powerful technology for asynchronous stream processing. RTI Data Distribution Service (DDS) will be used as the source of data and as a communication channel for asynchronous data streams. On top of DDS, we'll use Rx to subscribe, observe, project, filter, aggregate, merge, zip, and correlate one or more data streams (Observables). The live demo will be very visual as bouncing shapes of different colors will be transformed in front of you using C# lambdas, Rx.NET, and Visual Studio. You will also learn about the new Rx4DDS.NET library that integrates RTI DDS with Rx.NET. Rx and DDS are a great match because both are reactive. Rx is based on the subject-observer pattern, which is quite analogous to the publish-subscribe pattern of DDS. When used together they support distributed dataflows seamlessly. If time permits, we will touch upon advanced Rx concepts such as stream of streams (IGroupedObservable) and how it captures DDS "keyed topics". The DDS applications using Rx4DDS.NET dramatically simplify concurrency to the extent that it can be simply configured.

Published in: Engineering
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total views
1,505
On SlideShare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
11
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

Reactive Stream Processing Using DDS and Rx

  1. 1. Your systems. Working as one. Reactive Stream Processingusing DDSand Rx www.rti.com Sumant Tambe, Ph.D. Senior Software Research Engineer and Microsoft MVP Real-Time Innovations, Inc. @sutambe Oct. 11, 2014
  2. 2. Outline •Reactive Systems •Stream Processing •Overview of Reactive Extensions •Overview of DDS •Streaming Shapes Demo in C# 10/10/2014 Real-Time Innovations, Inc. 2
  3. 3. Systems Everyone Wants to Build •Event-Driven:Modular, pipelined, asynchronous •Elasic:Scales easily up/down with load and cpucores. •Resilient:fault-tolerant •Responsive:Reacts to events at the speed of environment 10/10/2014 Real-Time Innovations, Inc. 3 Networking Middleware App
  4. 4. Stream Processing •Stream Processing is the term used to describe an architectural style that operate on a continuous sequence of data. 10/10/2014 Real-Time Innovations, Inc. 4
  5. 5. Shape of an application o/p Where Once CombineLatest Select Scan Merge Raw Data i/p
  6. 6. *nix command line (pipes and filter) $ ls-1 | grep“search” | grep“research” research $ 10/10/2014 Real-Time Innovations, Inc. 6 $ cat -| grep“Real” R Real Real [Ctrl+C] $
  7. 7. Reactive Extensions •The Reactive Extensions (Rx) is a library for composing asynchronous and event-based programs using observablesequences •Rx = Observables + Composition + Schedulers •Streams are first-class •Filter, project, aggregate, compose and perform time-based operations on multiple streams •Uses Functional Programming Style •Rx.NET, RxJava, RxJS, RxCpp, RxRuby, RxPyton, and more… 10/10/2014 Real-Time Innovations, Inc. 7
  8. 8. 10/10/2014 Real-Time Innovations, Inc. 8 Data Distribution Service (DDS) A Reactive Middleware
  9. 9. DDS: Standards-based Integration Infrastructure forCritical Applications © 2009 Real-Time Innovations, Inc. StreamingData Sensors Events Real-Time Applications Enterprise Applications Actuators
  10. 10. Systems that interact with the Real World •Must adapt to changing environment •Cannot stop processing the information •Live within world-imposed timing Beyond traditional interpretation of real-time © 2010 Real-Time Innovations, Inc.
  11. 11. Real-Time Data Distribution 10/10/2014 © 2012 RTI • COMPANY 11
  12. 12. RPC over DDS 2014 DDS Security 2012 Family of Specifications © 2009 Real-Time Innovations, Inc. COMPANY 12 DDS Implementation Network / TCP / UDP / IP App DDS Implementation App DDS Implementation DDS Spec 2004 DDS Interoperablity 2006 UML Profile for DDS 2008 DDS for Lw CCM 2009 DDS X-Types 2010 2010 DDS-STD-C++ DDS-JAVA5 App
  13. 13. Evolution of DataBus Data-centricity basics
  14. 14. Everyday Example: Schedule Meeting via Emails Alternative Process #1 (message-centric): 1.Email: “Meeting Monday at 10:00.” 2.Email: “Here’s dial-in info for meeting…” 3.Email: “Meeting moved to Tuesday” 4.You: “Where do I have to be? When?” 5.You: (sifting through email messages…) 14
  15. 15. Everyday Example: Schedule Meeting Using a Calendar Alternative Process #2: 1.Calendar: (add meeting Monday at 10:00) 2.Calendar: (add dial-in info) 3.Calendar: (move meeting to Tuesday) 4.You: “Where do I have to be? When?” 5.You: (check calendar. Contains consolidated-state) 15 The difference is state! The infrastructure consolidates changes and maintains it
  16. 16. 10/10/2014 © 2012 RTI • COMPANY 16 DDS Communication Model
  17. 17. DDS for Distribution, Rx for Processing 10/10/2014 Real-Time Innovations, Inc. 17 DR DR DR Observable Observer DW DW Processing
  18. 18. Rx4DDS.NET = DDS + Rx 10/10/2014 © 2012 RTI • COMPANY 18 •DDS wrapper for Rx.NET •In C# •Anything that produces data is an Observable –Topics, Discovery, Statuses, statuses, etc.
  19. 19. DDS and Rx: A Great Match 10/10/2014 © 2012 RTI • COMPANY 19 DDS Concept Rx Concept/Type/Operator Topic of type T An objectthat implements IObservable<T>, which internally creates a DataReader<T> Communication status, Discovery event streams IObservable<SampleLostStatus> IObservable<SubscriptionBuiltinTopicData> Topic of type T with key type=Key IObservable<IGroupedObservable<Key, T>> Detect a new instance Notify Observers about a new IGroupedObservable<Key, T>with key==instance. Invoke IObserver<IGroupedObservable<Key, T>>.OnNext() Dispose an instance Notify Observers through IObserver<IGroupedObservable<Key,T>>.OnCompleted() Take an instance update of type T Notify Observers about a new value of T using Iobserver<T>.OnNext() Readwith history=N IObservable<T>.Replay(N)(Produces a new IObservable<T>)
  20. 20. DDS and Rx: A Great Match 10/10/2014 © 2012 RTI • COMPANY 20 DDS Concept Rx Concept/Type/Operation QueryConditions Iobservable<T>.Where(…) OR Iobservable<T>.GroupBy(…) SELECT in CFT expression IObservable<T>.Select(...) FROM in CFT expression DDSObservable.FromTopic(“Topic1”) DDSObservable.FromKeyedTopic(“Topic2”) WHEREin CFT expression IObservable<T>.Where(...) ORDER BY in CFT expression IObservable<T>.OrderBy(...) MultiTopic(INNER JOIN) IObservable<T>.Join(...) .Where(...) .Select(...) Join between DDS and non- DDS data Join, CombineLatest, Zip
  21. 21. Living Demo in C# 10/10/2014 Real-Time Innovations, Inc. 21
  22. 22. Background Code 10/12/2014 Real-Time Innovations, Inc. 22 publicclassShapeType: ICopyable<ShapeType> { publicstringcolor; publicintshapesize; publicintx; publicinty; publicShapeType(); } publicclassShapeTypeExtended: ShapeType{ publicfloatangle; publicShapeFillKindfillKind; publicShapeTypeExtended(); } privateDDS.TypedDataWriter<ShapeTypeExtended> triangle_writer;
  23. 23. Flower Demo 10/12/2014 Real-Time Innovations, Inc. 23 IDisposableflower(DDS.DomainParticipantparticipant) { inta = 30, b = 30, c = 10; returnObservable.Interval(TimeSpan.FromMilliseconds(1), Scheduler.Immediate) .Select((longx) => { intangle = (int)(x % 360); returnnewShapeTypeExtended{ x = (int)(120 + (a + b) * Math.Cos(angle) + b * Math.Cos((a / b -c) * angle)), y = (int)(120 + (a + b) * Math.Sin(angle) + b * Math.Sin((a / b -c) * angle)), color = "GREEN", shapesize= 5}; }) .Subscribe(triangle_writer); }
  24. 24. Simple Square to Triangle Transformation 10/12/2014 Real-Time Innovations, Inc. 24 IDisposableforward_verbose(DDS.DomainParticipantparticipant) { varrx_reader= DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square"); IDisposabledisposable = rx_reader.OnDataAvailable((ShapeTypeExtendedshape) => { DDS.InstanceHandle_thandle = DDS.InstanceHandle_t.HANDLE_NIL; triangle_writer.write(shape, refhandle); }); returndisposable; } IDisposableforward_shortest(DDS.DomainParticipantparticipant) { returnDDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square") .OnDataAvailable(triangle_writer); }
  25. 25. Swap Square’s x and y and Propagate Stream Dispose Event 10/12/2014 Real-Time Innovations, Inc. 25 IDisposableswap(DDS.DomainParticipantparticipant) { returnDDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square") .Select(shape => newShapeTypeExtended{ x = shape.y, y = shape.x, color = shape.color, shapesize= shape.shapesize}) .SubscribeAndDisposeOnCompleted(triangle_writer, newShapeTypeExtended{ color = "BLUE"}); }
  26. 26. Aggregator (Square + Circle = Triangle!) 10/12/2014 Real-Time Innovations, Inc. 26 IDisposableaggregator(DDS.DomainParticipantparticipant) { varrx_square_reader= DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square"); varrx_circle_reader= DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Circle"); returnnewCompositeDisposable( newIDisposable[] { rx_square_reader.Subscribe(triangle_writer), rx_circle_reader.Subscribe(triangle_writer) } ); }
  27. 27. Square/Circle Correlatorusing LINQ 10/12/2014 Real-Time Innovations, Inc. 27 IDisposableselectmany_correlator(DDS.DomainParticipantparticipant, booluseLinq) { varrx_circle_reader= DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Circle", Scheduler.Default); varrx_square_reader= DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square", Scheduler.Default); varcorrelator= fromsquare inrx_square_readerfromcircle inrx_circle_reader.Take(1) wheresquare.color== circle.colorselectnewShapeTypeExtended{ x = square.x, y = square.y, color = square.color, shapesize= circle.x}; returncorrelator.Subscribe(triangle_writer); }
  28. 28. Thank You! •Rx4DDS.NET –https://github.com/rticommunity/rticonnextdds-reactive •Data Distribution Service –http://portals.omg.org/dds •Real-Time Innovations, Inc. –www.rti.com 10/12/2014 Real-Time Innovations, Inc. 28

×