This document discusses democratizing data access and processing through LINQ, Rx, and CoSQL. It introduces LINQ for querying objects and LINQ to SQL for querying tables relationally. It discusses the object-relational impedance mismatch and how Rx makes events first-class. CoSQL is proposed to bring SQL-style querying to NoSQL databases by treating them relationally while keeping their flexibility. Duality principles from category theory are discussed as enabling asynchronous and reactive programming models.

1. Category theory, Monads, and Duality in the world of (BIG) DataBart J.F. De Smetbartde@microsoft.comCloud Programmability Team

2. What’s in a name?Cloud Programmability TeamLogical role:Research-oriented teamCollaboration with MSRPhysical placement:Oasis within the product teamClose to the SQL Server business(Dual) 80/20 rule of successPortfolioLanguage Integrated Query (LINQ)XML literals in Visual Basic 9Reactive Extensions (Rx)Various undisclosed projectsDemocratizing the cloud

3. Take OneDemocratizing data access with LINQ

4. A quick reminder on LINQSolving the impedance mismatch between objects and data through querying.

5. Back to the future5+ years agoCensored

6. Democratizing data accessvarres = from p inctx.Productswherep.UnitPrice > 100group p byp.Categoryinto gselectnew { Category = g.Key, Avg = g.Average() };“Lost in translation”varres = ctx.Products .Where(p => p.UnitPrice > 100) .GroupBy(p => p.Category) .Select(g => new { Category = g.Key, Avg = g.Average() });(In-memory) iteratorsQuery providers

7. C# 3.0 compilation to C# 2.0

8. Language Integrated MonadsIEnumerable<T>IQueryable<T>IEnumerable<R> SelectMany<T, R>(this IEnumerable<T> source,Func<T, IEnumerable<R>> selector)SelectMany

9. Maybe baby!BillionNull-propagating dotstring s = name?.ToUpper();Syntactic sugarname.SelectMany( _ => _.ToUpper(), s => s)from _ in namefrom s in _.ToUpper()select sCompilerCan useextension method

10. Closing the loopLINQ to Haskell 

11. Take twoDemocratizing event processing with Rx

12. Once upon a time…Tier splitting“LINQ to Events”

13. Reactive Extensions (Rx)GPSRSS feedsStock tickersSocialmediaUI eventsServer management

14. Pull-based data accessinterfaceIEnumerable<out T>{IEnumerator<T> GetEnumerator();}interfaceIEnumerator<out T> : IDisposable{boolMoveNext(); T Current { get; }void Reset();}You could get stuck

15. Duality in the world around us(Or… the Dutch are cheap)Electricity: inductor and capacitorLogic: De Morgan’s LawProgramming?¬𝐴∨𝐵≡¬𝐴∧¬𝐵 ¬𝐴∧𝐵≡¬𝐴∨¬𝐵

16. Duality as the secret sauce?Give me a recipehttp://en.wikipedia.org/wiki/Dual_(category_theory)Reversing arrows…Input becomes output and vice versaMaking a U-turnin synchrony

17. Distilling the essenceProperties and unchecked exceptionsinterfaceIEnumerable<out T>{IEnumerator<T> GetEnumerator();}interfaceIEnumerator<out T> : IDisposable{boolMoveNext(); T Current { get; }}

18. Distilling the essenceProperties and unchecked exceptionsinterfaceIEnumerable<out T>{IEnumerator<T> GetEnumerator();}interfaceIEnumerator<out T> : IDisposable{boolMoveNext() throwsException; T GetCurrent();}

19. Distilling the essenceEmbracing a (more) functional styleinterfaceIEnumerable<out T>{IEnumerator<T> GetEnumerator();}interfaceIEnumerator<out T> : IDisposable{boolMoveNext() throwsException; T GetCurrent();}

20. Distilling the essenceEmbracing a (more) functional styleinterfaceIEnumerable<out T>{IEnumerator<T> GetEnumerator();}interfaceIEnumerator<out T> : IDisposable{ (void | T | Exception) MoveNext();}() -> (() -> (void | T | Exception))

21. Flipping the arrowsPurely mechanical transformation() -> (() -> (void | T | Exception))((void | T | Exception) -> ()) -> ()

22. Harvesting the resultSo far for abstract nonsenseinterfaceIBar<out T>{voidQux(IFoo<T> foo);}interfaceIFoo<in T>{voidWibble(); void Wobble(T value); voidWubble(Exception error);}

23. Harvesting the resultThe observer pattern in disguiseinterfaceIObservable<out T>{void Subscribe(IObserver<T> observer);}interfaceIObserver<in T>{voidOnCompleted(); voidOnNext(T value); voidOnError(Exception error);}

24. The observer pattern revisitedStateful!

25. Interface hierarchyinterfaceIObservable<out T>{IDisposableSubscribe(IObserver<T> observer);}

26. Message grammarOnNext(42)OnNext(43)OnCompletedsource1OnNext(“Hello”)OnError(error)source2OnNext* [OnError | OnCompleted]

27. Observable.Create<T> operatorIObservable<int> o = Observable.Create<int>(observer => { // Assume we introduce concurrency (see later)…observer.OnNext(42);observer.OnCompleted();return () => { /* unsubscribe action */ };});IDisposable subscription = o.Subscribe(onNext: x => { Console.WriteLine("Next: " + x); },onError: ex => { Console.WriteLine("Oops: " + ex); },onCompleted: () => { Console.WriteLine("Done"); });C# doesn’t have anonymous interface implementation, so we provide various extension methods that take lambdas.C# 4.0 named parameter syntax

28. Observable.Create<T> operatorIObservable<int> o = Observable.Create<int>(observer => {// Assume we introduce concurrency (see later)…observer.OnNext(42);observer.OnCompleted();return () => { /* unsubscribe action */ };});IDisposable subscription = o.Subscribe(onNext: x => { Console.WriteLine("Next: " + x); },onError: ex => { Console.WriteLine("Oops: " + ex); },onCompleted: () => { Console.WriteLine("Done"); });Thread.Sleep(30000); // Main thread is blocked…F10

29. Observable.Create<T> operatorIObservable<int> o = Observable.Create<int>(observer => {// Assume we introduce concurrency (see later)…observer.OnNext(42);observer.OnCompleted();return () => { /* unsubscribe action */ };});IDisposable subscription = o.Subscribe(onNext: x => { Console.WriteLine("Next: " + x); },onError: ex => { Console.WriteLine("Oops: " + ex); },onCompleted: () => { Console.WriteLine("Done"); });Thread.Sleep(30000); // Main thread is blocked…F10

30. Observable.Create<T> operatorIObservable<int> o = Observable.Create<int>(observer => { // Assume we introduce concurrency (see later)…observer.OnNext(42);observer.OnCompleted();return () => { /* unsubscribe action */ };});IDisposable subscription = o.Subscribe(onNext: x => { Console.WriteLine("Next: " + x); },onError: ex => { Console.WriteLine("Oops: " + ex); },onCompleted: () => { Console.WriteLine("Done"); });Thread.Sleep(30000); // Main thread is blocked…F5

31. Observable.Create<T> operatorIObservable<int> o = Observable.Create<int>(observer => { // Assume we introduce concurrency (see later)…observer.OnNext(42);observer.OnCompleted();return () => { /* unsubscribe action */ };});IDisposable subscription = o.Subscribe(onNext: x => { Console.WriteLine("Next: " + x); },onError: ex => { Console.WriteLine("Oops: " + ex); },onCompleted: () => { Console.WriteLine("Done"); });Thread.Sleep(30000); // Main thread is blocked…Breakpoint got hit

32. Iterators dualizedIObservable<int> GetXs() {returnObservable.Create(o =>for(int i = 0; i < 10; i++)o.OnNext(i * i);o.OnCompleted(); );}GetXs().Subscribe(x => {Console.WriteLine(x);});IEnumerable<int> GetXs() { for (int i = 0; i < 10; i++)yieldreturni * i;yield break;}foreach(var x inGetXs()) {Console.WriteLine(x);}SynchronousAsynchronous

33. Compositionality mattersIObservable<T>Merge<T>(thisIObservable<T> left,IObservable<T> right){return Create<T>(observer => {// Ignoring a few details for OnCompletedvar d1 = left.Subscribe(observer);var d2 = right.Subscribe(observer);returnnewCompositeDisposable(d1, d2); });}Lazy evaluation

34. Bridging Rx with the WorldWhy .NET events aren’t first-class…Hidden data sourceHow to pass around?form1.MouseMove+= (sender, args) => {if(args.Location.X==args.Location.Y)// I’d like to raise another event};form1.MouseMove -=/* what goes here? */Lack of compositionResource maintenance?

36. Bridging Rx with the World…but observable sequences are first-classSource of Point valuesObjects can be passedIObservable<Point>mouseMoves= Observable.FromEvent(frm, "MouseMove");varfiltered = mouseMoves.Where(pos => pos.X == pos.Y);varsubscription = filtered.Subscribe(…);subscription.Dispose();Can define operatorsResource maintenance!

37. Composition and QueryingIt’s the continuation monad!// IObservable<string> from TextChanged eventsvarchanged = Observable.FromEvent(txt, "TextChanged");var input = (from e in changedlet text = ((TextBox)e.Sender).Textwheretext.Length >= 3select text) .DistinctUntilChanged() .Throttle(TimeSpan.FromSeconds(1));// Bridge with the dictionary web servicevar svc = newDictServiceSoapClient();var lookup = Observable.FromAsyncPattern<string, DictionaryWord[]> (svc.BeginLookup, svc.EndLookup);// Compose both sources using SelectManyvar res = from term in inputfrom words in lookup(term)select words;input.SelectMany(term => lookup(term))

38. Introducing schedulersHow to be asynchronous?Different ways to Introduce of concurrencyParameterization by schedulersinterfaceIScheduler{DateTimeOffset Now { get; }IDisposableSchedule<T>( T state,Func<IScheduler, T, IDisposable> f);// Overloads for time-based scheduling}

39. Example: creation operatorsstatic classObservable{static IObservable<T> Return<T>(T value,IScheduler scheduler) {return Create<T>(observer => {var state = new { value, observer };returnscheduler.Schedule(state, (self, s) => {s.observer.OnNext(s.value);s.observer.OnCompleted(); }); }); }}ResourcemgmtAvoiding closures(serialization)

40. Operational layering

41. IQbservable<T>LINQ to TwitterHow?ToQbservableTranslatable(Expression trees)IQueryable<T>LINQ to SQLToQueryableLINQ to *.*AsObservableHomo-iconicAsEnumerableAsQbservableAsQueryableToObservableIEnumerable<T>LINQ to ObjectsIObservable<T>LINQ to EventsFixed(MSIL)ToEnumerablePull(interactive)Push(reactive)What?DualityConcurrency(IScheduler)Where?Message loopsDistributedWorker poolsThreads

42. Take THREEDemocratizing cloud data processing with CoSQL

43. NoSQL is CoSQL!

44. The NoSQL trend

45. Object graphsvar_1579124585 = newProduct{ Title = “The Right Stuff”, Author = “Tom Wolfe”, Year = 1979, Pages = 304, Keywords = new[] { “Book”, “Hardcover”, “American” }, Ratings = new[] { “****”, “4 stars” },};var Products = new[] { _1579124585 };

46. Queries over object graphsvarq = from product in Productswhereproduct.Ratings.Any(rating => rating == “****”)selectnew { product.Title, product.Keywords };

47. The O/R paradoxObjectsFully compositional value ::= scalar new {…, name = value, … }TablesNon compositional value ::= new {…, name = scalar, … }

48. Relational (de)normalization

49. Queries over tablesvar q = from product in Productsfromrating in Ratingswhereproduct.ID == rating.ProductId && rating == “****”fromkeyword in Keywordswhereproduct.ID == keyword.ProductIDselectnew { product.Title, keyword.Keyword };varq = from product inProductsjoinrating in Ratingsonproduct.ID equalsrating.ProductIdwhererating == “****”selectproduct intoFourStarProductsfromfourstarproductinFourStarProductsjoinkeyword in Keywordsonproduct.ID equalskeyword.ProductIDselectnew { product.Title, keyword.Keyword };

50. Welcome to O/R voodoovarq = from product in Productswhereproduct.Ratings.Any(rating => rating == “****”)selectnew { product.Title, product.Keywords };

51. What did we gain?Ad-hoc queries?But what about scale…The relational Gods invented indexesGoing against the PK-FK flow…from p1 in WWWfrom p2 in WWWwhere p2.Contains(p1.URL)selectnew { p1, p2 };

52. Job security?

53. Spot the difference

54. Duality to the rescue again?

55. Consequences of duality

56. More work in the area

57. Thank you!Bart J.F. De Smetbartde@microsoft.comCloud Programmability Team