Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CenterEdge: LINQing to data: easing the transition from SQL – Couchbase Connect 2016

286 views

Published on

For a C# or VB.NET development shop, moving from the cozy world of SQL to the modern, scalable world of NoSQL can be an unnerving proposition. But it shouldn't be. With the aid of LINQ, .NET developers can seamlessly transition their existing skills and knowledge, applying it to Couchbase Server the same day. This session will lay the groundwork for this transition, covering JSON data modeling, querying, indexing, and more, all from the perspective of a .NET/SQL developer. Learn just how easy NoSQL can be with the help of Couchbase, N1QL, and LINQ.

Published in: Software

CenterEdge: LINQing to data: easing the transition from SQL – Couchbase Connect 2016

  1. 1. @ Couchbase Connect16 LINQing to data: Easing the transition from SQL Welcome
  2. 2. Brant Burnett Software Development Team Lead Couchbase Community Expert Welcome
  3. 3.  Quick Facts  .Net Data Modeling Basics  Querying with LINQ  Advanced LINQ Features for Couchbase  Efficient Indexing for LINQ Queries  Change Tracking + Read Your Own Write  Q & A The Agenda
  4. 4. Celebrating 12 Year Anniversary Team of 50 in Roxboro, NC Sister company is Palace Pointe, a 100k sq. ft. Entertainment Venue for which we were developed as an in-house system Over 600 users across the US and abroad FEC’s, Waterparks, Trampoline Parks, Amusement Parks, Skating Rinks, Bowling Centers, Zoos & Museums Quick Facts
  5. 5.  Point of Sale  Admissions & Ticketing  Party, Group & Event Bookings  Online Sales & Party Reservations  Time Clock & Labor Management  & More! Quick Facts
  6. 6. .Net since inception in 2004 Customers use a local client/server SQL app, but these integrate with our cloud-based products Started using Couchbase Server 1.8 for online stores in 2012 Cache SQL Server query results, persisted shopping carts Needed scalability and stability New products operate using a pure Couchbase Server 4.5 database layer Quick Facts
  7. 7. New Cloud Platform Data Flow Data Data Data IndexQuery Web ServersUsers Remote Application Servers
  8. 8. How to build your POCOs for consistency Nesting documents Nesting arrays Including reference primary keys .Net Data Modeling Basics
  9. 9. .Net Data Modeling Basics public class Airline { public string Callsign { get; set; } public string Country { get; set; } [JsonProperty("iata")] public string IATA { get; set; } [JsonProperty("iaco")] public string IACO { get; set; } public int Id { get; set; } public string Name { get; set; } public string Type { get; set; } } Key: airline_10 { "callsign": "MILE-AIR", "country": "United States", "iata": "Q5", "icao": "MLA", "id": 10, "name": "40-Mile Air", "type": "airline" }
  10. 10. // JSON decorators not shown to simplify display public class Airport { public string AirportName { get; set; } public string City { get; set; } public string Country { get; set; } public string FAA { get; set; } public Coordinate Geo { get; set; } public string ICAO { get; set; } public int Id { get; set; } public string Timezone { get; set; } public string Type { get; set; } } public class Coordinate { public int Altitude { get; set; } public double Latitude { get; set; } public double Longitude { get; set; } } Key: airport_1254 { "airportname": "Calais Dunkerque", "city": "Calais", "country": "France", "faa": "CQF", "geo": { "alt": 12, "lat": 50.962097, "lon": 1.954764 }, "icao": "LFAC", "id": 1254, "type": "airport", "tz": "Europe/Paris" } .Net Data Modeling Basics Nesting Subdocuments as Properties Note: Subdocuments don’t need “type” attributes, just the root document
  11. 11. // JSON decorators not shown to simplify display public class Route { public string Airline { get; set; } public string AirlineId { get; set; } public string DestinationAirport { get; set; } public double Distance { get; set; } public string Equipment { get; set; } public int Id { get; set; } public List<Schedule> Schedule { get; set; } public string SourceAirport { get; set; } public int Stops { get; set; } public string Type { get; set; } } public class Schedule { public int Day { get; set; } public string Flight { get; set; } public TimeSpan UTC { get; set; } } Key: route_10000 { "airline": "AF", "airlineid": "airline_137", "destinationairport": "MRS", "distance": 2881.617376098415, "equipment": "320", "id": 10000, "schedule": [ { "day": 0, "flight": "AF198", "utc": "10:13:00" }, { "day": 0, "flight": "AF547", "utc": "19:14:00" } ], "sourceairport": "TLV", "stops": 0, "type": "route" } .Net Data Modeling Basics Nesting Arrays as Lists Note: List<T> or similar list construct
  12. 12. [DocumentTypeFilter(TypeString)] public class Airline { public const string TypeString = "airline"; public string Callsign { get; set; } public string Country { get; set; } [JsonProperty("iata")] public string IATA { get; set; } [JsonProperty("iaco")] public string IACO { get; set; } public int Id { get; set; } public string Name { get; set; } // Type is now read only to maintain consistency public string Type => TypeString; } Key: airline_10 { "callsign": "MILE-AIR", "country": "United States", "iata": "Q5", "icao": "MLA", "id": 10, "name": "40-Mile Air", "type": "airline" } .Net Data Modeling Basics Good Practices And LINQ Improvements Note: DocumentTypeFilter automatically applies a WHERE type = ‘x’ predicate Making Type read only ensures it’s always saved correctly
  13. 13. Key: route_10000 { "airline": "AF", "airlineid": "airline_137", "destinationairport": "MRS", "distance": 2881.617376098415, "equipment": "320", "id": 10000, "schedule": [ { "day": 0, "flight": "AF198", "utc": "10:13:00" }, { "day": 0, "flight": "AF547", "utc": "19:14:00" } ], "sourceairport": "TLV", "stops": 0, "type": "route" } Key: airline_137 { "callsign": "AIRFRANS", "country": "France", "iata": "AF", "icao": "AFR", "id": 137, "name": "Air France", "type": "airline" } .Net Data Modeling Basics Note: To support joining, include the document key (or have a way to build it)
  14. 14. Querying with LINQ Basic and Advanced Queries Joining Documents by Primary Key Unnesting Arrays
  15. 15. Querying with LINQ public ActionResult BasicQuery() { var db = new BucketContext(ClusterHelper.GetBucket("travel-sample")); var query = from p in db.Query<Airline>() orderby p.Name select p; return View(query); } SELECT `Extent1`.* FROM `travel-sample` as `Extent1` WHERE (`Extent1`.`type` = 'airline') ORDER BY `Extent1`.`name` ASC Note: DocumentTypeFilter (slide 12) automatically added the type predicate
  16. 16. Querying with LINQ Easily add filters, sorts, projections and pagination to the query public ActionResult AdvancedQuery() { var db = new BucketContext(ClusterHelper.GetBucket("travel-sample")); var query = (from p in db.Query<Airline>() where p.Callsign.StartsWith("A") orderby p.Name select new AirlineModel {Callsign = p.Callsign, Name = p.Name}).Take(10); return View(query); } SELECT `Extent1`.`callsign` as `callsign`, `Extent1`.`name` as `name` FROM `travel-sample` as `Extent1` WHERE (`Extent1`.`type` = 'airline') AND (`Extent1`.`callsign` LIKE 'A%') ORDER BY `Extent1`.`name` ASC LIMIT 10
  17. 17. Querying with LINQ Join to other documents using their primary key public ActionResult Join() { var db = new BucketContext(ClusterHelper.GetBucket("travel-sample")); var query = from route in db.Query<Route>() join airline in db.Query<Airline>() on route.AirlineId equals N1QlFunctions.Key(airline) where route.SourceAirport == "ATL" && route.DestinationAirport == "ABE" orderby airline.Name, route.Stops select new RouteModel { AirlineName = airline.Name, Stops = route.Stops, Schedule = route.Schedule }; return View(query); } SELECT `Extent2`.`name` as `airlineName`, `Extent1`.`stops` as `stops`, `Extent1`.`schedule` as `schedule` FROM `travel-sample` as `Extent1` INNER JOIN `travel-sample` as `Extent2` ON KEYS `Extent1`.`airlineid` WHERE (`Extent1`.`type` = 'route') AND (`Extent2`.`type` = 'airline') AND ((`Extent1`.`sourceairport` = 'ATL') AND (`Extent1`.`destinationairport` = 'ABE')) ORDER BY `Extent2`.`name` ASC, `Extent1`.`stops` ASC
  18. 18. Querying with LINQ UNNEST in N1QL { "airline": "AF", "airlineid": "airline_137", "destinationairport": "MRS", "distance": 2881.617376098415, "equipment": "320", "id": 10000, "schedule": [ { "day": 0, "flight": "AF198", "utc": "10:13:00" }, { "day": 0, "flight": "AF547", "utc": "19:14:00" } ], "sourceairport": "TLV", "stops": 0, "type": "route" } [ { "day": 0, "destinationairport": "MRS", "flight": "AF198", "sourceairport": "TLV", "utc": "10:13:00" }, { "day": 0, "destinationairport": "MRS", "flight": "AF547", "sourceairport": "TLV", "utc": "19:14:00" } ] SELECT route.sourceairport, route.destinationairport, schedule.day, schedule.flight, schedule.utc FROM `travel-sample` AS route UNNEST route.schedule AS schedule WHERE route.type = 'route'
  19. 19. Querying with LINQ Using UNNEST with a second FROM clause public ActionResult Unnest() { var db = new BucketContext(ClusterHelper.GetBucket("travel-sample")); var query = from route in db.Query<Route>() from schedule in route.Schedule where route.SourceAirport == "ATL" && route.DestinationAirport == "ABE" orderby schedule.Day, schedule.UTC select new UnnestedScheduleModel { Airline = route.Airline, Day = schedule.Day, UTC = schedule.UTC }; return View(query); } SELECT `Extent1`.`airline` as `airline`, `Extent2`.`day` as `day`, `Extent2`.`utc` as `utc` FROM `travel-sample` as `Extent1` INNER UNNEST `Extent1`.`schedule` as `Extent2` WHERE (`Extent1`.`type` = 'route') AND ((`Extent1`.`sourceairport` = 'ATL') AND (`Extent1`.`destinationairport` = 'ABE')) ORDER BY `Extent2`.`day` ASC, `Extent2`.`utc` ASC
  20. 20. Advanced LINQ Features for Couchbase NULL != MISSING UseKeys – Get documents by their primary key UseIndex – Provide index hints to the query engine Asynchronous querying
  21. 21. Advanced LINQ Features for Couchbase Handling undefined JSON attributes – IsMissing, IsNotMissing, IsValued, IsNotValued public ActionResult IsMissing() { var db = new BucketContext(ClusterHelper.GetBucket("travel-sample")); var query = from p in db.Query<Airline>() where N1QlFunctions.IsMissing(p.IACO) select p; return View(query); } SELECT `Extent1`.* FROM `travel-sample` as `Extent1` WHERE (`Extent1`.`type` = 'airline') AND `Extent1`.`iaco` IS MISSING
  22. 22. Advanced LINQ Features for Couchbase Select documents directly using their primary key – UseKeys public ActionResult UseKeys() { var db = new BucketContext(ClusterHelper.GetBucket("travel-sample")); var query = from p in db.Query<Airline>() .UseKeys(new[] {"airline_137", "airline_10765", "airline_1316" }) select p; return View(query); } SELECT `Extent1`.* FROM `travel-sample` as `Extent1` USE KEYS ['airline_137', 'airline_10765', 'airline_1316'] WHERE (`Extent1`.`type` = 'airline')
  23. 23. Advanced LINQ Features for Couchbase Provide Query Plan hints – UseIndex public ActionResult UseIndex() { var db = new BucketContext(ClusterHelper.GetBucket("travel-sample")); var query = from p in db.Query<Route>().UseIndex("def_sourceairport") where p.SourceAirport == "ATL" orderby p.DestinationAirport select p; return View(query); } SELECT `Extent1`.* FROM `travel-sample` as `Extent1` USE INDEX (`def_sourceairport` USING GSI) WHERE (`Extent1`.`type` = 'route') AND (`Extent1`.`sourceairport` = 'ATL') ORDER BY `Extent1`.`destinationairport` ASC
  24. 24. Advanced LINQ Features for Couchbase Run Queries Asynchronously – ExecuteAsync public async Task<ActionResult> Async() { var db = new BucketContext(ClusterHelper.GetBucket("travel-sample")); var query = from p in db.Query<Airline>() orderby p.Name select p; return View( await query.ExecuteAsync()); }
  25. 25. Advanced LINQ Features for Couchbase Run Aggregate Queries Asynchronously – ExecuteAsync public async Task<ActionResult> AsyncAggregate() { var db = new BucketContext(ClusterHelper.GetBucket("travel-sample")); var query = from p in db.Query<Route>() select p; return View( await query.ExecuteAsync( p => p.Average(q => q.Distance))); } ExecuteAsync() can even run any immediate execution method (aggregates, First, Single, Explain) by passing the method as a lambda expression
  26. 26. Efficient Indexing For LINQ Queries Differences between SQL and N1QL indexes Indexing on “type” attribute for speed Indexing DateTime attributes
  27. 27. Efficient Indexing For LINQ Queries Airline SQL Table Airport SQL Table travel-sample Bucket Airline Indexes Airport Indexes Bucket Indexes Note: Remember that GSI indexes are similar to SQL indexes, but not the same
  28. 28. Efficient Indexing For LINQ Queries /* Use predicate to only index documents of a certain type */ CREATE INDEX `airport_sourceairport` ON `travel-sample` (`sourceairport`) WHERE `type` = 'airport' /* Index the same attribute across multiple document types by including type first */ CREATE INDEX `def_type_id` ON `travel-sample` (`type`, `id`) /* A good practice is to create a fallback in case other indexes aren't used */ CREATE INDEX `def_type` ON `travel-sample` (`type`) Note: Be sure to index on the “type” attribute if you’re using [DocumentTypeFilter(“…”)]
  29. 29. Efficient Indexing For LINQ Queries Indexing DateTime attributes – STR_TO_MILLIS var cutoffDateTime = new DateTime(2010, 1, 1, 0, 0, 0, DateTimeKind.Utc); var query = db.Query<Beer>() .Where(e => e.Updated <= cutoffDateTime); /* LINQ automatically wraps all DateTime constants and properties in STR_TO_MILLIS */ /* STR_TO_MILLIS converts an ISO8601 string to a Unix numeric representation */ /* It also handles the time zone specifier */ SELECT `Extent1`.* FROM `beer-sample` as `Extent1` WHERE (`type` = 'beer') AND (STR_TO_MILLIS(`Extent1`.`updated`) <= STR_TO_MILLIS("2010-01-01T00:00:00Z")) /* So STR_TO_MILLIS must also be used in the index, or the index cannot be used */ CREATE INDEX `beer_updated` ON `beer-sample` (STR_TO_MILLIS(`updated`)) WHERE `type` = 'beer'
  30. 30. Change Tracking + Read Your Own Write Document POCO considerations Updating, inserting, and deleting documents Using MutationState to read your own writes Note: Linq2Couchbase change tracking is currently in developer preview
  31. 31. Change Tracking + Read Your Own Write [DocumentTypeFilter(TypeString)] public class Route { public const string TypeString = "route"; // Read only property to return the calculated primary key // This is used when saving the document [Key] public string Key => TypeString + "_" + Id; // Properties must be virtual for changes to be tracked public virtual string Airline { get; set; } // some properties not shown for clarity... // Lists now use IList<T> instead of List<T> public virtual IList<Schedule> Schedule { get; set; } // some properties not shown for clarity... public string Type => TypeString; } Update Your Document Models To Support Proxies
  32. 32. Change Tracking + Read Your Own Write Unit of Work – BeginChangeTracking, SubmitChanges _db.BeginChangeTracking(); // Query must return the document class, without a select projection var query = from p in _db.Query<Airline>() where p.Id == id select p; // Query must execute after call to BeginChangeTracking var airline = query.FirstOrDefault(); if (airline == null) { return HttpNotFound(); } airline.Name = model.Name; airline.Callsign = model.Callsign; // Save the changes, if any _db.SubmitChanges();
  33. 33. Change Tracking + Read Your Own Write Document INSERT/DELETE – Save, Remove _db.BeginChangeTracking(); // To delete a document when SubmitChanges is called _db.Remove(airline); // To insert a document when SubmitChanges is called _db.Save(new Airline() { Id = 1, Name = model.Name, Callsign = model.Callsign }); // Save the changes _db.SubmitChanges(); Note: Save and Remove execute immediately if called before BeginChangeTracking
  34. 34. Change Tracking + Read Your Own Write Do I have the latest changes? – MutationState, ConsistentWith _db.SubmitChanges(); // MutationState represents the changes made by SubmitChanges // Pass into the query to ensure the changes are indexed before the query executes // This will slow the query, but much less than using RequestPlus consistency // Couchbase Server 4.5 is required for this feature var query = from p in _db.Query<Airline>().ConsistentWith(_db.MutationState) orderby p.Name select p;
  35. 35. Why use Couchbase, N1QL and LINQ?  More scalable and performant than traditional SQL in the cloud  Easy transition for .Net developers trained on SQL and LINQ, and still produces unit testable code  More logical, nested data models rather than dozens of subsidiary tables  Schema-less JSON increases flexibility as your system evolves, leaving schema enforcement in your data access layer Conclusion
  36. 36. What's next? Source and Documentation: https://github.com/couchbaselabs/Linq2Couchbase Example project: https://github.com/brantburnett/Couchbase.Linq.Example Questions: https://forums.couchbase.com/c/net-sdk - @btburnett3 Email - bburnett@centeredgesoftware.com Twitter - @btburnett3 Conclusion
  37. 37. Q & A
  38. 38. Thank You Visit us at centeredgesoftware.com

×