Big Data or Fast Data - CONFESS_2013

592 views

Published on

Steve Millidge, Director of C2B2 Consulting Ltd, presenting at CONFESS_2013 Conference in Vienna on the 4th of April 2013

In-memory Data Grids (IMDGs) and NoSQL data stores are very alike conceptually. Many utilise a Key/Value store programming model and architecturally they are engineered with the same principals distributed horizontal scalability; high availability; performance and reliability. So the questions is when to use one over the other? This session will look at many of the common use cases seen in large scale web systems today and explore whether the problem to solve is a Big Data problem or a Fast Data problem. We'll look at capabilities of the two technologies and decide whether the best technology for each use case is NoSQL or an IMDG.

Published in: Technology, Business
2 Comments
1 Like
Statistics
Notes
No Downloads
Views
Total views
592
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
0
Comments
2
Likes
1
Embeds 0
No embeds

No notes for slide

Big Data or Fast Data - CONFESS_2013

  1. 1. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukBig Data or Fast Data?Steve MillidgeDirectorC2B2@c2b2consultingwww.c2b2.co.uk
  2. 2. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukAbout Me• Founder of C2B2– Leading Independent Middleware Experts– Non-functional Experts• Vendor Neutral– Red Hat (JBoss), Oracle (Fusion), VMWare(vFabric), Open Source (Apache)• 20 Years Middleware Expertise• 15 years Field Consultancy
  3. 3. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukAgenda• Principles of RASP• NoSQL and IMDG Characteristics• Use Cases–Big Data or Fast Data?
  4. 4. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.uk“Reliability, Availability, Scalabilityand Performance are prerequisitesfor functionality!”They are Priority 1 Requirements
  5. 5. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukReliability• No Data Loss or Corruption• Prevents Exploits• Fraud• Reputation Risk• Legal ChallengeCustomers need confidencein their transactions
  6. 6. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukAvailability• System is availablefor customers to use• No availability resultsin no transactions• Transactions = £££• Receive your P45 ifyou can’t sort it!
  7. 7. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukMultipliers in AvailabilitySystem1System2System399% Availability 99% Availability 99% AvailabilityOverall Availability = 0.99*0.99*0.99 = 97%
  8. 8. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukHA TechniquesRedundancy DecouplingSystemSystem99% Availability99% AvailabilityPair = 1 – (0.01*0.01) = 99.99%Overall = 0.9999 x 0.9999 x 0.9999 = 99%System199% AvailabilitySystem2System399% Availability99% AvailabilityOverall = 99%
  9. 9. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukChoose Your Poison• Consistency– The client perceives that a set of operations hasoccurred all at once.• Availability– Every operation must terminate in an intendedresponse.• Partition Tolerance– Operations will complete, even if individualcomponents are unavailable.http://citeseer.ist.psu.edu/544596.html
  10. 10. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukPerformanceHow fast does a single transaction take toexecute!• Faster Performance = Happier Customers• Faster Performance = More Transactions
  11. 11. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukBarriers to Performance• Raw Algorithmic Performance• Resource Limitations– Not enough cpu, disk, memory• Resource Contention– Locks• IO Latency– Network, Disk
  12. 12. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukLatencyTime delay in requesting an operation and itbeing initiated• Key factor in large scale distributedapplications• Typically not taken into account duringdevelopment
  13. 13. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukLatency Factors• Network Distance• Network Reliability• Data Size• Operation Granularity• Resource Contention• JVM GC
  14. 14. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukScalabilityAbility to add more hardware inresponse to more demand.Without a reduction inperformance!
  15. 15. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukScaling OutHorizontal Scaling• Add AdditionalServers• Add Load Balancer• Distribute trafficacross the servers• Much Cheaper thanScale Up• Has HA benefits
  16. 16. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukLinear Scalability(Nirvana)01002003004005006007008009001 2 3 4Linear ScalabilityTypical ScalabilityUsersCluster Nodes
  17. 17. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukIn-Memory Data Grids andNo SQL Stores
  18. 18. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukThe Key Value Store• Basically a big Map!• Provides MapSemantics– put(Key,Object)– Object get(Key)• Schema Less• Decouples ObjectsOrderID -> OrderName -> PersonPostcode -> Address
  19. 19. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukDistributed Key PartitioningApplicationKeyPartitionApplicationKeyPartitionApplicationKeyPartitionApplicationKeyPartitionPUTBBGETBBBCPUTCC
  20. 20. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukConsistent HashingKey SpaceHash RingNode 1HashNode 2HashNode 3HashNode 4HashKeyHash
  21. 21. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukKey Partitioning• Keys AssignedPartition via Hash• Nodes Own aPartition• Number of PartitionsFixed• Number of Nodesisn’tPartitionPartitionPartitionPartitionPartitionPartitionPartition
  22. 22. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukHA Key PartitioningApplicationKeyPartitionApplicationKeyPartitionApplicationKeyPartitionApplicationKeyPartitionPUTBBBBNODECRASH!!!B
  23. 23. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukRASP Benefits• Linear Scalability– 2 hops for Read (Worst Case), 1 for client aware consistent hash– 2 hops for Write (Worst Case), 1 for client aware consistent hash– More Nodes = More Data• High Availability– Configurable Duplicates• Reliability– Eventual Consistency and Quorum• Location Independent Access– Grid knows where data is
  24. 24. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukPerformance Issues?Key PartitioningApplicationKeyPartitionApplicationKeyPartitionApplicationKeyPartitionApplicationKeypartitionProcess
  25. 25. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukIn Place ProcessingApplicationCacheApplicationCacheApplicationCacheApplicationCacheProcessProcessProcessProcess
  26. 26. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukIMDG and NoSQLData Grid• Stored in Memory• Optimised for Network IONoSQL Store• Stored on Disk• Optimised for Disc IOApplicationKeysApplicationKeysApplicationKeysApplicationKeysApplicationKeysApplicationKeysData Data Data
  27. 27. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukWhich to use When?
  28. 28. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.uk3VsReproduced From:DIYA SOUBRAhttp://www.datasciencecentral.com/forum/topics/the-3vs-that-define-big-dataKey Factors in technologySelection• Volume• Velocity
  29. 29. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukVolume(Big Data)• Petabytes and Terrabytes– Suitable for No SQL– Uses cheap disc• 100s Gigabytes and below– Ideal for IMDB
  30. 30. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukVelocity(Fast Data)• Real and Near Real Time– In-memory access speeds– Ideal for IMDB• Batch and Periodic– Process large data volumes– Ideal for NoSQL stores
  31. 31. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukVariety(Fast Data)• Both Support Unstructured Data• Data Size is Determinant• Large Blobs– Video etc.– Ideal for NoSQL
  32. 32. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukSingle Key Retrieval Scenario• Content Management– Get Photo• Personalisation– Get User Info• Order Retrieval– Get Order ID• IMDB typically Faster– Depends on Volume• NoSQL response readthrough is typicalApplicationKeysApplicationKeysApplicationKeysGET
  33. 33. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukKey Processing Small Result• Financial Risk Calculation– Can I book this Order• Bet Placement– Can I place this Bet• Hotel Availability Search– What hotels are available• Web Translation– Gather Translated Text• IMDB the Best ChoiceApplicationKeysApplicationKeysApplicationKeys
  34. 34. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukWrite Lots, Read a Little• Social Media Dumping– You Tube• Log Analysis– Capture Logs• SMS Billing– Capture Events• Cyber Security– Capture Packets• Ideal for NoSQLApplicationKeysApplicationKeysApplicationKeys
  35. 35. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukAnalytical Intelligence• Consumer Trend Analysis– Who orders what when• Route Optimisation– Delivery Trucks• Social Graph Search– Patterns in social mediagraphs• Fraud Detection– Transaction analysis• NoSQL the Best ChoiceApplicationKeysApplicationKeysApplicationKeys
  36. 36. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukReal Time “Tipping”• ProductRecommendations– Related products• Alerting– Truck has Stopped!• Financial Position Calc– Check regs during booking• Nudge Purchasing– 5 people are looking at thishotel• IMDB the Best ChoiceApplicationKeysApplicationKeysApplicationKeys
  37. 37. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.ukSummaryFast Data (IMDB)• 10s Mb – 100s GB• Few Mb Element Size• Response Times– 10ms – few seconds• Typically in-line with userrequest• Also capable of Push viaEventsBig Data (NoSQL)• 100sGb – Petabyte• Size limited by disc• Response times– Few s – long batch• Typically analytical• Used inline for smallscale direct key retrieval
  38. 38. © C2B2 Consulting Limited 2013www.c2b2.co.ukAll Rights ReservedInterested in Hard Core Middleware?Join us! careers@c2b2.co.uk

×