Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Azure Messaging Crossroads

39 views

Published on

You’ve decided to develop in Azure and need to make a decision on the messaging technology. Storage Queues, Service Bus, Event Grid, Event Hubs, etc. Which technology should you use? How do you pick the right one if they all deal with messages? This session will help you answer these questions.

Published in: Technology
  • Be the first to comment

Azure Messaging Crossroads

  1. 1. Azure Messaging Crossroads Sean Feldman Azure MVP, Particular Software
  2. 2. Event HubsService Bus Event GridStorage Queues Azure Messaging Services
  3. 3. Send a message Service BusStorage Queues
  4. 4. Is Storage Queue service dead or not?
  5. 5. Send an event Service Bus Event Hubs Event Grid
  6. 6. With EventGrid, is ServiceBus dead?
  7. 7. Azure SignalR
  8. 8. Somany choices, so many options…
  9. 9. How does this fit into the bigger picture?
  10. 10. Storage Queues (a.k.a. Simple Queues)
  11. 11. Storage Queues are Task Queues • Coordinate work across compute • VMs, App Service, Functions, etc. • Messages contain data and senders have expectations • Low cost • Easy to use • Unlimited number of queues • Unlimited number of concurrent clients • Maximum queue size 500TB • Maximum message size 64KB (48KB with Base64 encoding) • Limited features
  12. 12. What did get sacrificed? • No headers/metadata • No pub/sub • No scheduling • No dead-lettering • No transactional guarantees • Cannot peek more than 32 messages • 7 days retention period • Infinite (almost) lately, but careful • and other compute-related features
  13. 13. Simple Queue in Action • Sender sends message to queue • Queue ACKs receipt • Receiver connects to queue & retrieves message • Receiver ACKs complete (or other action)
  14. 14. Remember TASK QUEUES FOR WORK ITEMS AS MESSAGES SMALL SIZE MESSAGES HTTP-BASED LOW COST
  15. 15. Service Bus
  16. 16. Where does Service Bus fit in? Enterprise messaging Front End / UX Back End / Processing Queue/Topics/Subscriptions
  17. 17. Advanced Features • Message headers • Duplication detections • Auto-forwarding • Dead-lettering • Scheduled delivery • Message expiration • Message Deferral • Auto-delete on idle • Batching • Transactions • Message Sessions • Pub/sub • Filtering and Actions • Claim check (.NET)
  18. 18. Receiving Messages Traditional approach – pump all the time Serverless approach – pump when needed
  19. 19. .NET Client Helper • Concurrent Message Pump • RegisterMessageHandler() • Full control over • Concurrency • Lock renewal • Message completion
  20. 20. Premium Features • Scale up (MUs) • Geo-DR • Availability Zones • Virtual Network Service Endpoints
  21. 21. Migrating from Standard to Premium • No downtime • Same connection string for Senders/Receivers as before • Some messages need to be drained • Up to 1000 entities
  22. 22. QUEUES & TOPICS COMPUTE BASED FEATURES TRANSACTIONAL RELIABLE COMPLEX WORKFLOWS Remember
  23. 23. Want more? Azure Service Bus - Deep Dive session Tomorrow at 11AM (Prince Albert room)
  24. 24. Event Hubs
  25. 25. How Event Hubs is different from queues • Records a stream • Recoding moves forward only • You can play a stream over and over again • Like a cassette tape it has channels • When recording, all channels record • But the data on each channel is different • Event Hubs calls these channels “partitions”
  26. 26. Event Hubs conceptual architecture HTTP AMQP Kafka
  27. 27. How are Event Hubs and Kafka similar? • Both are ingesting technologies, not a queue • Both use partitioned consumer • Partitions are independent • Both use a client side cursor concept • Both can scale to very high workloads • Conceptually they are nearly the same • Every Kafka user is close to being an Event Hubs user
  28. 28. Event Hubs for Apache Kafka™ Ecosystems • Kafka 1.0 compatible endpoint can be backed by Event Hubs • Use existing Kafka application & tools with Event Hubs • Only need to change the connection string
  29. 29. Event Hubs Scaling • Throughput Units (TUs) • Variable reserved capacity (paid for) –Dedicated Tier • 1 MB/second or 1,000 events/second ingress • 2 MB/second or 2,000 events/second egress • Overages are throttled (Server Busy Exception) • A feature to help with it
  30. 30. Auto Inflation • With insufficient TUs senders will be throttled • Will go up (up to a cap), but won’t scale down
  31. 31. Be Aware: Up to 7 days of retention
  32. 32. Event Hubs Capture lets you go fast and slow • Automatically send Event Hub data into the Storage account or Azure Data Lake • Minimize your overhead: no code or other services to configure • Focus on data processing, not capturing
  33. 33. How Capture Works {Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second} Example: https://mystorageaccount.blob.core.windows.net/mycontainer/mynamespace/myeventhub/0/2017/12/08/03/03/17.avro
  34. 34. .NET Helpers • Safe Batching Support • EventDataBatch.TryAdd(EventData) • Consuming events • EventHubClient & EventProcessorHost
  35. 35. Remember THROUGHPUT OVER FEATURES NOT A QUEUE REPLACEMENT MANAGE CURSOR
  36. 36. Event Grid
  37. 37. The Traditional Events Receiving Something happens Continuously running process
  38. 38. There’s a Theme… •Pub/Sub Model •Reactive •Event Driven
  39. 39. Service Bus
  40. 40. Event Grid Concepts 1. Event: what happened 2. Event source: where it took place 3. Topic: where publishers send events 4. Event Subscription: what events to handle 5. Event Handler: the app or service reacting to the event
  41. 41. Event Schema 64KB 1 MB
  42. 42. Event Filtering • Event type / subject / topic • Event data (data.key) • Values (numbers, strings, Boolean, array) • NumberLessThan • NumberLessThanOrEquals • NumberGreaterThan • NumberGreaterThanOrEquals • NumberIn – the value for data.key equals a value in the set [0, 2.08, 3.14] • NumberNotIn – the value for data.key is not in the set [1, 11, 112, 1124] • StringContains – the value for data.key contains “the” • StringIn – the value for data.key equals a value in the set [“small”, “brown”, “fox”] • StringNotIn – the value for data.key equals a value in the set [“jumped”, “over”, “the”] • StringBeginsWith – the value for data.key begins with “lazy” • StringEndsWith – the value for data.key ends with “dog” • BoolEquals
  43. 43. Event Domains
  44. 44. Topics Subscriptions Filters Haven’t we seen this before?
  45. 45. Why is Event Grid different? • Cloud Native by design • Serverless friendly • Engineered for reliability and scale • Supports CNCF Cloud Events v0.1
  46. 46. Cloud native by design • Always available (99.99% SLA) • Near real-time event delivery (<1s e2e) • At-least-once delivery • Dynamic scale • 10,000,000 events per second per region • 100,000,000 subscriptions per region • Platform agnostic (WebHook) • Language agnostic (HTTP protocol)
  47. 47. Engineered for scale and reliability • Defaults: 30 delivery attempts / 24 hours • Retry intervals • 10 seconds • 30 seconds • 1 minute • 5 minutes • 10 minutes • 30 minutes • 1 hour • Once an hour up to 24 hours
  48. 48. Engineered for scale and reliability Dead-lettering • Requires Storage account + container • Dead-lettered events stored as blobs
  49. 49. Azure Functions Logic Apps Azure Automation WebHooks Event Hubs Storage Queues IoT Hub Blob Storage Resource Groups Azure Subscriptions Event Hubs Custom Topics Storage (GPv2) Container Registry Service Bus (Prem)
  50. 50. CloudEvent Schema v0.1
  51. 51. Ubiquitous • Today there are 10+ Azure publishers https://docs.microsoft.com/en-us/azure/event-grid/event-sources • At some point in time most Azure services will be publishers
  52. 52. Serverless friendly • Ease of setup from the Portal • Azure CLI • Powershell
  53. 53. How Event Grid composes with Queues and Streams Messaging services can be publishers or subscribers to Event Grid sub1 sub2 mytopic Event Hubs Storage queues / Service Bus
  54. 54. Security and Authentication • Validation Handshake (WebHook event delivery) • Event of type Microsoft.EventGrid.SubscriptionValidationEvent • With validation data • Prove • Echo back {validationCode: “value”} • Send GET to validationURL
  55. 55. EVENT DRIVEN PUB/SUB NOT ENTERPRISE MESSAGING SCALE THROUGHPUT RETRIES DEAD-LETTERING CNCF CLOUD EVENT SUPPORT CROSS-CLOUD Remember
  56. 56. How to choose the right Azure messaging service
  57. 57. Serverless
  58. 58. The right tools for the job
  59. 59. ActiveMessagesAvailableWithNoListeners
  60. 60. Q&A @sfeldman | weblogs.asp.net/sfeldman

×