Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Проектирование и разработка Big Data
решений на базе Microsoft Azure
Андр...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data :: Mobile Phones Usage
.NET CONFERENCE #1 IN UKRAINE KYIV 20...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data :: Connected Car
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Text
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data :: Smart City
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Text
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data :: Around the World
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
...
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Big Data Solutions :: Getting Started
.NET CONFERENCE #1 IN UKRAINE
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data Pipeline
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data Reference Architecture for IoT
.NET CONFERENCE #1 IN UKRAINE...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Lambda Architecture
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Batch Processing
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Stream Processing
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Big Data Solutions :: Best Practices
.NET CONFERENCE #1 IN UKRAINE
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data Solutions :: Best Practices
.NET CONFERENCE #1 IN UKRAINE KY...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Best Practices :: Think Big. Start Small
.NET CONFERENCE #1 IN UKRAIN...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Best Practices :: Data First
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Best Practices :: Don’t Interrupt Fast Path
.NET CONFERENCE #1 IN UKR...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Best Practices :: Security as a Culture
.NET CONFERENCE #1 IN UKRAINE...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Best Practices :: Understand Business Model
.NET CONFERENCE #1 IN UKR...
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Big Data Solutions :: Data Collection
.NET CONFERENCE #1 IN UKRAINE
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Collection :: How To
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Tra...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Collection :: Data Factory
.NET CONFERENCE #1 IN UKRAINE KYIV 20...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Collection :: Data Hubs
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Hubs :: Comparison
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Capab...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Hubs :: Main Problems
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• ...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Connection :: Kafka on HDInsight
.NET CONFERENCE #1 IN UKRAINE K...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Kafka on HDInsight :: Pros & Cons
.NET CONFERENCE #1 IN UKRAINE KYIV ...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Connection :: What to Choose?
.NET CONFERENCE #1 IN UKRAINE KYIV...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Connection :: Custom Solution
.NET CONFERENCE #1 IN UKRAINE KYIV...
.NET LEVEL UP
Custom Solution :: Connector
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
.NET LEVEL UP
Custom Solution :: Connector & Queue
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Device
Registry
Queue
Connector
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Big Data Solutions :: Processing
.NET CONFERENCE #1 IN UKRAINE
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Processing :: Azure Stream Analytics
.NET CONFERENCE #1 IN UKRAINE KY...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Processing :: Stream Analytics :: Repartitioning
.NET CONFERENCE #1 I...
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Processing :: HDInsight
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
HDInsight :: Pros & Cons
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• No...
.NET LEVEL UP
Processing :: HDInsight :: Storm
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Low latency. True streaming. High...
.NET LEVEL UP
Processing :: HDInsight :: Spark
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• High throughput. Complete fault t...
.NET LEVEL UP
HDInsight :: Spark & Storm Benchmarking
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Demo JSON files { 100 user...
.NET LEVEL UP
HDInsight :: Spark Benchmarking
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
70 sec
0
Latency Percentiles (130 00...
.NET LEVEL UP
HDInsight :: Spark Benchmarking
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
16 sec
0
Latency Percentiles (100 00...
.NET LEVEL UP
HDInsight :: Spark Benchmarking
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
5 sec
0
Latency Percentiles (100 000...
.NET LEVEL UP
HDInsight :: Storm Benchmarking
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
3 sec
0
Latency Percentiles (up to 1...
.NET LEVEL UP
Processing :: What to Choose?
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
NOT Avro, JSON, or CSV
Scale beyond 1 ...
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
One more thing…
.NET CONFERENCE #1 IN UKRAINE
.NET LEVEL UP
Big Data Solutions :: Cross-Cutting Concerns
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
.NET CONFERENCE #1 IN UKRAINE
Андрей Антиликаторов
Solution Architect
Upcoming SlideShare
Loading in …5
×

.NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure

60 views

Published on

В данном докладе автор расскажет об опыте построения высоконагруженных Big Data решений на базе Microsoft Azure. Особое внимание будет уделено не только архитектурным подходам и best practices, но и многим ньюансам, тонким моментам и подводным камням, с которыми придется столкнуться. В каждом конкретном случае будут даны практические советы, cheat sheets и варианты решений, которые помогут с выбором подхода и позволят избежать потенциальных проблем. Также автор отдельно расскажет об опыте решения еще одной распростаненной задачи - проектировании Rules Engines для аналитики данных в реальном времени.

Published in: Education
  • Be the first to comment

  • Be the first to like this

.NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure

  1. 1. Тема доклада Тема доклада Тема доклада KYIV 2019 Проектирование и разработка Big Data решений на базе Microsoft Azure Андрей Антиликаторов .NET CONFERENCE #1 IN UKRAINE
  2. 2. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Mobile Phones Usage .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  3. 3. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Connected Car .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  4. 4. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Smart City .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  5. 5. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Around the World .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  6. 6. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Getting Started .NET CONFERENCE #1 IN UKRAINE
  7. 7. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data Pipeline .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  8. 8. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data Reference Architecture for IoT .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  9. 9. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Lambda Architecture .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  10. 10. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Batch Processing .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  11. 11. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Stream Processing .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  12. 12. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Best Practices .NET CONFERENCE #1 IN UKRAINE
  13. 13. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data Solutions :: Best Practices .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  14. 14. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Think Big. Start Small .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Scalable architecture. • Prototyping is a must. • Azure Stream Analytics is NOT a silver bullet. Data Source On-Boarding
  15. 15. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Data First .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Data formats are true headache. • Hard to extract valuable data. • Diagnostics telemetry must be processed.
  16. 16. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Don’t Interrupt Fast Path .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Don’t process between high-scale components unless you know what you’re doing. • Pass only valuable data which really needs to be processed in real-time.
  17. 17. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Security as a Culture .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Think about security, identity and management from the very beginning. • Security is a shared responsibility between Azure and the customer.
  18. 18. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Understand Business Model .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Data architecture more important than solution architecture • Plan and budget appropriately through simple predictable scenarios • Remember that you need to support millions of assets (for IoT)
  19. 19. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Data Collection .NET CONFERENCE #1 IN UKRAINE
  20. 20. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Collection :: How To .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Transactional data (data sync) One-Time data migration IoT data
  21. 21. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Collection :: Data Factory .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  22. 22. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Collection :: Data Hubs .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Event Hub IoT Hub
  23. 23. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Hubs :: Comparison .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Capability Standard Basic Event Hub Cloud-to-device Yes Yes Yes HTTPS, AMQP Yes Yes Yes MQTT Yes Yes File upload from devices Yes Yes File upload from devices Yes Yes Device Provisioning Service Yes Yes Cloud-to-device messaging Yes Device twin and management Yes IoT Edge Yes
  24. 24. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Hubs :: Main Problems .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Strict quotas & limits • Limited amount of partitions (32) • Need to plan additional 20% - 30% capacity • Limited number of concurrent connections • Works great if resources located in single region • Cannot provide latency < 0.1 sec
  25. 25. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Connection :: Kafka on HDInsight .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  26. 26. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Kafka on HDInsight :: Pros & Cons .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Open-source • Managed (SLA 99.9%) • Zookeeper out of the box • Not cost effective • Requires a lot of tuning
  27. 27. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Connection :: What to Choose? .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Capability Cloud-to-device communications Yes No No Device-initiated file upload Yes No No Device state information Device Twins No No Protocol support HTTPS, MQTT. AMQP AMQP, HTTPS Kafka Protocol Security Per-device identity Shared access policies Through external authentication services
  28. 28. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Connection :: Custom Solution .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Queue Device registry Data Connector
  29. 29. .NET LEVEL UP Custom Solution :: Connector .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  30. 30. .NET LEVEL UP Custom Solution :: Connector & Queue .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Device Registry Queue Connector
  31. 31. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Processing .NET CONFERENCE #1 IN UKRAINE
  32. 32. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Processing :: Azure Stream Analytics .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Easy to use • Easy to scale • Good integration with Azure Services • Relatively cheap • HIPAA, HITRUST, CSA STAR, ISO, GDRP • Limited SAQL • Vendor lock • Possible data loss and crash on invalid data
  33. 33. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Processing :: Stream Analytics :: Repartitioning .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  34. 34. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Processing :: HDInsight .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  35. 35. Тема доклада Тема доклада Тема доклада .NET LEVEL UP HDInsight :: Pros & Cons .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Not cost effective • Software is outdated • Software requires a lot of tuning • Provides managed versions of 1st class frameworks • Seamless integration • Frameworks are open-source
  36. 36. .NET LEVEL UP Processing :: HDInsight :: Storm .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Low latency. True streaming. High throughput • Excellent for non-complicated use cases • No state management • “At least once” guarantee
  37. 37. .NET LEVEL UP Processing :: HDInsight :: Spark .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • High throughput. Complete fault tolerance • Exactly once guarantees • No true streaming (micro batches). High latency. • Too complex to tune.
  38. 38. .NET LEVEL UP HDInsight :: Spark & Storm Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Demo JSON files { 100 users; 10 events per user } • 1 Redis node. 5 Kafka nodes. 10 worker nodes.
  39. 39. .NET LEVEL UP HDInsight :: Spark Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 70 sec 0 Latency Percentiles (130 000 events/s; 2-second micro-batches) 50 sec 60 sec 40 sec 10050 90 30 sec 25 10 sec 20 sec
  40. 40. .NET LEVEL UP HDInsight :: Spark Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 16 sec 0 Latency Percentiles (100 000 events/s; 10 second micro-batches) 8 sec 12 sec 4 sec 10050 90
  41. 41. .NET LEVEL UP HDInsight :: Spark Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 5 sec 0 Latency Percentiles (100 000 events/s; 3-second micro-batches) 3 sec 4 sec 2 sec 10050 90 1 sec 25
  42. 42. .NET LEVEL UP HDInsight :: Storm Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 3 sec 0 Latency Percentiles (up to 150 000 events/s) 2 sec 2.5 sec 1.5 sec 10050 90 1 sec 25
  43. 43. .NET LEVEL UP Processing :: What to Choose? .NET CONFERENCE #1 IN UKRAINE KYIV 2019 NOT Avro, JSON, or CSV Scale beyond 1 Gb/s Low latency is a must Zero data loss
  44. 44. Тема доклада Тема доклада Тема доклада KYIV 2019 One more thing… .NET CONFERENCE #1 IN UKRAINE
  45. 45. .NET LEVEL UP Big Data Solutions :: Cross-Cutting Concerns .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  46. 46. Тема доклада Тема доклада Тема доклада KYIV 2019 .NET CONFERENCE #1 IN UKRAINE Андрей Антиликаторов Solution Architect

×