Successfully reported this slideshow.
Your SlideShare is downloading. ×

.NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 46 Ad

.NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure

В данном докладе автор расскажет об опыте построения высоконагруженных Big Data решений на базе Microsoft Azure. Особое внимание будет уделено не только архитектурным подходам и best practices, но и многим ньюансам, тонким моментам и подводным камням, с которыми придется столкнуться. В каждом конкретном случае будут даны практические советы, cheat sheets и варианты решений, которые помогут с выбором подхода и позволят избежать потенциальных проблем. Также автор отдельно расскажет об опыте решения еще одной распростаненной задачи - проектировании Rules Engines для аналитики данных в реальном времени.

В данном докладе автор расскажет об опыте построения высоконагруженных Big Data решений на базе Microsoft Azure. Особое внимание будет уделено не только архитектурным подходам и best practices, но и многим ньюансам, тонким моментам и подводным камням, с которыми придется столкнуться. В каждом конкретном случае будут даны практические советы, cheat sheets и варианты решений, которые помогут с выбором подхода и позволят избежать потенциальных проблем. Также автор отдельно расскажет об опыте решения еще одной распростаненной задачи - проектировании Rules Engines для аналитики данных в реальном времени.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to .NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure (20)

Advertisement

More from NETFest (20)

Recently uploaded (20)

Advertisement

.NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure

  1. 1. Тема доклада Тема доклада Тема доклада KYIV 2019 Проектирование и разработка Big Data решений на базе Microsoft Azure Андрей Антиликаторов .NET CONFERENCE #1 IN UKRAINE
  2. 2. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Mobile Phones Usage .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  3. 3. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Connected Car .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  4. 4. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Smart City .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  5. 5. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Around the World .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  6. 6. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Getting Started .NET CONFERENCE #1 IN UKRAINE
  7. 7. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data Pipeline .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  8. 8. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data Reference Architecture for IoT .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  9. 9. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Lambda Architecture .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  10. 10. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Batch Processing .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  11. 11. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Stream Processing .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  12. 12. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Best Practices .NET CONFERENCE #1 IN UKRAINE
  13. 13. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data Solutions :: Best Practices .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  14. 14. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Think Big. Start Small .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Scalable architecture. • Prototyping is a must. • Azure Stream Analytics is NOT a silver bullet. Data Source On-Boarding
  15. 15. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Data First .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Data formats are true headache. • Hard to extract valuable data. • Diagnostics telemetry must be processed.
  16. 16. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Don’t Interrupt Fast Path .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Don’t process between high-scale components unless you know what you’re doing. • Pass only valuable data which really needs to be processed in real-time.
  17. 17. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Security as a Culture .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Think about security, identity and management from the very beginning. • Security is a shared responsibility between Azure and the customer.
  18. 18. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Understand Business Model .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Data architecture more important than solution architecture • Plan and budget appropriately through simple predictable scenarios • Remember that you need to support millions of assets (for IoT)
  19. 19. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Data Collection .NET CONFERENCE #1 IN UKRAINE
  20. 20. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Collection :: How To .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Transactional data (data sync) One-Time data migration IoT data
  21. 21. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Collection :: Data Factory .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  22. 22. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Collection :: Data Hubs .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Event Hub IoT Hub
  23. 23. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Hubs :: Comparison .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Capability Standard Basic Event Hub Cloud-to-device Yes Yes Yes HTTPS, AMQP Yes Yes Yes MQTT Yes Yes File upload from devices Yes Yes File upload from devices Yes Yes Device Provisioning Service Yes Yes Cloud-to-device messaging Yes Device twin and management Yes IoT Edge Yes
  24. 24. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Hubs :: Main Problems .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Strict quotas & limits • Limited amount of partitions (32) • Need to plan additional 20% - 30% capacity • Limited number of concurrent connections • Works great if resources located in single region • Cannot provide latency < 0.1 sec
  25. 25. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Connection :: Kafka on HDInsight .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  26. 26. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Kafka on HDInsight :: Pros & Cons .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Open-source • Managed (SLA 99.9%) • Zookeeper out of the box • Not cost effective • Requires a lot of tuning
  27. 27. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Connection :: What to Choose? .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Capability Cloud-to-device communications Yes No No Device-initiated file upload Yes No No Device state information Device Twins No No Protocol support HTTPS, MQTT. AMQP AMQP, HTTPS Kafka Protocol Security Per-device identity Shared access policies Through external authentication services
  28. 28. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Connection :: Custom Solution .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Queue Device registry Data Connector
  29. 29. .NET LEVEL UP Custom Solution :: Connector .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  30. 30. .NET LEVEL UP Custom Solution :: Connector & Queue .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Device Registry Queue Connector
  31. 31. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Processing .NET CONFERENCE #1 IN UKRAINE
  32. 32. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Processing :: Azure Stream Analytics .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Easy to use • Easy to scale • Good integration with Azure Services • Relatively cheap • HIPAA, HITRUST, CSA STAR, ISO, GDRP • Limited SAQL • Vendor lock • Possible data loss and crash on invalid data
  33. 33. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Processing :: Stream Analytics :: Repartitioning .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  34. 34. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Processing :: HDInsight .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  35. 35. Тема доклада Тема доклада Тема доклада .NET LEVEL UP HDInsight :: Pros & Cons .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Not cost effective • Software is outdated • Software requires a lot of tuning • Provides managed versions of 1st class frameworks • Seamless integration • Frameworks are open-source
  36. 36. .NET LEVEL UP Processing :: HDInsight :: Storm .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Low latency. True streaming. High throughput • Excellent for non-complicated use cases • No state management • “At least once” guarantee
  37. 37. .NET LEVEL UP Processing :: HDInsight :: Spark .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • High throughput. Complete fault tolerance • Exactly once guarantees • No true streaming (micro batches). High latency. • Too complex to tune.
  38. 38. .NET LEVEL UP HDInsight :: Spark & Storm Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Demo JSON files { 100 users; 10 events per user } • 1 Redis node. 5 Kafka nodes. 10 worker nodes.
  39. 39. .NET LEVEL UP HDInsight :: Spark Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 70 sec 0 Latency Percentiles (130 000 events/s; 2-second micro-batches) 50 sec 60 sec 40 sec 10050 90 30 sec 25 10 sec 20 sec
  40. 40. .NET LEVEL UP HDInsight :: Spark Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 16 sec 0 Latency Percentiles (100 000 events/s; 10 second micro-batches) 8 sec 12 sec 4 sec 10050 90
  41. 41. .NET LEVEL UP HDInsight :: Spark Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 5 sec 0 Latency Percentiles (100 000 events/s; 3-second micro-batches) 3 sec 4 sec 2 sec 10050 90 1 sec 25
  42. 42. .NET LEVEL UP HDInsight :: Storm Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 3 sec 0 Latency Percentiles (up to 150 000 events/s) 2 sec 2.5 sec 1.5 sec 10050 90 1 sec 25
  43. 43. .NET LEVEL UP Processing :: What to Choose? .NET CONFERENCE #1 IN UKRAINE KYIV 2019 NOT Avro, JSON, or CSV Scale beyond 1 Gb/s Low latency is a must Zero data loss
  44. 44. Тема доклада Тема доклада Тема доклада KYIV 2019 One more thing… .NET CONFERENCE #1 IN UKRAINE
  45. 45. .NET LEVEL UP Big Data Solutions :: Cross-Cutting Concerns .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  46. 46. Тема доклада Тема доклада Тема доклада KYIV 2019 .NET CONFERENCE #1 IN UKRAINE Андрей Антиликаторов Solution Architect

×