Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 46

.NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure

Oct. 28, 2019
• 1 like • 194 views

1

Share

Education

В данном докладе автор расскажет об опыте построения высоконагруженных Big Data решений на базе Microsoft Azure. Особое внимание будет уделено не только архитектурным подходам и best practices, но и многим ньюансам, тонким моментам и подводным камням, с которыми придется столкнуться. В каждом конкретном случае будут даны практические советы, cheat sheets и варианты решений, которые помогут с выбором подхода и позволят избежать потенциальных проблем. Также автор отдельно расскажет об опыте решения еще одной распростаненной задачи - проектировании Rules Engines для аналитики данных в реальном времени.

Recommended

Related Books

Free with a 30 day trial from Scribd

See all
Fighting Forward: Your Nitty-Gritty Guide to Beating the Lies That Hold You Back Hannah Brencher
(4/5)
Free
Dedicated: The Case for Commitment in an Age of Infinite Browsing Pete Davis
(5/5)
Free
No One Succeeds Alone: Learn Everything You Can from Everyone You Can Robert Reffkin
(4/5)
Free
Let's Talk About Hard Things Anna Sale
(4/5)
Free
How May I Serve Karen Mathews
(3.5/5)
Free
Decluttering at the Speed of Life: Winning Your Never-Ending Battle with Stuff Dana K. White
(4.5/5)
Free
The 7 Habits of Highly Effective People Stephen R. Covey
(4/5)
Free
The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life Mark Manson
(4.5/5)
Free
Present Over Perfect: Leaving Behind Frantic for a Simpler, More Soulful Way of Living Shauna Niequist
(4.5/5)
Free
Everything Is F*cked: A Book About Hope Mark Manson
(4.5/5)
Free
Girl, Stop Apologizing: A Shame-Free Plan for Embracing and Achieving Your Goals Rachel Hollis
(3.5/5)
Free
Dry: A Memoir Augusten Burroughs
(4.5/5)
Free
The 7 Habits of Highly Effective People Personal Workbook Stephen R. Covey
(4/5)
Free
Less Fret, More Faith: An 11-Week Action Plan to Overcome Anxiety Max Lucado
(4.5/5)
Free
Uninvited: Living Loved When You Feel Less Than, Left Out, and Lonely Lysa TerKeurst
(4.5/5)
Free
Girl, Wash Your Face: Stop Believing the Lies About Who You Are so You Can Become Who You Were Meant to Be Rachel Hollis
(3.5/5)
Free

Related Audiobooks

Free with a 30 day trial from Scribd

See all
The Authentic Leader: Five Essential Traits of Effective, Inspiring Leaders Merinda Smith
(3.5/5)
Free
A Body to Love: Cultivate Community, Body Positivity, and Self-Love in the Age of Social Media Angelina Caruso
(4.5/5)
Free
Full Out: Lessons in Life and Leadership from America's Favorite Coach Findaway
(3/5)
Free
Out with the Old, In with the You Tess Brigham
(4.5/5)
Free
Reshape Your Body Image Stacie Garland
(3.5/5)
Free
Necessary Conversations: Changing Your Mindset to Communicate Confidently and Productively Liz Nolley Tillman
(4.5/5)
Free
Beyond Small Talk: How to Have More Dynamic, Charismatic and Persuasive Conversations Patrick King
(4.5/5)
Free
The Book of Hope: A Survival Guide for Trying Times Jane Goodall
(4.5/5)
Free
Memory Craft: Improve Your Memory with the Most Powerful Methods in History Lynne Kelly
(5/5)
Free
Four Thousand Weeks: Time Management for Mortals Oliver Burkeman
(5/5)
Free
Fearless Leadership: How to Embody the Strength and Confidence of Great Leaders Kim Martin
(4/5)
Free
Getting More Done: Wielding Intention and Planning to Achieve Your Most Ambitious Goals Michelle Loucadoux
(4.5/5)
Free
10 Rules for Resilience: Mental Toughness for Families Joe De Sena
(5/5)
Free
Making Sense of Anxiety and Stress: A Comprehensive Stress Management Toolkit Saarim Aslam
(4.5/5)
Free
Empath Up!: How to Embrace the Gift of Empathy Cheryl Hutchinson
(4.5/5)
Free
Impact Players: How to Take the Lead, Play Bigger, and Multiply Your Impact Liz Wiseman
(5/5)
Free

.NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure

  1. 1. Тема доклада Тема доклада Тема доклада KYIV 2019 Проектирование и разработка Big Data решений на базе Microsoft Azure Андрей Антиликаторов .NET CONFERENCE #1 IN UKRAINE
  2. 2. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Mobile Phones Usage .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  3. 3. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Connected Car .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  4. 4. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Smart City .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  5. 5. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data :: Around the World .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Text
  6. 6. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Getting Started .NET CONFERENCE #1 IN UKRAINE
  7. 7. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data Pipeline .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  8. 8. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data Reference Architecture for IoT .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  9. 9. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Lambda Architecture .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  10. 10. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Batch Processing .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  11. 11. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Stream Processing .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  12. 12. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Best Practices .NET CONFERENCE #1 IN UKRAINE
  13. 13. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Big Data Solutions :: Best Practices .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  14. 14. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Think Big. Start Small .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Scalable architecture. • Prototyping is a must. • Azure Stream Analytics is NOT a silver bullet. Data Source On-Boarding
  15. 15. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Data First .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Data formats are true headache. • Hard to extract valuable data. • Diagnostics telemetry must be processed.
  16. 16. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Don’t Interrupt Fast Path .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Don’t process between high-scale components unless you know what you’re doing. • Pass only valuable data which really needs to be processed in real-time.
  17. 17. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Security as a Culture .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Think about security, identity and management from the very beginning. • Security is a shared responsibility between Azure and the customer.
  18. 18. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Best Practices :: Understand Business Model .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Data architecture more important than solution architecture • Plan and budget appropriately through simple predictable scenarios • Remember that you need to support millions of assets (for IoT)
  19. 19. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Data Collection .NET CONFERENCE #1 IN UKRAINE
  20. 20. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Collection :: How To .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Transactional data (data sync) One-Time data migration IoT data
  21. 21. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Collection :: Data Factory .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  22. 22. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Collection :: Data Hubs .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Event Hub IoT Hub
  23. 23. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Hubs :: Comparison .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Capability Standard Basic Event Hub Cloud-to-device Yes Yes Yes HTTPS, AMQP Yes Yes Yes MQTT Yes Yes File upload from devices Yes Yes File upload from devices Yes Yes Device Provisioning Service Yes Yes Cloud-to-device messaging Yes Device twin and management Yes IoT Edge Yes
  24. 24. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Hubs :: Main Problems .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Strict quotas & limits • Limited amount of partitions (32) • Need to plan additional 20% - 30% capacity • Limited number of concurrent connections • Works great if resources located in single region • Cannot provide latency < 0.1 sec
  25. 25. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Connection :: Kafka on HDInsight .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  26. 26. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Kafka on HDInsight :: Pros & Cons .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Open-source • Managed (SLA 99.9%) • Zookeeper out of the box • Not cost effective • Requires a lot of tuning
  27. 27. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Connection :: What to Choose? .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Capability Cloud-to-device communications Yes No No Device-initiated file upload Yes No No Device state information Device Twins No No Protocol support HTTPS, MQTT. AMQP AMQP, HTTPS Kafka Protocol Security Per-device identity Shared access policies Through external authentication services
  28. 28. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Data Connection :: Custom Solution .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Queue Device registry Data Connector
  29. 29. .NET LEVEL UP Custom Solution :: Connector .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  30. 30. .NET LEVEL UP Custom Solution :: Connector & Queue .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Device Registry Queue Connector
  31. 31. Тема доклада Тема доклада Тема доклада KYIV 2019 Big Data Solutions :: Processing .NET CONFERENCE #1 IN UKRAINE
  32. 32. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Processing :: Azure Stream Analytics .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Easy to use • Easy to scale • Good integration with Azure Services • Relatively cheap • HIPAA, HITRUST, CSA STAR, ISO, GDRP • Limited SAQL • Vendor lock • Possible data loss and crash on invalid data
  33. 33. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Processing :: Stream Analytics :: Repartitioning .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  34. 34. Тема доклада Тема доклада Тема доклада .NET LEVEL UP Processing :: HDInsight .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  35. 35. Тема доклада Тема доклада Тема доклада .NET LEVEL UP HDInsight :: Pros & Cons .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Not cost effective • Software is outdated • Software requires a lot of tuning • Provides managed versions of 1st class frameworks • Seamless integration • Frameworks are open-source
  36. 36. .NET LEVEL UP Processing :: HDInsight :: Storm .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Low latency. True streaming. High throughput • Excellent for non-complicated use cases • No state management • “At least once” guarantee
  37. 37. .NET LEVEL UP Processing :: HDInsight :: Spark .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • High throughput. Complete fault tolerance • Exactly once guarantees • No true streaming (micro batches). High latency. • Too complex to tune.
  38. 38. .NET LEVEL UP HDInsight :: Spark & Storm Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Demo JSON files { 100 users; 10 events per user } • 1 Redis node. 5 Kafka nodes. 10 worker nodes.
  39. 39. .NET LEVEL UP HDInsight :: Spark Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 70 sec 0 Latency Percentiles (130 000 events/s; 2-second micro-batches) 50 sec 60 sec 40 sec 10050 90 30 sec 25 10 sec 20 sec
  40. 40. .NET LEVEL UP HDInsight :: Spark Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 16 sec 0 Latency Percentiles (100 000 events/s; 10 second micro-batches) 8 sec 12 sec 4 sec 10050 90
  41. 41. .NET LEVEL UP HDInsight :: Spark Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 5 sec 0 Latency Percentiles (100 000 events/s; 3-second micro-batches) 3 sec 4 sec 2 sec 10050 90 1 sec 25
  42. 42. .NET LEVEL UP HDInsight :: Storm Benchmarking .NET CONFERENCE #1 IN UKRAINE KYIV 2019 3 sec 0 Latency Percentiles (up to 150 000 events/s) 2 sec 2.5 sec 1.5 sec 10050 90 1 sec 25
  43. 43. .NET LEVEL UP Processing :: What to Choose? .NET CONFERENCE #1 IN UKRAINE KYIV 2019 NOT Avro, JSON, or CSV Scale beyond 1 Gb/s Low latency is a must Zero data loss
  44. 44. Тема доклада Тема доклада Тема доклада KYIV 2019 One more thing… .NET CONFERENCE #1 IN UKRAINE
  45. 45. .NET LEVEL UP Big Data Solutions :: Cross-Cutting Concerns .NET CONFERENCE #1 IN UKRAINE KYIV 2019
  46. 46. Тема доклада Тема доклада Тема доклада KYIV 2019 .NET CONFERENCE #1 IN UKRAINE Андрей Антиликаторов Solution Architect

×