Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Designing Big Data Systems Like a Pro

599 views

Published on

Smart Decisions: An Architecture Design Game

Published in: Software
  • Great job on that, also I've been using a service called MLMRC which automates your downline building. Works with any MLM business. Try it: www.mlmrc.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Designing Big Data Systems Like a Pro

  1. 1. webinar@softserveinc.com Designing Big Data Systems Like a Pro Smart Decisions: An Architecture Design Game Humberto Cervantes, Serge Haziyev, Olha Hrytsay, Rick Kazman September 2015
  2. 2. Presenters Dr. Rick Kazman is a Professor at the University of Hawaii and a Principal Researcher at the Software Engineering Institute of Carnegie Mellon University (SEI). His primary research interests are software architecture, design and analysis tools, software visualization, and software engineering economics. Rick has created several highly influential methods and tools for architecture analysis, including the ATAM (Architecture Tradeoff Analysis Method). Dr. Humberto Cervantes is a professor at Universidad Autónoma Metropolitana–Iztapalapa in Mexico City. His primary research interests include software architecture design methods and their adoption in industrial settings. Dr. Cervantes is also a consultant for software development companies in topics related to software architecture. He holds the Software Architecture Professional and ATAM Evaluator certificates from the SEI. Serge Haziyev is VP of Software Architecture at SoftServe. Serhiy has more than 17 years of experience in enterprise-level solutions including Big Data, SaaS/Clouds, SOA and Carrier-grade telecommunication services. He specializes in software architecture methodologies, architectural patterns and software development practices for large and complex projects in multiple industry verticals, including healthcare. Olha Hrytsay works as a BI/DW consultant at SoftServe, Inc., a leading global outsourced product and application development company. Olha has more than seven years of experience in building business intelligence, data warehousing, and big-data solutions for a number of global companies in the network security, health care, and finance business domains. Her current activities at SoftServe include leading the BI Center of Excellence as well as design and implementation of data warehousing, data visualization, and analytics solutions.
  3. 3. webinar@softserveinc.comwebinar Agenda Game Motivation Game Rules Game Scenario Questions
  4. 4. webinar@softserveinc.comwebinar Agenda Game Motivation Game Rules Game Scenario Questions
  5. 5. webinar@softserveinc.comwebinar Game Motivation This game intends to illustrate the essentials of architecture design using an iterative method such as ADD. You will be competing against other software architects (or other teams) from rival companies, so you need to make smart design decisions or else your competitors will leave you behind!
  6. 6. webinar@softserveinc.comwebinar Past Game Events SATURN 2015 Architecture Gathering 2014 SEI ACE Educators Workshop 2015
  7. 7. Game Inventory 1. Playing cards 3. Game board 4. Dice 5. Markers 2. Game scenario 6. Scorecard Download materials at: www.smartdecisionsgame.com
  8. 8. Brief ADD Method Introduction
  9. 9. Design Concepts Catalog
  10. 10. webinar@softserveinc.comwebinar Agenda Game Motivation Game Rules Game Scenario Questions
  11. 11. Game Rules ADD Step 1: Review Inputs Let’s start by reviewing the inputs to the design process…
  12. 12. Input Requirements 1. Functional Requirements 3. Constraints 2. Quality Attributes
  13. 13. Game Rules The game is played in rounds which represent the iterations. For each round the game provides: - Iteration goal (i.e. selected drivers) - Element to refine ADD Step 2: Establish iteration goal by selecting drivers ADD Step 3: Choose one or more elements of the system to refine
  14. 14. webinar@softserveinc.comwebinar Let’s Start!
  15. 15. webinar@softserveinc.comwebinar Iteration 1 Goal: Logically Structure The System Drivers for the iteration: - Ad-Hoc Analysis - Real-time Analysis - Unstructured data processing - Scalability - Cost Economy Big Data System Element to refine:
  16. 16. Game Rules You will make the design decision of selecting design concepts: - Reference architectures* - Technology families* - Specific technologies * In the game they are considered as a type of pattern ADD Step 4: Choose one or more design concepts that satisfy the selected drivers
  17. 17. Game Rules: Design Concepts Cards Name and type of design concept Influence on drivers Technologies Patterns
  18. 18. Iteration 1 Goal: Logically Structure The System Select 1 Reference Architecture Card Drivers for the iteration: - Ad-Hoc Analysis - Real-time Analysis - Unstructured data processing - Scalability - Cost Economy Alternatives: • Extended Relational • Pure Non-Relational • Data Refinery • Lambda Architecture Big Data System Element to refine:
  19. 19. Fill The Scorecard Smart Decisions: Game Scorecard  © 2015 H. Cervantes, S. Haziyev, O. Hrytsay, R. Kazman      Player name: _______________________________________________________      Iteration  #1  Iteration  #2  Iteration  #3  Iteration  #4  Iteration  #5    (a) Design  Decisions    (Names of  selected design  concept(s))                        (b) Driver  selection points  (from cards)                (c) Instantiation  points   (from dice)              (d) Analysis  bonus points  (from review)            Final  score:  (e) Iteration total  (b + c + d)                  Fill (b) by adding the points for the drivers considered for the iteration, in this case: - Ad-Hoc Analysis (2.5) - Real-time Analysis (3) - Unstructured data processing (3) - Scalability (3) - Cost Economy (3) = 1 Point 2.5+3+3 +3+3=14.5 Lambda Architecture Record design decisions in (a) Some iterations require you to draw two cards. For these iterations you will need to: - Record the name of both design concepts - Add the points for both of the cards Please note that some drivers may not be associated to both cards, for example: - Performance (for Family and Technology) - Compatibility (for Family) - Reliability (for Technology) In these cases, you only count points for the drivers that are associated to the card
  20. 20. Introduction ADD Step 5: Instantiate elements, allocate responsibilities and define interfaces. ADD Step 6: Sketch views and record design decisions You will: - Record the design decision - Throw two dice to simulate how well you instantiate your selected design concepts
  21. 21. Smart Decisions: Game Scorecard  © 2015 H. Cervantes, S. Haziyev, O. Hrytsay, R. Kazman      Player name: _______________________________________________________      Iteration  #1  Iteration  #2  Iteration  #3  Iteration  #4  Iteration  #5    (a) Design  Decisions    (Names of  selected design  concept(s))                        (b) Driver  selection points  (from cards)                (c) Instantiation  points   (from dice)              (d) Analysis  bonus points  (from review)            Final  score:  (e) Iteration total  (b + c + d)                  Fill The Scorecard Roll two dice once and add or subtract points according to the following table, fill (c). 2.5+3+3 +3+3=14.5 Lambda Architecture +2
  22. 22. Introduction We will review the decisions together. The first iteration will be reviewed now but the rest will be reviewed at the end. ADD Step 7: Perform Analysis of Current Design and Review Iteration goal and Design Objective
  23. 23. webinar@softserveinc.comwebinar Iteration 1 Review Design decision Driver points Bonus points Comments Extended Relational 3+2+2+2+1=10 -4 This reference architecture is less appropriate for this solution mostly because of cost and real-time analysis limitation Pure Non-Relational 2+2.5+3+3+3=13.5 This reference architecture is closer to the goal than the others except Lambda Architecture Lambda Architecture (Hybrid) 2.5+3+3+3+3=14.5 +2 This is the most appropriate reference architecture for this solution! From the provided reference architectures Lambda Architecture promises the largest number of benefits, such as access to real-time and historical data at the same time. Data Refinery (Hybrid) 3+1+3+2+1=10 -4 This reference architecture is less appropriate for this solution mostly because of cost and real-time analysis limitation Score Ad-Hoc Analysis, Real-time Analysis, Unstructured data processing, Scalability, Cost Economy
  24. 24. webinar@softserveinc.comwebinar Smart Decisions: Game Scorecard  © 2015 H. Cervantes, S. Haziyev, O. Hrytsay, R. Kazman      Player name: _______________________________________________________      Iteration  #1  Iteration  #2  Iteration  #3  Iteration  #4  Iteration  #5    (a) Design  Decisions    (Names of  selected design  concept(s))                        (b) Driver  selection points  (from cards)                (c) Instantiation  points   (from dice)              (d) Analysis  bonus points  (from review)            Final  score:  (e) Iteration total  (b + c + d)                  Fill The Scorecard 2.5+3+3 +3+3=14.5 Lambda Architecture +2 Add bonus points, if any and fill (d) +2 Sum the points and calculate the total for the iteration in (e) 18.5
  25. 25. webinar@softserveinc.comwebinar Agenda Game Motivation Game Rules Game Scenario Questions
  26. 26. Game Scenario: Big Data System Web Servers 24/7 Operations, Support Engineers, Developers Real-time Dashboard Management Static Reports • Real-time monitoring • Full-text search • Historical static reports • Available through BI corporate tool • Hundreds of servers • Massive logs from multiple sources Data Scientists/ Analysts Ad-Hoc Reports • Raw and aggregated historical data • Ad-hoc analysis • Human-time queries UC-1,2 UC-3 UC-4 UC1 - Monitor online services UC2 - Troubleshoot online service issues UC3 - Provide management reports UC4 - Provide ad-hoc data analytics
  27. 27. Big Data System: Quality Attributes and Constraints
  28. 28. webinar@softserveinc.comwebinar 1st Decision: Lambda Architecture Batch Layer Serving Layer Speed Layer Master Dataset Data Stream Real-time Views Pre-Computing Batch Views Query & Reporting Source: http://lambda-architecture.net/
  29. 29. Lambda Architecture: Design Iterations Batch Layer Serving Layer Speed Layer Master Dataset Data Stream Real-time Views Pre-Computing Batch Views Query & Reporting 2 3 4 5 Iteration 2 – Refine Data Stream element Iteration 3 – Refine Master Dataset element Iteration 4 – Refine Batch Views element Iteration 5 – Refine Real-time Views element
  30. 30. Iteration 2: Data Stream Technology Alternatives
  31. 31. Iteration 2 review Design decision Driver points Bonus points Comments Data Collector 2+3=5 +2 Additional bonus is added for extensibility Distributed Message Broker 3+1=4 Design decision Driver points Bonus points Comments Apache Flume 2+2=4 Logstash 2+2=4 Fluentd 2+3=5 RabbitMQ 2+2=4 Apache Kafka 3+2=5 +2 Additional bonus for easier deployment and configuration comparing with other alternatives Amazon SQS 0 Disqualified due to deployment constraint (support On-premise and Cloud) Apache ActiveMQ 2+2=4 Family card: score Performance and Compatibility Technology card: score Performance and Reliability
  32. 32. webinar@softserveinc.comwebinar Game Result Sample Batch Layer Serving Layer Speed Layer Master Dataset Data Stream Real-time Views Pre-Computing Batch Views Query & Reporting
  33. 33. webinar@softserveinc.comwebinar Agenda Game Motivation Game Rules Game Scenario Questions
  34. 34. webinar@softserveinc.comwebinar QUESTIONS & ANSWERS E-mail your questions to webinar@softserveinc.com
  35. 35. webinar@softserveinc.comwebinar Oct 13 Outsourcing: Been there, done that, didn’t work out Nov 11 Software Application Management OUR NEXT WEBINARS
  36. 36. webinar@softserveinc.comwebinarwww.smartdecisionsgame.com Thank you!

×