Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PEnDAR webinar 1 with notes

961 views

Published on

First webinar from the PEnDAR project, outlining the distributed system design challenge, and begin to investigate the application of advanced system engineering techniques for all phases of system design to ensure that critical cost and performance targets can be met. The goal is to make expensive performance failures and cost overruns a thing of the past!

Published in: Technology
  • Be the first to comment

  • Be the first to like this

PEnDAR webinar 1 with notes

  1. 1. © Predictable Network Solutions Ltd 2017 www.pnsol.com PEnDAR Focus Group Performance Ensurance by Design, Assuring Requirements First Webinar
  2. 2. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 2 Goals • Consider how to enable Validation & Verification of cost and performance • For distributed and hierarchical systems • via sophisticated but easy-to-use tools • Supporting both initial and ongoing incremental development • Provide early visibility of cost/performance hazards • Avoiding costly failures • Maximising the chances of successful in-budget delivery of acceptable end-user outcomes Partners Supported by: PEnDAR project
  3. 3. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 3 Our offer to you Learn what we know about the laws of the distributed world: • Where the realm of the possible lies; • How to push the envelope in a way that benefits you most; • How to innovate without getting your fingers burnt. Get an insight into the future: • Understand more about achieving sustainability and viability; • See where competition may lie in the future. Our ‘ask’ Help us to find out: • In which markets/sectors/application domains will the benefits outweigh the costs? • Which are the major benefits? • Which are just ‘nice to have’? • How could this methodology be consumed? • Tools? • Services? Role of the focus group
  4. 4. © Predictable Network Solutions Ltd 2017 www.pnsol.com The system delivery challenge
  5. 5. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 5 System delivery challenge System Requirements Time pressure Complexity Cost/resource constraints Forces hierarchical decomposition Forces parallel development Change during development Subcontracted subsystems Use of COTs componentsVague/contradictory Leave ‘hard’ problems to the end Existing asset re-use constraints
  6. 6. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 6 How to specify and verify subsystem requirements/acce ptance criteria Functional outcomes Performance outcomes Resource constraints MEET Risks MANAGE Peak Loads Integration testing Rework MINIMISE Post-deployment scaling System delivery challenge
  7. 7. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 7 Constrained performance hazards Unconstrained performance hazards Distributed Centralised Dedicated Shared Resources Structure Traditional avionics Cloud apps Traditional mainframe Standalone microcontrollers Modern avionics Client-server systems Established approaches running out of steam?
  8. 8. © Predictable Network Solutions Ltd 2017 www.pnsol.com Dimensions of the challenge
  9. 9. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 9 Outcomes Delivered Resources Consumed Variability Exception /failure Externally created Mitigation System impact Propagation Scale Schedulability Capacity Distance Number Time Space Density
  10. 10. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 10 Outcomes Delivered Resources Consumed Variability Exception /failure Scale
  11. 11. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 11 Outcomes Delivered Resources Consumed Variability Exception /failure Scale Distance Number Time Space Schedulability Capacity Density
  12. 12. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 12 Outcomes Delivered Resources Consumed Variability Exception /failure Mitigation Propagation Scale Schedulability Capacity Distance Number Time Space Density
  13. 13. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 13 Outcomes Delivered Resources Consumed Variability Exception /failure Externally created Mitigation Propagation Scale Schedulability Capacity Distance Number Time Space Density
  14. 14. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 14 Outcomes Delivered Resources Consumed Variability Exception /failure Externally created Mitigation System impact Propagation Scale Schedulability Capacity Distance Number Time Space Density
  15. 15. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 15 Outcomes Delivered Resources Consumed Variability Exception /failure Externally created Mitigation System impact Propagation Scale Schedulability Capacity Distance Number Time Space Density
  16. 16. © Predictable Network Solutions Ltd 2017 www.pnsol.com Practical methodology
  17. 17. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 17 Support stages of the SDLC • Design • Feasibility analysis • Hierarchical decomposition • Subsystem acceptance criteria • Verification • Checking delivery of quantified outcomes • Evaluating resource usage • Re-verification during system lifetime • Validation • Quantification of performance criteria • Checking coverage and consistency Quantify hazards • Failure to meet outcome requirements • Physical constraints • Schedulability constraints • Supply chain constraints • Failure to meet resource constraints • Scaling • Correlations Interaction with System Development Life Cycle
  18. 18. © Predictable Network Solutions Ltd 2017 18 www.pnsol.com Quantifying intent • The key challenge is to establish quantified intentions • For outcomes/resources/costs • Then a variety of mathematical techniques can be applied • queuing theory • large deviation theory • ∆Q algebra • This is “only rocket science” • Not brain surgery!
  19. 19. © Predictable Network Solutions Ltd 2017 19 www.pnsol.com POLL Imagine two technical scenarios: • In the first, a client asks for “a replacement for its phone system”. • In the second, a client asks for “a VoIP system with an average MOS of 4.2, the ability to take 30 concurrent calls, a failover system for when it hits capacity (e.g. voicemail/recorded message), first priority for emergency calls and second priority for current calls”. If the first scenario is 1 and the second is 5, where on a scale of 1 to 5 would you place the requirements you generally encounter in your field?
  20. 20. © Predictable Network Solutions Ltd 2017 20 www.pnsol.com 20 Quantifying timeliness Outcome requirement: Suppose we could get the marketing department to specify how long it’s acceptable to wait for the first frame of a video: • 50% of responses within 3 seconds • 95% of responses within 10 seconds • 99.9% of responses within 15s • 0.1% failure rate
  21. 21. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 21 • Suppose the black line shows the delivered CDF • From measurement, simulation or analysis • This is everywhere above and to the left of the requirement curve • This means that the timeliness requirement is satisfied • If not, there is a performance hazard Meeting a timeliness requirement
  22. 22. © Predictable Network Solutions Ltd 2017 www.pnsol.com Questionnaire analysis Who are you and what do you think?
  23. 23. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 23 0 1 2 3 4 5 Telecomms / Infrastructure Telecomms / Services Cloud Services media production infrastructure Research & Innovation App Development / Design IoT Security Operations Aerospace middleware App Development / Testing 0 1 2 3 4 Management System Design / Architect Capability development / Research Marketing / Sales 0 1 2 3 4 5 6 7 8 Requirements Definition System / Product Design System / Product Development System / Product Integration and/or Testing Deployment Operations and Maintainance Performance Monitoring and Usage Analytics What field do you work in? What is your role/what level do you work at? Which parts of the product lifecycle are you involved in?
  24. 24. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 24 Where do system/product requirements come from? 0 1 2 3 4 1 2 3 4 5 Precise and unambiguous? 0 1 2 3 4 1 2 3 4 5 Quantifiable and quantified? 0 1 2 3 4 5 6 They come directly from the client/customer They are derived in collaboration with the client/customer Generated internally using experience / engineering judgement Derived from standards / regulations or other definitive external sources Developed using internal funding and experience of anticipated customer needs
  25. 25. © Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com 25 How / when does your organisation determine whether requirements have been / will be met? 0 1 2 3 4 5 6 7 Redesign / rework at our client/customer's cost Carry on, leaving the issue to be resolved later Attempt to change the requirements Redesign / rework at my organisation's cost What is the typical response to the discovery that requirements will not be / have not been met? 0 1 2 3 Through customer feedback after deployment/delivery, e.g. by waiting for complaints or a lack of them There are test and integration steps whose completion is determined informally, e.g. by demonstration There is a test and integration methodology with formal sign-off procedures that are purely internal There is a test and integration methodology with formal sign-off procedures involving the client/customer
  26. 26. © Predictable Network Solutions Ltd 2017 26 www.pnsol.com Importance of performance and resources For the systems / products you produce / operate, how critical is performance to success? 4.4 To what extent is your role accountable for aspects of performance that are critical for success? 3.2 To what extent do you, in your role, have influence/control over how resources are shared? 2.8 Given that resources are shared, other parts of the system / other systems may consume resources on which you rely: to what extent is this a concern? 4.0 Given that resources are shared, your system / product may consume resources on which others rely: to what extent is this a concern? 3.8 My role is primarily concerned with the performance impact on the individual users / uses My role is concerned with both the overall performance and the impact on individual users / uses System performance is not a concern of my role
  27. 27. © Predictable Network Solutions Ltd 2017 www.pnsol.com Summing up
  28. 28. © Predictable Network Solutions Ltd 2017 www.pnsol.com Click to edit Master title style 28 Benefits • Avoid infeasible developments • 'fail early’ • prune blind alleys • Address scalability early • including real-world constraints • avoid 'heavy tail’ • See the whole risk landscape • not just the 'first problem’ • Be able to write a safety case • Even for shared-resource distributed systems Costs • Have to ‘quantify intent’ • may meet resistance • Effort in adopting new tools and techniques • Upfront work needed before “real” development starts • may not fit expectations/metrics of ‘progress’ Expected benefits and costs of this approach
  29. 29. © Predictable Network Solutions Ltd 2017 www.pnsol.com Thank you!

×