PEnDAR webinar 3

© Predictable Network Solutions Ltd 2017 www.pnsol.com
PEnDAR Focus Group
Performance Ensurance by Design, Assuring Requirements
Third Webinar

Recap
Previous webinar recordings at:
https://attendee.gotowebinar.com/register/1615742612853821699
https://attendee.gotowebinar.com/register/4810357155138990338
Also on SlideShare: https://www.slideshare.net/pnsol-slides

3
Goals
• Consider how to enable Validation & Verification
of cost and performance
• For distributed and hierarchical systems
• via sophisticated but easy-to-use tools
• Supporting both initial and ongoing incremental
development
• Provide early visibility of cost/performance
hazards
• Avoiding costly failures
• Maximising the chances of successful in-budget
delivery of acceptable end-user outcomes
Partners
Supported by:
PEnDAR project

4
Our offer to you
Learn what we know about the laws of the
distributed world:
• Where the realm of the possible lies;
• How to push the envelope in a way that
benefits you most;
• How to innovate without getting your fingers
burnt.
Get an insight into the future:
• Understand more about achieving
sustainability and viability;
• See where competition may lie in the
future.
Our ‘ask’
Help us to find out:
• In which markets/sectors/application
domains will the benefits outweigh the
costs?
• Which are the major benefits?
• Which are just ‘nice to have’?
• How could this methodology be
consumed?
• Tools?
• Services?
Role of the focus group

© Predictable Network Solutions Ltd 2017
5
www.pnsol.com
Nature of performance
• In an ‘ideal world’, systems would always respond instantaneously
• and without exceptions/failures/errors
• In practice this doesn’t happen
• there is always some delay and some chance of failure: some impairment
• Thus performance is a privation
• the absence of impairment
• like ‘darkness’ or ‘silence’
• Quantity also matters
• require a certain rate or volume of responses with a given bound on
impairment

6Sources of impairment
Synchronisation
Implicit: resource sharing
Exclusive
Discrete:
Locks etc
Long timescales
Statistical
Continuous:
CPU, interface…
Short timescales
Explicit
Communication
Data
dependency
Causality
Information:
Takes time!
Communicated
Computed
Imperfection
Discrete
Statistical
Exceptions
Failures
Resource
exhaustion
Process algebra Stochastic process algebra ∆Q Framework

7
www.pnsol.com
Measure of performance: ∆Q
• ∆Q is a measure of the ‘quality impairment’ of an outcome
• the extent of deviation from ‘instantaneous and infallible’
• nothing in the real world is perfect so ∆Q always exists
• ∆Q is conserved
• a delayed outcome can’t be ‘undelayed’
• a failed outcome can’t be ‘unfailed’
• ∆Q can be traded
• e.g. accept more delay in return for more certainty of completion
• ∆Q has an algebra
• can manipulate it mathematically

8
∆Q can be represented with an
improper random variable
• Combines continuous and discrete
probabilities
• Thus encompasses normal
behaviour and exceptions/failures
in one model
∆Q is composable
• Supports hierarchical V&V
Representation of ∆Q
0.1
0.2
0.3
0.4
0.6
0.5
0.7
0.8
0.9
1.0
0.0
Cumulativeprobability
Response time
2 4 6 8 10 12 14 16
Tangible mass
encodes distribution
of response time
Intangible mass
encodes probability
of exception/failure

Outline of a coherent methodology

© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
10
Outcomes
Delivered
Resources
Consumed
Variability
Exception
/failure
Externally created
Mitigation
System impact
Propagation
Scale
Schedulability
Capacity
Distance
Number
Time
Space
Density

What is the state of the art?
How are performance issues dealt with within existing organisations?

12
Worst-case is important
• Safety-of-life applications
• aeronautics
• military
• FinTech
• e.g. Cardano blockchain project
• Hardware/silicon
• cost of post-deployment upgrade is very
high
• Performance issues addressed by
exhaustive testing
• checking corner cases
Not so much
• Telecomms and cloud
• QoE depends on many factors
• no contractual commitments for real-time
performance
• sell averages
• (Distributed) software
• upgrades are ‘cheap’ to distribute
• problems can be ‘fixed later’
• obsolescence/upgrade cycle is short
• performance issues may not emerge in the
commercially relevant timescale
When performance does(n’t) matter

13
• Only have to be a ‘bit better’ than
the competition
• No expectation that a high quality
product will be delivered
• only that the next version will be
‘better’
• No quantified requirements
• Little sense of responsibility for
potential issues
• fix problems as they manifest
• Feeds into contracts
• little scope for creating value
• cost control is the main focus
• Reputational risk can be deflected
• spending money
• following standards and general
‘best practice’
Sources of low ambition in the s/w industry

Barriers to engaging with performance and
resource cost in the SDLC

15
True but not believed
• It is possible to do better
• there is a lack of positive exemplars
• Failure to address this is expensive
• collective amnesia of costs of failure
• Sound formalisms are effective
• some poor experiences in the past
• unsound approaches
• lack of necessary skills/expertise
• Necessary to start from quantifiable
intent
• assumed only functional requirements
can be clearly captured
Believed but not true
• Perfection is at hand
• apart from a few ‘bugs/faults’ that can be
eliminated
• Performance issues can be fixed by
adding resources
• ignoring fundamental constraints
• Addressing performance issues is costly
and complex
• assumed not worth the value added
• Standard management principles for
managing variability are applicable to
distributed systems
• using KPIs
Failure of understanding

16
www.pnsol.com
Contractual and organisational issues
• Contractual chains fail to contain performance hazards
• performance failures can be turned into (small) financial penalties
• SLAs are inadequate – refer only to long-period averages
• Performance is either ‘everything’ or ‘nothing’ for the customer
• specific uses worry about corner cases – have established approaches
• general case is to neglect the issue and rely on contracts
• Organisations do not have time to engage with ‘complicated’ new
approaches
• want tools specialised to their particular problem domain and processes
• focus on cost-reduction prevents investment of staff time in improving skills

Conclusion

18
Virtualisation changes the game
• Introduces new contention for shared
resources
• and new digital supply-chain hazards
• Cloud suppliers make explicit what is and is
not under user control
• in-house virtualisation will have same
constraints implicitly
• Going to be applied very widely
• even in cyber-physical systems
• Failure to model the consequences will be
obvious
• who will hold the problem?
• contractual framing will be key
Capturing intent is crucial
• Forcing a decision of what to measure
• enables meaningful (and composable)
contracts
• Permits engineering tradeoffs
• creates potential to add value
• stops the race to the bottom
• Relate achievable results to resource
constraints
• fiscal
• technical
Key take-aways

19
www.pnsol.com
Final steps for PEnDAR
• Survey the focus group
• check consensus on conclusions from interviews
• invite suggestions for ways forward
• Create a public report
• anonymised findings for the wider community
• articulate the ‘gaps’ that have emerged
• process vs methodology
• hardware vs software V&V

Thank you!

PEnDAR webinar 3

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Similar to PEnDAR webinar 3

Similar to PEnDAR webinar 3 (20)

Recently uploaded

Recently uploaded (20)

PEnDAR webinar 3

Editor's Notes