Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1© 2017 Rogue Wave Software, Inc. All Rights Reserved. 1
Confronting the mission-
critical software testing
challenge
Epis...
2© 2017 Rogue Wave Software, Inc. All Rights Reserved. 2
Presenter
Rod Cope
CTO
Rogue Wave Software
rod.cope@roguewave.com...
3© 2017 Rogue Wave Software, Inc. All Rights Reserved. 3
1. A brief history of testing
2. Code security
3. Software reliab...
4© 2017 Rogue Wave Software, Inc. All Rights Reserved. 4
What is mission-critical?
5© 2017 Rogue Wave Software, Inc. All Rights Reserved. 5
A brief history of testing
6© 2017 Rogue Wave Software, Inc. All Rights Reserved. 6
The evolution of testing
1970s – 80s Debugging == testing
All I n...
7© 2017 Rogue Wave Software, Inc. All Rights Reserved. 7
Challenges with different methods
Advantages Disadvantages
Debugg...
8© 2017 Rogue Wave Software, Inc. All Rights Reserved. 8
Challenges with different methods
Advantages Disadvantages
Debugg...
9© 2017 Rogue Wave Software, Inc. All Rights Reserved. 9
Challenges with different methods
Advantages Disadvantages
Debugg...
10© 2017 Rogue Wave Software, Inc. All Rights Reserved. 10
Challenges with different methods
Advantages Disadvantages
Debu...
11© 2017 Rogue Wave Software, Inc. All Rights Reserved. 11
All have sources of risk
• Human error
• Software issues
• Hard...
12© 2017 Rogue Wave Software, Inc. All Rights Reserved. 12
Poll #1
What is the primary method you use to test code?
• Code...
13© 2017 Rogue Wave Software, Inc. All Rights Reserved. 13
Code security
14© 2017 Rogue Wave Software, Inc. All Rights Reserved. 14
Changing security landscape
More complex software
running insid...
15© 2017 Rogue Wave Software, Inc. All Rights Reserved. 15
Some research
22%
22%
24%
0% 5% 10% 15% 20% 25%
"Security is no...
16© 2017 Rogue Wave Software, Inc. All Rights Reserved. 16
One of the top flaws
in the 2015 National
Vulnerability
Databas...
17© 2017 Rogue Wave Software, Inc. All Rights Reserved. 17
Real vulnerability: GNU libc
CVE-2015-1472
https://sourceware.o...
18© 2017 Rogue Wave Software, Inc. All Rights Reserved. 18
GNU libc example: fail
19© 2017 Rogue Wave Software, Inc. All Rights Reserved. 19
GNU libc example: fix
20© 2017 Rogue Wave Software, Inc. All Rights Reserved. 20
Top four best security practices
• Numeric errors
• Code inject...
21© 2017 Rogue Wave Software, Inc. All Rights Reserved. 21
Poll #2
How much time do developers in your company spend
on se...
22© 2017 Rogue Wave Software, Inc. All Rights Reserved. 22
Software reliability
23© 2017 Rogue Wave Software, Inc. All Rights Reserved. 23
Why is reliability important?
May 2015
• Boeing 787 Dreamliner ...
24© 2017 Rogue Wave Software, Inc. All Rights Reserved. 24
January 6, 2016
• NEST ‘smart’ thermostat software update cause...
25© 2017 Rogue Wave Software, Inc. All Rights Reserved. 25
Key industry standards
The argument for standards compliance
• ...
26© 2017 Rogue Wave Software, Inc. All Rights Reserved. 26
MISRA C example
a |= 256;
b |= 128;
c |= 064;
Sets bit 8 of var...
27© 2017 Rogue Wave Software, Inc. All Rights Reserved. 27
Pulling it together
28© 2017 Rogue Wave Software, Inc. All Rights Reserved. 28
Why Agile?
Agile increasing
29© 2017 Rogue Wave Software, Inc. All Rights Reserved. 29
People over processes
Collaborate to build
trust and foster cha...
30© 2017 Rogue Wave Software, Inc. All Rights Reserved. 30
Continuous testing
• Check for security issues
• Measure confor...
31© 2017 Rogue Wave Software, Inc. All Rights Reserved. 31
Keys to successful CI
To work in a true CI environment test too...
32© 2017 Rogue Wave Software, Inc. All Rights Reserved. 32
Summary
• Identify and prevent vulnerabilities before release
S...
33© 2017 Rogue Wave Software, Inc. All Rights Reserved. 33
Q & A
34© 2017 Rogue Wave Software, Inc. All Rights Reserved. 34
Follow up
Free white paper:
Fitting static code analysis into c...
35© 2017 Rogue Wave Software, Inc. All Rights Reserved. 35
Missed this webinar? Watch it on-demand
How to achieve security...
36© 2017 Rogue Wave Software, Inc. All Rights Reserved. 36
Stay tuned
Confronting the mission-critical software testing
ch...
37© 2017 Rogue Wave Software, Inc. All Rights Reserved. 37
Upcoming SlideShare
Loading in …5
×

How to achieve security, reliability, and productivity in less time

244 views

Published on

This introductory session lays the foundation for boosting the effectiveness of mission-critical systems testing by covering industry best practices for code security, software reliability, and team productivity. For each area, you will learn how to mitigate the top issues by seeing real examples and understanding the tools and techniques to overcome them. This includes: The value of different testing methods; The importance of standards compliance; and understanding how DevOps and continuous integration fit in.

Published in: Software
  • Be the first to comment

  • Be the first to like this

How to achieve security, reliability, and productivity in less time

  1. 1. 1© 2017 Rogue Wave Software, Inc. All Rights Reserved. 1 Confronting the mission- critical software testing challenge Episode 1: How to achieve security, reliability, and productivity in less time Rod Cope CTO
  2. 2. 2© 2017 Rogue Wave Software, Inc. All Rights Reserved. 2 Presenter Rod Cope CTO Rogue Wave Software rod.cope@roguewave.com Twitter: @RodCope
  3. 3. 3© 2017 Rogue Wave Software, Inc. All Rights Reserved. 3 1. A brief history of testing 2. Code security 3. Software reliability 4. Pulling it together 5. Q&A Agenda
  4. 4. 4© 2017 Rogue Wave Software, Inc. All Rights Reserved. 4 What is mission-critical?
  5. 5. 5© 2017 Rogue Wave Software, Inc. All Rights Reserved. 5 A brief history of testing
  6. 6. 6© 2017 Rogue Wave Software, Inc. All Rights Reserved. 6 The evolution of testing 1970s – 80s Debugging == testing All I need is unit testing How did we survive without automated testing? DevOps is awesome! 1990s 2010s 2000s
  7. 7. 7© 2017 Rogue Wave Software, Inc. All Rights Reserved. 7 Challenges with different methods Advantages Disadvantages Debugging/printfs • Immediate • Minimal set up • Limited view of system • Limited tests • Doesn’t scale across code/team size Unit testing • Close to code • A form of documentation • Limited view of system • Limited tests • Cumbersome for single developer to set up Basic automated testing • Consistency and repeatability • Speed • Frees developer time • Can be slow to run • Can be slow to update DevOps/CI testing • Consistency and repeatability • Scalable & fast • Frees developer time • Initial set-up costs • Only effective for larger teams
  8. 8. 8© 2017 Rogue Wave Software, Inc. All Rights Reserved. 8 Challenges with different methods Advantages Disadvantages Debugging/printfs • Immediate • Minimal set up • Limited view of system • Limited tests • Doesn’t scale across code/team size Unit testing • Close to code • A form of documentation • Limited view of system • Limited tests • Cumbersome for single developer to set up Basic automated testing • Consistency and repeatability • Speed • Frees developer time • Can be slow to run • Can be slow to update DevOps/CI testing • Consistency and repeatability • Scalable & fast • Frees developer time • Initial set-up costs • Only effective for larger teams
  9. 9. 9© 2017 Rogue Wave Software, Inc. All Rights Reserved. 9 Challenges with different methods Advantages Disadvantages Debugging/printfs • Immediate • Minimal set up • Limited view of system • Limited tests • Doesn’t scale across code/team size Unit testing • Close to code • A form of documentation • Limited view of system • Limited tests • Cumbersome for single developer to set up Basic automated testing • Consistency and repeatability • Speed • Frees developer time • Can be slow to run • Can be slow to update DevOps/CI testing • Consistency and repeatability • Scalable & fast • Frees developer time • Initial set-up costs • Only effective for larger teams
  10. 10. 10© 2017 Rogue Wave Software, Inc. All Rights Reserved. 10 Challenges with different methods Advantages Disadvantages Debugging/printfs • Immediate • Minimal set up • Limited view of system • Limited tests • Doesn’t scale across code/team size Unit testing • Close to code • A form of documentation • Limited view of system • Limited tests • Cumbersome for single developer to set up Basic automated testing • Consistency and repeatability • Speed • Frees developer time • Can be slow to run • Can be slow to update DevOps/CI testing • Consistency and repeatability • Scalable & fast • Frees developer time • Initial set-up costs • Only effective for larger teams
  11. 11. 11© 2017 Rogue Wave Software, Inc. All Rights Reserved. 11 All have sources of risk • Human error • Software issues • Hardware issues And challenges: • Demands for shorter release times • Increasing feature complexity • Requirements for standards compliance • Increasing open source use
  12. 12. 12© 2017 Rogue Wave Software, Inc. All Rights Reserved. 12 Poll #1 What is the primary method you use to test code? • Code reviews • Unit tests • Manual tests at build time • Automated tests at build time • Automated testing using CI tools
  13. 13. 13© 2017 Rogue Wave Software, Inc. All Rights Reserved. 13 Code security
  14. 14. 14© 2017 Rogue Wave Software, Inc. All Rights Reserved. 14 Changing security landscape More complex software running inside systems Multiple sources of software being integrated Software has to run for many years This requires a very significant security, safety, & functional verification process Harder to secure code
  15. 15. 15© 2017 Rogue Wave Software, Inc. All Rights Reserved. 15 Some research 22% 22% 24% 0% 5% 10% 15% 20% 25% "Security is not considered important" "Security takes too much time" "I feel pressured to complete development" Why are companies not putting more emphasis on security in their applications? 49% 51% 0% 10% 20% 30% 40% 50% Yes No Is security a priority for your company? 2015 Survey of Automakers and Suppliers Ponemon Institute / Rogue Wave Software / Security Innovation
  16. 16. 16© 2017 Rogue Wave Software, Inc. All Rights Reserved. 16 One of the top flaws in the 2015 National Vulnerability Database Example: Memory buffer problems CWE-119: Software can read or write to locations outside of the boundaries of the memory buffer • Not checking size of input on copy • Bug allowing writing to arbitrary locations • Out-of-bounds read • Pointers outside expected range • Untrusted pointer dereference • Uninitialized pointers • Expired pointer references • Access of memory beyond buffer end
  17. 17. 17© 2017 Rogue Wave Software, Inc. All Rights Reserved. 17 Real vulnerability: GNU libc CVE-2015-1472 https://sourceware.org/ml/libc-alpha/2015-02/msg00119.html • Under certain conditions wscanf can allocate too little memory for the to-be-scanned arguments and overflow the allocated buffer. • Theoretically, any Linux machine connected to the internet, using this version, is at risk
  18. 18. 18© 2017 Rogue Wave Software, Inc. All Rights Reserved. 18 GNU libc example: fail
  19. 19. 19© 2017 Rogue Wave Software, Inc. All Rights Reserved. 19 GNU libc example: fix
  20. 20. 20© 2017 Rogue Wave Software, Inc. All Rights Reserved. 20 Top four best security practices • Numeric errors • Code injection • Improper input validation • Memory buffer problems • Numeric errors • Cryptographic issues • Code injection • Memory buffer problems • Numeric errors • Cryptographic issues • Code injection • Resource management errors • Numeric errors • Resource management errors • Improper access control • Improper input validation Clean design Methodical process Good tools Careful analysis
  21. 21. 21© 2017 Rogue Wave Software, Inc. All Rights Reserved. 21 Poll #2 How much time do developers in your company spend on security (as a percentage of work time)? • 0% • 1 – 25% • 26 - 50% • 51 – 75% • 76 – 100%
  22. 22. 22© 2017 Rogue Wave Software, Inc. All Rights Reserved. 22 Software reliability
  23. 23. 23© 2017 Rogue Wave Software, Inc. All Rights Reserved. 23 Why is reliability important? May 2015 • Boeing 787 Dreamliner had software bug which caused “total loss of electrical power” after 248 days December 2015 • A software error which calculates prison sentences caused more than 3,200 US prisoners to be released 49 days early on average
  24. 24. 24© 2017 Rogue Wave Software, Inc. All Rights Reserved. 24 January 6, 2016 • NEST ‘smart’ thermostat software update caused complete battery drain, shutting off heat during January • Matt Rogers, NEST co-founder & VP Eng: ”the bug took a few weeks to show up” • 2.5 million smart thermostats in U.S. alone
  25. 25. 25© 2017 Rogue Wave Software, Inc. All Rights Reserved. 25 Key industry standards The argument for standards compliance • Re-use the expert research of others • Complements existing testing approaches • Recognizable by customers • May already by a requirement Significantly reduces the cost of producing reliable software Security:
  26. 26. 26© 2017 Rogue Wave Software, Inc. All Rights Reserved. 26 MISRA C example a |= 256; b |= 128; c |= 064; Sets bit 8 of variable a (256 decimal = 0100000000 binary) Sets bit 7 of variable b (128 decimal = 0010000000 binary) Is bit 6 set? (64 decimal = 000100000 binary) Rule 7.1: Octal constants (other than zero) and octal escape sequences shall not be used. • No, because in C, any constant that begins with 0 is interpreted as an octal number. • So c is set to the wrong value!
  27. 27. 27© 2017 Rogue Wave Software, Inc. All Rights Reserved. 27 Pulling it together
  28. 28. 28© 2017 Rogue Wave Software, Inc. All Rights Reserved. 28 Why Agile? Agile increasing
  29. 29. 29© 2017 Rogue Wave Software, Inc. All Rights Reserved. 29 People over processes Collaborate to build trust and foster change Set expectations clearly Test and measure Share successes Enable with tools Pick the right artifacts Choose what to keep/throw away Lessons learned
  30. 30. 30© 2017 Rogue Wave Software, Inc. All Rights Reserved. 30 Continuous testing • Check for security issues • Measure conformance to standards • Examples of CI systems: TeamCity, Jenkins • Examples of test tool: static code analysis Automate testing Accept Check in Dev 1 Check in Dev 2 Check in Dev 3 Release Change Adjust and Track Feedback Review Next Iteration No! Release to Market Test Test Test Yes !
  31. 31. 31© 2017 Rogue Wave Software, Inc. All Rights Reserved. 31 Keys to successful CI To work in a true CI environment test tools must be designed to be: Automated Fast(er) Scalable Relevant To reduce feedback time, only changed code should be tested (including regression) By requiring minimal resources & deploying across multiple agents By reporting only the information that is required for the given context (example: only the diffs since the last build / build X) Supporting the most important CI build management systems
  32. 32. 32© 2017 Rogue Wave Software, Inc. All Rights Reserved. 32 Summary • Identify and prevent vulnerabilities before release Security: Clean design, methodical process, careful analysis, good tools • MISRA, OWASP, ISO 26262 Reliability: Adopt proven standards • Jenkins, static code analysis Automate with tools that are fast, scalable, and relevant 1 2 3
  33. 33. 33© 2017 Rogue Wave Software, Inc. All Rights Reserved. 33 Q & A
  34. 34. 34© 2017 Rogue Wave Software, Inc. All Rights Reserved. 34 Follow up Free white paper: Fitting static code analysis into continuous integration www.roguewave.com/resources/white-papers/static-code-analysis-into-continuous-integration
  35. 35. 35© 2017 Rogue Wave Software, Inc. All Rights Reserved. 35 Missed this webinar? Watch it on-demand How to achieve security, reliability, and productivity in less time Watch now.
  36. 36. 36© 2017 Rogue Wave Software, Inc. All Rights Reserved. 36 Stay tuned Confronting the mission-critical software testing challenge Feb. 8: Static analysis works for mission-critical systems, why not yours? Compare different techniques for testing by analysis and dive into static code analysis, including the types of problems found, barriers to adoption, and fitting it into various developer environments. Feb. 22: What if you could eliminate the hidden costs of development? Combat different types of development inefficiency by examining error-prone tasks, waiting for resources, “bug fix crowdsourcing,” and more to learn what the industry is doing about them and what you can do to get ahead.
  37. 37. 37© 2017 Rogue Wave Software, Inc. All Rights Reserved. 37

×