Can AppSec Training Really Make a
Smart Developer?
Research From Denim Group
John B. Dickson, CISSP
@johnbdickson

Novembe...
About Me

• Application Security
Enthusiast
• Security Professional
• ISSA Distinguished Fellow
• MBA-type and Serial Entr...
About Denim Group

Denim Group Overview
•Professional services firm that builds & secures enterprise
applications
 Extern...
“I personally believe that training users in security
is generally a waste of time, and that the money
can be spent better...
How Developer Training is Different
•Both trying to change behaviors
– Target audience has more power to say “no”
– Deadli...
Yet Training is Mandated
•PCI DSS 3.0
 Train developers in secure coding techniques, including how to
avoid common coding...
But Results Are Not Measured
•Harvard Business Review
– Large-scale organization development is rare
– Measurement of resu...
Growth & Turnover Spur Sense of
Urgency
– Software development field growing 30%
– Turnover
• Industry – 14-15%
• General ...
Research Overview
•Focus: Assess the software developers depth of software
security knowledge
•Purpose: To measure the imp...
Respondent Demographics

Hosted by OWASP && the NYC Chapter
Hosted by OWASP the NYC Chapter
Respondent Demographics

Hosted by OWASP && the NYC Chapter
Hosted by OWASP the NYC Chapter
Respondent Demographics

Hosted by OWASP && the NYC Chapter
Hosted by OWASP the NYC Chapter
Methodology
 15 Multiple Choice Quiz-Style Questions
 Targeted at Software Developers
 Varied by years of experience, a...
Hypotheses
1. Most software developers do not have a basic
understanding of software security concepts.
2. Software securi...
Sample Questions
4. If an attacker were able to view sensitive customer records they should
not have had access to, this w...
Key Survey Results
• Enterprises of more than 10,000 personnel had the lowest
secure coding knowledge

Hosted by OWASP && ...
Key Survey Results
• Architects and software developers had a much higher level of
knowledge than QA, yet in many organiza...
Key Survey Results
•Slightly more than half of the respondents correctly answered
basic awareness questions on application...
Key Survey Results
•Almost 100 percent could define input validation, demonstrating a
choppy understanding of advanced sec...
Key Survey Results
•The majority of the respondents had no prior secure
coding training, which might be surprising

Hosted...
Key Survey Results
• There was no correlation between years of experience
and knowledge of secure coding highlighting the
...
Key Survey Results
•The respondents that had had more than 3 days of
app sec training in the past were able to answer more...
Key Survey Results
•100% correctly identified where cross site scripting
executes after completing training, an increase o...
Key Survey Results
•The number of respondents able to correctly identify
what is application security more than doubled af...
CONCLUSION
• Retention rose by more than 25 percent after completing
secure coding training.
• Other statistics also repor...
Software Developers Learn Differently
than Companies “Teach”
• Teaching methods are formalized and
structured in order to ...
So How Do Developers Learn?
•Informally and and in an unstructured way
via:
•
•
•
•
•

Blogs & RSS feeds
Social media with...
Don’t Ignore Basics of Training
•Refresher training is still needed
•Training must be included in performance
plans
•Manag...
Incentives Matter!

Hosted by OWASP && the NYC Chapter
Hosted by OWASP the NYC Chapter
CONCLUSION
• Software developers largely do not understand
key software security concepts
 73% of respondents “failed” th...
Where do we Go from Here?

Hosted by OWASP && the NYC Chapter
Hosted by OWASP the NYC Chapter
Questions and Answers?
John B. Dickson
@johnbdickson
john@denimgroup.com

Hosted by OWASP && the NYC Chapter
Hosted by OWA...
Upcoming SlideShare
Loading in …5
×

Appsec2013 presentation-dickson final-with_all_final_edits

586 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
586
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • http://gridinternational.com/fm_files/HBR_Articles/HBR_Breakthrough_in_Organization_Development_-_Harvard_Business_Review1.pdf
    Strangely enough, large-scale organization development is rare, and the measurement of results is even rarer. Even though
    management has sought for years to grasp and implement the important findings of behavioral science research, the task has
    proved more difficult than it first seemed. Many findings are subtle and complex. Other findings relate to individual insights or
    knowledge which is hard to build into the organization’s life stream.
    Connecting Workforce Analytics to Better Business Results
    http://talent.sumtotalsystems.com/rs/sumtotalsystems/images/HBR_SumTotal_Report_Aug2013_rev1.pdf
  • Growth & Turnover
    Employment of software developers is projected to grow 30 percent from 2010 to 2020,
    The main reason for the rapid growth is:
    a large increase in the demand for computer software
    Mobile technology requires new applications
    the healthcare industry is greatly increasing its use of computer systems and applications
    Source: http://www.bls.gov/ooh/Computer-and-Information-Technology/Software-developers.htm#tab-6
    http://compforce.typepad.com/compensation_force/2009/03/2008-turnover-rates-by-industry.html
    14-15% Source: http://www.shrm.org/research/benchmarks/documents/trends%20in%20turnover_final.pdf
  • 19% passing rate among companies with over 10,000 employees
    Observation/Note: Training tactics and frequency in larger organizations?
  • Yet, only about 20 percent could define the terms application security or threat modeling, the most basic of security skills
  • 19% passing rate among companies with over 10,000 employees
    Observation/Note: Training tactics and frequency in larger organizations?
  • 19% passing rate among companies with over 10,000 employees
    Observation/Note: Training tactics and frequency in larger organizations?
  • 19% passing rate among companies with over 10,000 employees
    Observation/Note: Training tactics and frequency in larger organizations?
  • 19% passing rate among companies with over 10,000 employees
    Observation/Note: Training tactics and frequency in larger organizations?
  • 19% passing rate among companies with over 10,000 employees
    Observation/Note: Training tactics and frequency in larger organizations?
  • Average score increased from 59% to 74%
    “Pass” rate increased from 27% to 67%
    Observation – 33% of respondents still failed after training
  • Bryan Beveryly I follow a series of RSS feeds that I keep pretty regularly pruned. I focus on feeds that cover applied technology and development practices over theory. The .NET teams have some okay feeds, but Scott Hu is a real no-shit .NET developer and he has a much better set of content. Usually the good ones lead me to other good ones. The lousy ones or ones with stale content get pruned. 
    I have moved all of my framework feeds to just following them on Twitter because it's generally less noise. That's how I find out about updates to Grails/Groovy, Spring libraries, .NET MVC, Java, etc.
  • Bryan Beveryly I follow a series of RSS feeds that I keep pretty regularly pruned. I focus on feeds that cover applied technology and development practices over theory. The .NET teams have some okay feeds, but Scott Hu is a real no-shit .NET developer and he has a much better set of content. Usually the good ones lead me to other good ones. The lousy ones or ones with stale content get pruned. 
    I have moved all of my framework feeds to just following them on Twitter because it's generally less noise. That's how I find out about updates to Grails/Groovy, Spring libraries, .NET MVC, Java, etc.
  • Bryan Beveryly I follow a series of RSS feeds that I keep pretty regularly pruned. I focus on feeds that cover applied technology and development practices over theory. The .NET teams have some okay feeds, but Scott Hu is a real no-shit .NET developer and he has a much better set of content. Usually the good ones lead me to other good ones. The lousy ones or ones with stale content get pruned. 
    I have moved all of my framework feeds to just following them on Twitter because it's generally less noise. That's how I find out about updates to Grails/Groovy, Spring libraries, .NET MVC, Java, etc.
  • http://www.entrepreneur.com/article/220569
    http://blogs.hbr.org/2010/06/how-to-translate-training-into/
    Luckily the “fix” for these kinds of programs is really quite simple: Require that participants come to the program with a specific business challenge (either individually or as a team); build time into the program to create a plan for addressing that challenge based on the content that is presented; and then insist that managers execute against these plans after the program
    http://gridinternational.com/fm_files/HBR_Articles/HBR_Breakthrough_in_Organization_Development_-_Harvard_Business_Review1.pdf
  • Appsec2013 presentation-dickson final-with_all_final_edits

    1. 1. Can AppSec Training Really Make a Smart Developer? Research From Denim Group John B. Dickson, CISSP @johnbdickson November 20, 2013
    2. 2. About Me • Application Security Enthusiast • Security Professional • ISSA Distinguished Fellow • MBA-type and Serial Entrepreneur • Dad Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    3. 3. About Denim Group Denim Group Overview •Professional services firm that builds & secures enterprise applications  External application assessments  Web, mobile, and cloud  Software development lifecycle development (SDLC) consulting •Secure development services:  Secure .NET and Java application development  Post-assessment remediation •Classroom and e-Learning for PCI compliance Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    4. 4. “I personally believe that training users in security is generally a waste of time, and that the money can be spent better elsewhere.” Bruce Schneier Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    5. 5. How Developer Training is Different •Both trying to change behaviors – Target audience has more power to say “no” – Deadlines and releases drive training •For developer, infrequent, but more disruptive • 15-45 minutes vs. 2-day class Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    6. 6. Yet Training is Mandated •PCI DSS 3.0  Train developers in secure coding techniques, including how to avoid common coding vulnerabilities, and understanding how sensitive data is handled in memory.  Testing Procedures: 6.5.a: Examine software development policies and procedures to verify that secure coding technique training is required for developers, based on best practices and guidance.  Testing Procedures: 6.5.b: Interview a sample of developers to verify that they are knowledgeable in secure coding techniques  Testing Procedures: 6.5.c : Examine training records to verify that software developers received training on secure coding techniques, including how to avoid common coding vulnerabilities, and understanding how sensitive data is handled in memory. Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    7. 7. But Results Are Not Measured •Harvard Business Review – Large-scale organization development is rare – Measurement of results is even rarer •Workforce analytics rare – More than 25% of survey respondents use little or no workforce analytics – The vast majority (>61%) report their use as tactical, ad hoc, and disconnected from other key systems and processes Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    8. 8. Growth & Turnover Spur Sense of Urgency – Software development field growing 30% – Turnover • Industry – 14-15% • General IT – ~20% • Software Development – ~20 – 30% Sources: Bureau for Labor Statistics and Society of Human Resources Management Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    9. 9. Research Overview •Focus: Assess the software developers depth of software security knowledge •Purpose: To measure the impact of software security training on that level of understanding •Survey size: 600 software developers surveyed in North America (US and Canada) •Vertical markets represented: financial, government, retail, educational, technology, energy and healthcare segments. Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    10. 10. Respondent Demographics Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    11. 11. Respondent Demographics Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    12. 12. Respondent Demographics Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    13. 13. Methodology  15 Multiple Choice Quiz-Style Questions  Targeted at Software Developers  Varied by years of experience, amounts of previous training, primary job function, company industry and company size  Distribution:  Online and hard-copy questionnaires given to instructor-led class trainees (before and after)  Social media networks (sharing and some paid promotion with incentives) Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    14. 14. Hypotheses 1. Most software developers do not have a basic understanding of software security concepts. 2. Software security training can improve a developer’s knowledge of security concepts in the short-term. 3. Certain industries, such as financial services, are more likely to have software developers that are already exposed to key software security concepts. Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    15. 15. Sample Questions 4. If an attacker were able to view sensitive customer records they should not have had access to, this would be a(n)_______breach. ___ Confidentiality ___ Integrity ___ Availability 7. Authentication is... ___ Proving to an application that the user is who they claim to be ___ Confirming that the user is allowed to access a certain page or function ___Verifying that the data displayed on a given page is authentic ___ Thoroughly logging all of a user's important activity Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    16. 16. Key Survey Results • Enterprises of more than 10,000 personnel had the lowest secure coding knowledge Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    17. 17. Key Survey Results • Architects and software developers had a much higher level of knowledge than QA, yet in many organizations QA has a material role in application security Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    18. 18. Key Survey Results •Slightly more than half of the respondents correctly answered basic awareness questions on application but struggled with ways to operationalize appsec concepts Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    19. 19. Key Survey Results •Almost 100 percent could define input validation, demonstrating a choppy understanding of advanced secure coding knowledge. •On the other hand, almost 90 percent correctly identified proper session IDs which is reassuring. Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    20. 20. Key Survey Results •The majority of the respondents had no prior secure coding training, which might be surprising Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    21. 21. Key Survey Results • There was no correlation between years of experience and knowledge of secure coding highlighting the continued need for effective security training Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    22. 22. Key Survey Results •The respondents that had had more than 3 days of app sec training in the past were able to answer more than half of the questions correctly. Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    23. 23. Key Survey Results •100% correctly identified where cross site scripting executes after completing training, an increase of almost 20 percentage points. Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    24. 24. Key Survey Results •The number of respondents able to correctly identify what is application security more than doubled after training was complete. Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    25. 25. CONCLUSION • Retention rose by more than 25 percent after completing secure coding training. • Other statistics also reported that application vulnerabilities reduced by over 70 percent after training Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    26. 26. Software Developers Learn Differently than Companies “Teach” • Teaching methods are formalized and structured in order to be repeatable • Type of structures consist of: – On-site & off-site classroom training – E-learning for compliance – Videos, webinars, etc. Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    27. 27. So How Do Developers Learn? •Informally and and in an unstructured way via: • • • • • Blogs & RSS feeds Social media with emphasis Developer websites Influential e-mail lists Safarionline Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    28. 28. Don’t Ignore Basics of Training •Refresher training is still needed •Training must be included in performance plans •Managers increasingly want an ROI Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    29. 29. Incentives Matter! Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    30. 30. CONCLUSION • Software developers largely do not understand key software security concepts  73% of respondents “failed” the initial survey  Average score of 59% before training • However, software developers’ understanding of key software security concepts did increase after training Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    31. 31. Where do we Go from Here? Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter
    32. 32. Questions and Answers? John B. Dickson @johnbdickson john@denimgroup.com Hosted by OWASP && the NYC Chapter Hosted by OWASP the NYC Chapter

    ×