Talk given by Max Feldman, Product Security Engineer at Salesforce, at AppSec USA. One challenging aspect of achieving software security is the struggle to catch up with the speed of development and deployment. We built Providence with the goal of preventing obvious bugs from ever being deployed into production. Providence is a lightweight and scalable tool which finds bugs and anti-patterns of varying complexity from code commits, and we’ve used it to prevent vulnerabilities ranging from XSS, to access control issues, to XXE. It works by continuously monitoring and pulling commits from version control systems and scanning them for bugs with rules defined in plugins. Additional plugins are easy to create and deploy, which has allowed for quick reaction to new bugs or problems as they are discovered. Providence is easily integrated with SDLC workflows or bug-tracking tools, and we will discuss how we have integrated it in-house in an unobtrusive manner. This model of addressing issues also provides relative immediacy of resolution; on average, potential problems found by Providence are resolved more quickly than other vulnerabilities because developers are presented the issues right after they commit the code, instead of weeks to months later. We are currently in the process of open-sourcing Providence in order to share the tool with the DevOps/security community (or any interested parties). This talk will cover the internals of Providence, its engine and plugin architecture (including examples of plugins and their ease of creation), as well as its integration with our SDLC and the faster and more efficient responses we’ve achieved as a result. We’re continuing to build new plugins and features, and we’re excited see what ideas others may have in mind!

1. Providence: rapid vulnerability prevention
Max Feldman, Xiaoran Wang, Hormzard Billimoria

2. Agenda
• Background
• Intro to Providence
• Plugins, Plugins Demos
• Our Deployment and Metrics
• Future Work
• Conclusion

3. Team Intro
• Max Feldman
• Xiaoran Wang
www.attacker-domain.com
• Hormazd Billimoria
@hormazdb

4. Current state of SCA/RTA
• Static code analysis
– Slow, scans can’t catch up with daily deployment
– thorough, but resource intensive
– A lot of false positives
• Run-time analysis
– fast, but requires continuous building and running of application
– may not hit all code paths
– may require special instrumentation

5. What we needed
• Something faster, more scalable
– SCA - could take up to a few weeks for a complete scan
– but many vulnerabilities are easier to find (exec, hardcoded passwords…)
– in that time it’s also easy for devs to lose context

6. What we needed
• Something easier to update
– SCA can have complicated rules, higher learning curve
– Ideal - rules which are easy to write, even for new users

7. What we needed
• Something with low false negatives
– if something is low-hanging fruit or easy to catch, we didn’t want to miss it
– acceptable to have a bit more noise as long as we aren’t missing anything
• Providence- check-in time analysis

8. Providence overview and architecture

9. How Providence works
• Polls one or more version control systems, runs plugins against the commits
– for us- p4 (and a bit of git), variety of plugins
• Plugin finds something suspicious
– looks up committer
– files an investigation to the security person in charge of their area
– more functionality possible
• Plugins can look at the diffs, full code, whatever is necessary

10. Providence architecture

11. Providence architecture (cont.d)
• scheduler
– schedules things to run every x minutes
– necessary when API doesn’t provide push
• eg p4
• watcher engine
– watchers correspond to a particular VCS
– they have the code to interact with that api

12. Providence architecture (cont.d)
• rule engine
– if a change meets the criteria of a plugin,
the plugin’s rules are run
– could be a regex, with arbitrary logic
– alerts and logging (or investigation
creation) can be created as well
• plugins
– logics done on the diff (or the file itself)
– the magic behind providence
– we will dive into it more soon
• alerts
– email
– custom alerts
– custom integration with workflows

13. VCS integrations
• we currently support
– P4
– github (internal or external)
• Extensions depend on ease of API
• other things we would like to support
– jira
– hipchat
– your platform of choice?

14. Benefits
• doesn’t miss anything that comes in
– rules are meant to be simpler, eg. searching for a certain line
• immediacy of results
– dev doesn’t have to context switch
• rapid deployment of new rules
– eg. new vuln comes out- add a new rule, from then on out it will be prevented
• scalable
• failure of one rule doesn’t affect others
• lightweight

15. Providence plugins

16. Plugins
• Runtime Exec
• Direct usage of Crypto functions
• IP whitelisting bypass
• XXE

17. Plugins
• Secret stored in the source code
• Servlet created without access check
• Usage of privileged modes in the source code
• Direct access to sensitive services like HSM
• XSS in HTML pages

18. PLUGINS DEMO
• XXE in Java
• LocalStorage in JavaScript

19. Plugin Demos (backup) - Java Rules

20. Plugin Demos (backup) - Java Source

21. Plugin Demos (backup) - Email Alert

22. Plugin Demos (backup) - JS Rules

23. Plugin Demos (backup) - JS File

24. Plugin Demos (backup) - Email Alert

25. Providence in SDLC, metrics

26. Deployment to SDLC
• Integrate with bug tracking
• Integrate with environment for
stats/metrics

27. Metrics (March - mid august)
• 48,000 commits
• 250 alerts
• 40 bugs prevented

28. Future work

29. Future work
• run rules against all commits (go back to beginning of history)
• more VCS
• more plugins (alerts to new platforms)
• more integrations with bug trackers, alert systems, etc.
• IDE enhancements and process integrations. (pre, post commit)

30. Future stats and metrics
• More advanced metrics
– severity predictions/estimates
• ML algorithms to evaluate likelihood of severity
• More data- learn as we go, improve the accuracy and efficacy of Providence in our own
SDLC and its effectiveness as a tool

31. Open Source
• https://github.com/SalesforceEng/Providence
• Welcome contributors/suggestions/sharing of knowledge and best practices

32. Conclusion
• Providence is a new tool that bridges security automation and rapid, continuous
deployment
• How an organization can benefit from this tool
• How Providence can be integrated into an organization’s SDLC
• How to create and deploy plugins and rules for particular needs

33. thank y u