This document discusses the challenges of security testing and how development-driven security testing can help address them. It outlines three main reasons security testing is difficult: large scope, difficulty hooking tests into code, and the halting problem. It then demonstrates how tools like Freud and PowerMock can help security testers iterate over code, write tests for unsafe calls and missing implementations, find code calling dangerous APIs to fuzz, and test for forbidden parameters - addressing problems that regular testing can miss due to unintended behaviors. By working with developers using techniques like these, security testing can be more comprehensive.

1. The power of development-driven
security testing
Marat Vyshegorodtsev
System Security Office
Rakuten, Inc.
https://global.rakuten.com

2. About Rakuten

3. World coverage
E-commerce in 14 countries and regions
All services and businesses in 27 countries
2011200920082005 201320122010
INVESTMENT
2014

4. Bio & Disclaimer
Technical Program Manager “Group Core Services” at Rakuten
University of Tokyo graduate
Member of the world-famous CTF team “More Smoked Leet Chicken”
The Russian hacker of Japan :-)
!
I’m not a Java developer. In fact, I’m not a developer at all.

5. Security & Quality Assurance
• Regular QA tests cover “intended” functionality
• Security QA tests try to find all other unintended behavior

6. Security & Quality Assurance
• Regular QA tests cover “intended” functionality
• Security QA tests try to find all other unintended behavior

7. QA is hard, Security QA is harder
Main reasons why security tests are hard:
1. Big scope: number of methods times number of tests
2. Hard to hook in
3. Halting problem

8. QA is hard, Security QA is harder
Main reasons why security tests are hard:
1. Big scope: number of methods times number of tests
2. Hard to hook in
3. Halting problem

9. QA is hard, Security QA is harder
Main reasons why security tests are hard:
1. Big scope: number of methods times number of tests
2. Hard to hook in
3. Halting problem

10. QA is hard, Security QA is harder
Main reasons why security tests are hard:
1. Big scope: number of methods times number of tests
2. Hard to hook in
3. Halting problem

11. Halting problem in one slide
It is impossible to determine
if program will halt or not on given inputs
Hence, it is impossible to perform all
possible security tests
Give it up.
xkcd.com/1266

12. Problem 1: Traversing the code
Given:
A service that has 70,000+ lines of code, a build system, and some tests
Find:
• All classes and their methods that use unsafe or deprecated calls
• Classes that must implement certain methods, but didn’t
• Classes that call certain dangerous APIs to fuzz them later

13. Freud — a framework for writing static analysis tests
LMAX-Exchange/freud

14. Freud
• Enables iteration over source code and byte code files’ contents
• Supports custom hamcrest matchers to write rules
• Implements DSL-like syntax for writing tests with JUnit or Groovy

15. Unsafe and deprecated calls
Ban all direct input/output trough files:
@Test!
public void noDirectFileInput() throws Exception {!
!
Freud.iterateOver(ImportDeclaration.class).!
!!assertThat(no(importDeclarationPathAsString(),
containsString("java.io.File"))).analyse(listener);!
!
}

16. Mandatory implementation
@RolesAllowed("Administrator")
public void setNewRate(int rate) {
...
}

17. Mandatory implementation
Freud.iterateOver(CodeBlock.class).
forEach(method(publicMethod())).
assertThat(hasDeclaredAnnotation(“RolesAllowed”))
!
.in(codeBlocksWithin(methodDeclarationsWithin(classDeclar
ationsWithin(javaSourceOf(asList(
//
list
of
class
files
URLs
here
)))))).analyse(listener);

18. Finding bad apples
Freud.iterateOver(CodeBlock.class).
forEach(method(hasMethodCall(“Session.createSQLQuery”)))
!
//
find
out
who
calls
unsafe
APIs
and
try
to
fuzz
it

19. Fuzzing
// this.function is a iterator for forEach!
!
public T next(){!
for(Fuzzer f = fuzzDB.createFuzzer("031-B16-HEX", 4);
f.hasNext();) {!
return function(f.next());!
}!
}

20. Problem 2. Going deep

21. Problem 2. Going deep

22. Problem 2: Going deep
Given
• Big application with full code coverage
• No security checks implemented
Find
• Certain method is called with a certain parameter
• Some parameters should never be passed to a method

23. Power of mocking with PowerMock
PowerMock is a custom class-loader
and byte-code manipulator allowing to
mock static methods
Extends Mockito and JUnit perfectly

24. Deprecating MD5
Problem: MessageDigest.getInstance(“MD5”) must not be used
Solution: Let’s just grep for a string getInstance(“MD5”)!
!
But… remember the halting problem?
MessageDigest.getInstance(Config.getConfiguredHashAlgorithm())
↑ is it MD5?

25. @RunWith(PowerMockRunner.class)
@PrepareForTest({MyClass.class,
MessageDigest.class})
public
class
md5Test
extends
PowerMockTestCase
{
!
@Test
public
void
testDoHash()
throws
Exception
{
PowerMockito.mockStatic(MessageDigest.class);
when(MessageDigest.getInstance("MD5")).thenReturn(null);
PowerMockito.verifyStatic();
!
assertEquals(“acbd18db4cc2f85cedef654fccc4a4d8",
MyClass.doHash("foo"));
}
!
}

26. PowerMock + Hamcrest
• Problem: See if Log function never accepts credit card number-looking
strings, given that a developer wrote a test that triggers this behavior
• Solution: Mock Log class, when its functions are called, there is no
argThat is a 16 digit string (as an easy example)

27. @RunWith(PowerMockRunner.class)
@PrepareForTest({MyClass.class,
Log.class})
public
class
logTest
extends
PowerMockTestCase
{
!
@Test
public
void
testSomeFunction()
throws
Exception
{
PowerMockito.mockStatic(Log.class);
when(Log.i(new
IsCreditCard())).thenReturn(null);
PowerMockito.verifyStatic();
!
//
continue
test
}
!
}

28. class IsCreditCard extends ArgumentMatcher<String> {!
public boolean matches(String message) {!
return RegExp.matches(message,”[0-9]{16}”);!
}!
}

29. Summary
In most of the languages running on VMs it is possible to test certain
weaknesses using unit tests
Security engineers working together with TDD/BDD dev teams can write
many business logic-aware tests easily using the frameworks I have
described
BDD is for intended behavior, security-driven development is for
unintended one

30. Thank you!
marat.vyshegorodtsev@mail.rakuten.com
!
@MaratVy
maratto.blogspot.com