This document provides an outline for a lecture on software security. It introduces the lecturer, Roman Oliynykov, and covers various topics related to software vulnerabilities like buffer overflows, heap overflows, integer overflows, and format string vulnerabilities. It provides examples of vulnerable code and exploits, and recommendations for writing more secure code to avoid these vulnerabilities.

1. Software Security December 2014
Roman Oliynykov
Professor at
Information Technologies Security Department
Kharkov National University of Radioelectronics
Head of Scientific Research Department
JSC “Institute of Information Technologies”
Ukraine
Visiting professor at
Samsung Advanced Technology Training Institute
Korea
ROliynykov@gmail.com

2. Lectures outline
 A few words about myself
 Importance of secure solutions
 Buffer overflow vulnerability and
countermeasures
 Heap overruns and integer overflows,
recommendations for avoiding
 Format string vulnerabilities and security
recommendations

3. For these lectures
I suppose you are familiar
 for general understanding of discussing
problems:
 C and C++ programming languages (basic level)
 it will be discussed, but would be an advantage if
you already familiar with
 operation system architecture (process, address
space, stack, heap, etc.)
 x86 assembler language source code (preferably
AT&T notation)
 basics of Linux (command line) and computer
networking

4. About myself (I)
 I’m from Ukraine (Eastern part of
Europe),
host country of Euro2012 football
championship
 I live in Kharkov (the second biggest
city in the country, population is 1.5
million people), Eastern Ukraine
(near Russia),
former capital of the Soviet Ukraine
(1918-1934)
three Nobel prize winners worked at
Kharkov University

5. About myself (II)
 Professor of Information Technologies
Security Department at Kharkov
National University of Radioelectronics
 courses on computer networks and
operation system security, special
mathematics for cryptographic applications
 Head of Scientific Research
Department at JSC “Institute of
Information Technologies”
 Scientific interests: symmetric
cryptographic primitives synthesis and
cryptanalysis

6. A few words about importance
of secure solutions

7. Modern malware
 functions of different categories: viruses,
worms and Trojan horses
 belongs to network of bot programs (botnet)
 executes commands received from the
botnet control center
 has stealth/polymorphysm/armoring
functions against antivirus software

8. Consequences of cyberattacks
for the society
[Norton/Symantec cybercrime report, 2013]

9. Consequences of cyberattacks
for the society
[Norton/Symantec cybercrime report, 2013]

10. Consequences of cyberattacks
for the society
[Norton/Symantec cybercrime report, 2013]

11. Buffer overflow vulnerability
overview

12. Address space layout in
different operation systems

13. Address space layout in different
32-bit operation systems

14. Process switching example

15. Linux process address space
layout (32-bit)

16. Example of code without
boundary check

17. How it looks like after a
compilation

18. What happens if strcpy() argument is
longer than the destination buffer
strcpy( &dst, &src ) in contrast to
strncpy( &dst, &src, sizeof (dst) )
takes into account only
destination string length (buffer
size) and copies data until finds
termination zero in src

19. Example of vulnerable network
daemon (service) for Linux,
and exploit for it

20. netcalcd – vulnerable daemon
(service) for Linux (x86)
 intentionally written for this lecture and
contains intentionally man-made
vulnerabilities
 processes simple network text requests for
basic calculations
 prints debug information about its stack on
the server console

21. netcalcd normal operation

22. netcalcd normal operation

23. netcacld source code:
part of the main() function

24. netcacld source code:
process_request() function

25. netcacld source code:
get_result() function

26. netcacld source code in asm:
get_result() function

27. Vulnerability in
get_result() function
strcpy( &dst, &src ) in contrast to
strncpy( &dst, &src, sizeof (dst) )
takes into account only
destination string length (buffer
size) and copies data until finds
termination zero in src

28. netcalcd stack after strcpy() call with
malicious data (hacker’s code) from the
network

29. netcalcd normal operation

30. Running exploit against
netcalcd

31. netcalcd buffer overflow in
get_result()

32. Open ports on the victim
computer: before and after

33. Victim computer successfully
cracked

34. What’s inside exploit and how
it works?

35. Exploit: usual C program for Windows
sending block of data (shellcode):

36. Shellcode in the example: relocatable
binary code can be run at any user address
Protect the running code in the stack, find absolute address it is
run at and decode the rest part of the shellcode

37. Why encode the main part of
the shellcode?

38. After encoding the rest part of the
shellcode runs web server at port 8801
or does everything
intruder wants to do with
the vulnerable process
privileges
NB: cross-platform exploits working well different processors and OS

39. How to protect our software
against such an attack?

40. Possible countermeasures
against buffer overflow
 write secure code based on secure functions calls
and all necessary user input verification (the most
important recommendation)
 make your operation system to use Address Space
Layout Randomization (ASLR)
 make your operation system use processor NX bit
(on x86 platform)
 keep on canary words in your compiler
 run the code with the least necessary privileges

41. Write secure code based on
secure functions calls
strcpy( &dst, &src ) fills destination buffer without taking into account its size;
strncpy( &dst, &src, sizeof( dst ) ) won’t write outside the destination buffer (but
it’s possible the lost of terminating zero)

42. Write secure code based on
secure functions calls
And many other recommendations for writing secure code…

43. Security check of existing
projects: automated tools
But no guarantee that all vulnerabilities are discovered

44. Address Space Layout
Randomization
computer security method which involves
randomly arranging the positions of key data
areas, usually including the base of the
executable and position of libraries, heap,
and stack, in a process's address space
Each running time stack, heap, etc. are put at
random addresses in the process address space

45. Address Space Layout
Randomization (example)
It’s difficult to guess correct return address to be written on the stack
smashing. But it is possible: only16 less bits of address are changed
Running code addresses are NOT changed

46. ASLR appeared:
 Linux kernel support: 2.6.12 (released June
2005)
 Microsoft's Windows Vista (released January
2007), Windows Server 2008, Windows 7,
and later have ASLR enabled by default
 Android 4.0 Ice Cream Sandwich provides
ASLR
 …

47. ASLR evasion techniques
 brute force address search attempt
 return into code on non-randomized memory
 jmp *esp (ret address points to such bytes in code)
 etc.

48. x86 calling conventions
 cdecl (C declaration)
 arguments are pushed to the stack in the reverse
order (the last one has the higher address)
 calling function cleans stack (allows arbitrary
number of arguments after program compilation)
 stdcall (standard call)
 arguments are pushed to the stack in the reverse
order (like in cdecl)
 called function cleans stack

49. x86 calling conventions (cont.)
 pascal
 arguments are pushed to the stack in the direct order (the
last one has the lower address)
 called function cleans stack
 fastcall
 first two arguments are passed via registers, the rest (if
any) are pushed to the stack in the reverse order (like in
cdecl)
 called function cleans stack
 thiscall (for OOP non-static member functions)
 this pointer (to object *this) is an additional parameter
 for gcc: calling function cleans stack; for MS Visual C++: it
depends on fixed (called) or variable (calling) number of
arguments

50. x86_64 calling conventions
 the single calling conversion:
 first 4 arguments are passed via registers rcx, rdx,
r8, r9, the rest (if any) are pushed to the stack in
the reverse order (like in cdecl)
 calling function cleans stack (like in cdecl)

51. Make your operation system use
processor NX bit (on x86 platform)
NX bit, which stands for Never eXecute, is a technology used in
CPUs to segregate areas of memory for use by either storage
of processor instructions (or code) or for storage of data

52. NX bit protection evasion:
return-to-libc attack
 no code in the stack (no
processor exception)
 return address is
overwritten and points to
the existing code
 intruder calls standard
function and passes
arbitrary arguments to it
 in Windows it is possible
to call a sequence of
functions due to _stdcall_
convention

53. Never switch off canary words
in your compiler
Canary words are known values that are placed between a buffer and
control data on the stack to monitor buffer overflows

54. Canary words
 Implementation:
 GCC Stack-Smashing Protector (ProPolice)
 Microsoft Visual Studio 2003 and higher ( /GS )
 etc.
 What cannot be handled:
 buffer overflows in the heap
(intruder uses pointers to functions in virtual
method tables of dynamic objects)

55. Beyond stack overflow: heap
overruns
heap memory allocation:
 dynamic allocation (no fixed addresses)
 considered as data only (no execution)

56. Object oriented programming
 encapsulation
 inheritance
 polymorphism

57. Polymorphism implementation:
virtual methods table (VMT)

58. Heap overflow: VMT is
overwritten

59. C++ code example

60. C++ code examples (cont.)

61. Heap overflow influence

62. Threats for heap overflow
beyond OOP
 function pointers
 indexes for arrays (will be discussed with
integer overflows)
 other data structures located at heap may
have influence to program execution
cf: Heap Spraying methods used by intruders

63. Heap overflow
 NX bit does NOT protect
 no canary words
 ASLR evasion techniques work well

64. Integer overflows

65. Integer overflows (example for 64-bit
application)

66. Possible string length
size_t strlen( const char *s );
size_t: long unsigned int (64 bit for 64-bit platform)

67. Integer conversion rules
1) signed (n bits) to unsigned of the same size
 positive and zero: the same value
 negative: value + 2n
2) signed to signed of the bigger size
 positive and zero: the same value
 negative: most significant bit extends
3) signed to unsigned of the bigger size
 signed converts to signed of the bigger size, and then
converts to unsigned
 positive and zero: the same value
 negative: see item 2) then item 1)

68. Integer conversion rules (cont.)
4) unsigned to unsigned of the bigger size
 the same value (extended by zero bits)
5) signed to signed of the bigger size
 the same value
6) unsigned (n bits) to signed of the same size
 0.. 2n-1
: the same value
 bigger than 2n-1
: negative value
7) unsigned to signed of the bigger size
 unsigned converts to signed of the same size, see 6) and
then converts to signed of the necessary size

69. Integer conversion rules (cont.)
 signed or unsigned to integer with smaller
size:
 rather complex behavior depending on system
architecture (little endian, big endian)
 should be avoided unless predictable special
function appropriate for program logic is
implemented.

70. Rules for integer types conversion for
different operations
 if operands size are
less than 32 bit (char,
short), they
conversed to int (32
bit)
 bitwise logical
operators follow this
rule, e.g.: result of
(unsigned short) |
(unsigned short)
belongs to int

71. Rules for integer types conversion for
different operations (cont.)
 unary ~ (compliment) also
extends result type
 ++ and – do not change result
type
 % should be carefully analyzed
during index check:

72. Rules for integer types conversion for
different operations (cont.)
 / is also should be carefully
analyzed

73. Operations with pointer comparison: different
behavior on 32- and 64-bit platforms

74. Recommendations for
avoiding integer overflows
 keep in mind, that there is NO universal solution,
speed and effectiveness of C/C++ should be paid by
developer’s attention
 don’t compare signed and unsigned types
 carefully check array indexes, pointer operations
 use unsigned variables for array indexes and sizes
for memory allocation
 always pay attention to compiler’s warnings
 during software testing always check boundary
values (0, max, -1, -2, max-1, max+1, etc.) given to
input of your code

75. Format string vulnerabilities

76. Format string vulnerabilities
 untrusted input
(user, network,
etc.) is sent directly
to *prinf() function
from the standard
library

77. Vulnerable functions
and all functions directly passing arguments to these (error(), syslog(), etc.)

78. Examples of printf() application

79. Principles of vulnerability
exploitation (32-bit systems)
printf() allows variable number of arguments passed through the stack

80. Results of exploiting
vulnerability
 program/service/daemon crash (DoS attack)
 viewing local function variables
 viewing any block of memory available to the
process (thread)
 write access to memory available to the
process (thread)

81. Program crash
 “%s%s%s%s%s%s%s%s%s%s%s%s” is taken
from untrusted input
 printf() expects to find 12 pointers to the zero-ended
string at stack among it’s parameters
 if there is at least one “pointer” found on stack is
invalid, program crashes

82. Viewing local function
variables
 “%08x. %08x. %08x. %08x. %08xn” is taken from untrusted
input
 printf() expects to find 5 integer variables at stack as it’s
parameters
 values are taken from the stack (variables of the calling
function, its returning address, etc.) and printed

83. Viewing any block of memory
 “x10x01x48x08_%08x.%08x.%08x.%08x.%08x|%s|” is
taken from untrusted input
 it’s very likely that buffer for untrusted input is also located in
the stack
 printf() is made to take necessary number of bytes from the
stack by %08x parameters given by hacker
 address of interesting memory is given in little-endian format
by x10x01x48x08 and printed out with %s parameter

84. Write-access and code execution (I)
 it seems (but seems only) that attack like buffer overflow
is protected by %400 string limit
 "%497dx3cxd3xffxbf<nops><shellcode>“is taken from
untrusted input

85. Write-access and code execution (I)
 buffer for untrusted input is located in the stack and can be used for
smashing returning address
 x3cxd3xffxbf address points to nops or to the beginning of
shellcode
 example works in case of canary absence and disabled NX bit (but
there are evasion methods)

86. Write-access and code execution(I): is it
enough to use such solution for
protection?
 snprintf() is used instead of insecure sprintf()
 end of string array is marked by 0 in any case
 secure functions are used for further processing of this string
 is it enough for securing a program?

87. printf() and other similar
functions can write variables:
 %n is used to get the number of already written bytes to
the variable which address is given as a parameter to
printf()
 number written by printf() can be adjusted with %100n,
%150n, etc. format string value

88. Write-access to process memory (II):
tasks for exploiters
 find format-string vulnerability
 find a block of code putting parameter to the
stack
 write a basic sequence for format string
 find an offset in stack for taking parameters
and addresses
 in case of code execution (not only memory
read/write): ASLR, canary and NX bit evasion
[beyond this attack]

89. Write-access to process
memory (II)
 format string contains:
 address of memory to be modified (0xbfffc8c0)
 %08x parameters for setting enough stack offset
 %n parameter to make printf() writing to process
memory

90. Write-access to process
memory (II)
 additional features:
 %hn may be used to write 2 bytes instead of 4
 global offset table (GOT) in Linux process or
Windows stubs may be modified instead of stack
returning address (preserving canaries and
evading NX bit), including “return-to-libc method”
 addresses of virtual functions, destructors, etc. in
heap can be also overwritten

91. NB: Sequential calls with untrusted
input are also vulnerable!
 any nested sequence of snprintf() can be
used
 for attacker it’s enough to find one call with
snprintf( outbuf, buf ) without %s specifier
 all similar functions are also affected

92. Recommendations for
functions taking format string:
 use snprintf( outbuf, “%s”, buf ) in all cases; remember
that snprintf( outbuf, buf ) vulnerable in any nested call if
some part of buf is taken from untrusted input
 use only secure functions (snprintf() instead of sprintf(),
etc., as pointed out in man pages and msdn)
 mark the end of string array by 0 end in any case
 keep in mind, that there is NO universal solution, speed
and effectiveness of C/C++ should be paid by
developer’s attention
 don’t believe that this is only C/C++ problem: Java virtual
machines (JVM), Perl, etc. are written in C/C++ and may
cause this vulnerability even to script languages

93. Recommended books:
further reading
 24 Deadly Sins of Software Security: Programming
Flaws and How to Fix Them (Michael Howard, David
LeBlanc, John Viega)
 Writing Secure Code (Michael Howard)
 Software Security: Building Security In (Gary McGraw)
 The Security Development Lifecycle (Michael Howard,
Steve Lipner)
 Secure Coding: Principles and Practices (Mark G. Graff,
Kenneth R. van Wyk)
 Scott Meyers. Effective C++: 55 Specific Ways to
Improve Your Programs and Designs (3rd Edition)