0
An Industrial Case Study of Bypass Testing on Web Applications Joint research Dr.  Ye Wu ,  Xiaochen Du ,  Hong Huang ,  V...
Outline of Talk <ul><li>Motivation </li></ul><ul><li>Applying bypass testing – early results </li></ul><ul><li>Automating ...
Web Application Input Validation Sensitive Data Client Server <ul><li>Bad Data </li></ul><ul><li>Corrupts data base </li><...
Deploying Software <ul><li>Bundled  : Pre-installed on computer </li></ul><ul><li>Shrink-wrap  : Bought and installed by e...
Problem Parameters <ul><li>HTTP is a  stateless protocol </li></ul><ul><ul><li>Each request is  independent  of previous r...
Bypass Testing <ul><li>“ bypass ” client-side  constraint enforcement </li></ul><ul><li>Bypass testing constructs tests to...
©  Jeff Offutt User Name: Small $150 Version to purchase: Age: Large $500 Medium $250 Simple Example Web Page ICST 2008
©  Jeff Offutt User Name: Small $150 Version to purchase: Age: Large $500 Medium $250 Proper Behavior ICST 2008 Username s...
Abbreviated HTML ©  Jeff Offutt <FORM > <INPUT Type=“text” Name=“username” Size=20> <INPUT Type=“text” Name=“age” Size=3 M...
Bypass Behavior <ul><li>Extremely loose  coupling  … </li></ul><ul><li>combined with the  stateless  protocol … </li></ul>...
Saved & Modified HTML ©  Jeff Offutt <FORM > <INPUT Type=“text” Name=“username” Size=20> <INPUT Type=“text” Name=“age” Siz...
SQL Injection ©  Jeff Offutt, 2004 User Name: Password: turing enigma Original SQL: SELECT username FROM adminuser WHERE u...
Applying Bypass Testing <ul><li>Analyze  HTML to extract each form element </li></ul><ul><li>Model  constraints imposed by...
Example Client-Side Constraint Rules <ul><li>Violate  size restrictions   on strings </li></ul><ul><li>Introduce values  n...
Example Server-Side Constraint Rules <ul><li>Data  type  conversion </li></ul><ul><li>Data  format  validation </li></ul><...
Example Security Violation Rules ©  Jeff Offutt, 2004 Potential Illegal Character Symbol Empty String Commas , Single and ...
First Example CyberChair <ul><li>CyberChair  : Web-based  conference  management </li></ul><ul><ul><li>www.cyberchair.org ...
Automating Bypass Testing <ul><li>Autobypass  : A web application that accepts a URL and generates  input data  for the HT...
Types of Client Input Validation <ul><li>Client side  input validation  is performed by HTML form controls, their attribut...
Example Interface: yahoo registration form ICST 2008 ©  Jeff Offutt Limited  Length  (HTML) Preset  Values  (HTML) Preset ...
Test Value Selection <ul><li>Challenge :  </li></ul><ul><ul><li>How to  automatically  provide effective test values? </li...
AutoBypass <ul><li>AutoBypass steps (the big picture) </li></ul>ICST 2008 ©  Jeff Offutt Parse  Interface Set Default  Val...
Real-World  Examples ICST 2008 ©  Jeff Offutt atutor.ca Atalker demo.joomla.or Poll, Users phpMyAdmin   Main page,  Set Th...
Classifying Output Responses <ul><li>( V )  Valid Responses  :  invalid inputs are adequately processed by the server </li...
Results ICST 2008 ©  Jeff Offutt v
Research to Practice ICST 2008 ©  Jeff Offutt “ Knowing is not enough, we must apply.  Willing is not enough, we must do.”...
Industrial Case Study <ul><li>Inventions  from scientists are slow to move into industrial practice </li></ul><ul><li>We w...
Avaya’s NPP Technology <ul><li>Avaya Labs Research creates  research prototypes  of software systems, then turns successfu...
NPP Design and Implementation <ul><li>NPP is a  highly user interactive  web application </li></ul><ul><li>Uses many  scre...
NPP Bypass Tests <ul><li>Bypass testing analyzes HTML ( statically ) and generates inputs that violate input constraints <...
NPP Testing Results <ul><li>Six  NPP screens were tested </li></ul><ul><li>Tests are  invalid  inputs – exceptions are exp...
Types of Faults <ul><li>Invalid data  saved  into the database </li></ul><ul><ul><li>These lead to more visible failures  ...
Conclusions <ul><li>Bypass testing  worked very well  in an industrial context </li></ul><ul><ul><li>There is  no technica...
Future Work <ul><li>A major  observability  problem with web application testing is detecting invalid database values </li...
Upcoming SlideShare
Loading in...5
×

Web Bypass Testing

1,083

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,083
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Web Bypass Testing"

  1. 1. An Industrial Case Study of Bypass Testing on Web Applications Joint research Dr. Ye Wu , Xiaochen Du , Hong Huang , Vasileios Papadimitriou , Qingxiang Wang and Joann J. Ordille of Avaya Labs Research Based on papers in ISSRE 2004 and ICST 2008 Jeff Offutt Software Engineering George Mason University Fairfax, VA USA www.cs.gmu.edu/~offutt/ [email_address] Expanded version of a talk given at the first International Conference on Software Testing, Verification and Validation
  2. 2. Outline of Talk <ul><li>Motivation </li></ul><ul><li>Applying bypass testing – early results </li></ul><ul><li>Automating bypass testing </li></ul><ul><li>Real-world examples </li></ul><ul><li>Industrial case study </li></ul><ul><li>Conclusions and future work </li></ul>ICST 2008 © Jeff Offutt
  3. 3. Web Application Input Validation Sensitive Data Client Server <ul><li>Bad Data </li></ul><ul><li>Corrupts data base </li></ul><ul><li>Crashes server </li></ul><ul><li>Security violations </li></ul>Check data Check data Malicious Data Can “bypass” data checking
  4. 4. Deploying Software <ul><li>Bundled : Pre-installed on computer </li></ul><ul><li>Shrink-wrap : Bought and installed by end-users </li></ul><ul><li>Contract : Purchaser pays developer to develop and install, usually for a fixed price </li></ul><ul><li>Embedded : Installed on a hardware device, usually with no direct communication with user </li></ul>© Jeff Offutt, 2004 <ul><li>Web : Executed across the Internet through HTTP </li></ul>
  5. 5. Problem Parameters <ul><li>HTTP is a stateless protocol </li></ul><ul><ul><li>Each request is independent of previous request </li></ul></ul><ul><li>Servers have little information about where a request comes from </li></ul><ul><li>Web site software is extremely loosely coupled </li></ul><ul><ul><li>Coupled through the Internet – separated by space </li></ul></ul><ul><ul><li>Coupled to diverse hardware devices </li></ul></ul><ul><ul><li>Written in diverse software languages </li></ul></ul>© Jeff Offutt ICST 2008
  6. 6. Bypass Testing <ul><li>“ bypass ” client-side constraint enforcement </li></ul><ul><li>Bypass testing constructs tests to intentionally violate constraints : </li></ul><ul><ul><li>Eases test automation </li></ul></ul><ul><ul><li>Validates input validation </li></ul></ul><ul><ul><li>Checks robustness </li></ul></ul><ul><ul><li>Evaluates security </li></ul></ul>© Jeff Offutt ICST 2008
  7. 7. © Jeff Offutt User Name: Small $150 Version to purchase: Age: Large $500 Medium $250 Simple Example Web Page ICST 2008
  8. 8. © Jeff Offutt User Name: Small $150 Version to purchase: Age: Large $500 Medium $250 Proper Behavior ICST 2008 Username should be plain text only. Age should be between 18 and 150. Invalid data, please correct … Alan < Turing 500
  9. 9. Abbreviated HTML © Jeff Offutt <FORM > <INPUT Type=“text” Name=“username” Size=20> <INPUT Type=“text” Name=“age” Size=3 Maxlength=3> <P> Version to purchase: … <INPUT Type=“radio” Name=“version” Value=“150” Checked> <INPUT Type=“radio” Name=“version” Value=“250”> <INPUT Type=“radio” Name=“version” Value=“500”> <INPUT Type=&quot;submit&quot; onClick=&quot;return checkInfo(this.form)&quot;> <INPUT Type=“hidden” isLoggedIn=“no”> </FORM> ICST 2008
  10. 10. Bypass Behavior <ul><li>Extremely loose coupling … </li></ul><ul><li>combined with the stateless protocol … </li></ul><ul><li>allows users to easily bypass client-side checking : </li></ul>© Jeff Offutt Users can save and modify the HTML ICST 2008
  11. 11. Saved & Modified HTML © Jeff Offutt <FORM > <INPUT Type=“text” Name=“username” Size=20> <INPUT Type=“text” Name=“age” Size=3 Maxlength=3> <P> Version to purchase: … <INPUT Type=“radio” Name=“version” Value=“150”> <INPUT Type=“radio” Name=“version” Value=“250”> <INPUT Type=“radio” Name=“version” Value=“500” Checked> <INPUT Type=&quot;submit&quot; onClick=&quot;return checkInfo(this.form)&quot;> <INPUT Type=“hidden” isLoggedIn= “no” > </FORM> Allows an input with arbitrary age, no checking, cost=$25 … ‘ <‘ can crash an XML parser Text fields can have SQL statements 25 yes ICST 2008
  12. 12. SQL Injection © Jeff Offutt, 2004 User Name: Password: turing enigma Original SQL: SELECT username FROM adminuser WHERE username='turing' AND password ='enigma' “ injected” SQL: SELECT username FROM adminuser WHERE username='turing‘ OR ‘1’ = ‘1’ AND password ='enigma‘ OR ‘1’ = ‘1’ ’ OR ‘1’=‘1 ’ OR ‘1’=‘1
  13. 13. Applying Bypass Testing <ul><li>Analyze HTML to extract each form element </li></ul><ul><li>Model constraints imposed by HTML and JavaScript </li></ul><ul><li>Rules for data generation : </li></ul><ul><ul><li>From client-side constraints </li></ul></ul><ul><ul><li>Typical security violations </li></ul></ul><ul><ul><li>Common input mistakes </li></ul></ul>© Jeff Offutt ICST 2008 Validating input data on the client is like asking your opponent to hold your shield in a sword fight
  14. 14. Example Client-Side Constraint Rules <ul><li>Violate size restrictions on strings </li></ul><ul><li>Introduce values not included in static choices </li></ul><ul><ul><li>Radio boxes </li></ul></ul><ul><ul><li>Select ( drop-down ) lists </li></ul></ul><ul><li>Violate hard-coded values </li></ul><ul><li>Use values that JavaScripts flag as errors </li></ul><ul><li>Change “ transfer mode ” (get, post, …) </li></ul><ul><li>Change destination URL s </li></ul>© Jeff Offutt, 2004
  15. 15. Example Server-Side Constraint Rules <ul><li>Data type conversion </li></ul><ul><li>Data format validation </li></ul><ul><li>Inter-field constraint validation </li></ul><ul><li>Inter-request data fields (cookies, hidden) </li></ul>© Jeff Offutt, 2004
  16. 16. Example Security Violation Rules © Jeff Offutt, 2004 Potential Illegal Character Symbol Empty String Commas , Single and double quotes ’ or ” Tag symbols Tag symbols < and > Directory paths .. ../ Strings starting with forward slash / Strings starting with a period . Ampersands & Control character NIL, newline Characters with high bit set 254 and 255 Script symbols < javascript > or < vbscript >
  17. 17. First Example CyberChair <ul><li>CyberChair : Web-based conference management </li></ul><ul><ul><li>www.cyberchair.org </li></ul></ul><ul><ul><li>ICSE , ICST, ISSRE, ICSM, … </li></ul></ul><ul><li>Bypass testing found 5 types of faults </li></ul><ul><ul><li>Submission without authentication </li></ul></ul><ul><ul><li>Unsafe use of hidden form field </li></ul></ul><ul><ul><li>Disclosing information (program crashes) </li></ul></ul><ul><ul><li>Lack of validation of file type </li></ul></ul><ul><ul><li>Allows papers of negative length </li></ul></ul>© Jeff Offutt, 2004
  18. 18. Automating Bypass Testing <ul><li>Autobypass : A web application that accepts a URL and generates input data for the HTML form fields </li></ul><ul><ul><li>Also accepts any needed login data </li></ul></ul><ul><ul><li>MS thesis by Vasileios Papadimitriou </li></ul></ul><ul><li>Built on top of HttpUnit </li></ul><ul><ul><li>Parses HMTL pages </li></ul></ul><ul><ul><li>Identifies forms and their fields </li></ul></ul><ul><ul><li>Creates bypass test cases </li></ul></ul><ul><ul><li>Submits test cases to the application’s server </li></ul></ul>ICST 2008 © Jeff Offutt
  19. 19. Types of Client Input Validation <ul><li>Client side input validation is performed by HTML form controls, their attributes, and client side scripts that access DOM </li></ul><ul><li>Validation types are categorized as HTML and scripting </li></ul><ul><ul><li>HTML supports syntactic validation </li></ul></ul><ul><ul><li>Client scripting can perform both syntactic and semantic validation </li></ul></ul>ICST 2008 © Jeff Offutt HTML Constraints Scripting Constraints <ul><li>Length (max input characters) </li></ul><ul><li>Value (preset values) </li></ul><ul><li>Transfer Mode (GET or POST) </li></ul><ul><li>Field Element (preset fields) </li></ul><ul><li>Target URL (links with values) </li></ul><ul><li>Data Type (e.g. integer check) </li></ul><ul><li>Data Format (e.g. ZIP code format) </li></ul><ul><li>Data Value (e.g. age value range) </li></ul><ul><li>Inter-Value (e.g. credit # + exp. date) </li></ul><ul><li>Invalid Characters (e.g. <,../,&) </li></ul>
  20. 20. Example Interface: yahoo registration form ICST 2008 © Jeff Offutt Limited Length (HTML) Preset Values (HTML) Preset Transfer Mode in form definition (HTML) Preset No of Fields (HTML) URL with preset Values (HTML) Data Value, Type, & Format validation (script) Inter Value validation (script)
  21. 21. Test Value Selection <ul><li>Challenge : </li></ul><ul><ul><li>How to automatically provide effective test values? </li></ul></ul><ul><li>Semantic Domain Problem (SDP) </li></ul><ul><ul><li>Values within the application domain are needed </li></ul></ul><ul><ul><li>Enumeration of all possible test values is inefficient </li></ul></ul><ul><li>Possible Solutions </li></ul><ul><ul><li>Random Values (ineffective – lots of junk ) </li></ul></ul><ul><ul><li>Automatically generated values (very hard ) </li></ul></ul><ul><ul><li>Taking values from session log files (feasible but incomplete ) </li></ul></ul><ul><ul><li>Tester input ( feasible ) </li></ul></ul><ul><li>AutoBypass uses an input domain created by parsing the interface and tester input </li></ul>ICST 2008 © Jeff Offutt
  22. 22. AutoBypass <ul><li>AutoBypass steps (the big picture) </li></ul>ICST 2008 © Jeff Offutt Parse Interface Set Default Values Generate Test Cases & Run Tests Review Results <ul><li>All HTML violation rules are used to generate test cases </li></ul><ul><li>First version of AutoBypass does NOT automatically violate scripting validation , but : </li></ul><ul><ul><li>AutoBypass behaves as a browser with scripts disabled </li></ul></ul><ul><ul><li>Tester can provide test inputs that will bypass scripting validation. </li></ul></ul>
  23. 23. Real-World Examples ICST 2008 © Jeff Offutt atutor.ca Atalker demo.joomla.or Poll, Users phpMyAdmin Main page, Set Theme, SQL Query, DB Stats brainbench.com Submit Request Info, New user myspace.com Events & Music Search bankofamerica.com ATM locator, Site search comcast.com Service availability ecost.com Detail submit, Shopping cart control google.com Froogle, Language tools pageflakes.com Registration wellsfargolife.com Quote search nytimes.com Us-markets mutex.gmu.edu Login form yahoo.com Notepad, Composer, Search reminder, Weather Search barnesandnoble.com Cart manager, Book search/results amazon.com Item dispatch, Handle buy Pure black-box testing means no source (or permission) needed !
  24. 24. Classifying Output Responses <ul><li>( V ) Valid Responses : invalid inputs are adequately processed by the server </li></ul><ul><li>( F ) Faults & Failures : invalid inputs that cause abnormal server behavior (typically caught by web server when application fails to handle the error) </li></ul><ul><li>( E ) Exposure : invalid input is not recognized by the server and abnormal software behavior is exposed to the users </li></ul>ICST 2008 © Jeff Offutt ( V1 ) Server acknowledges the invalid request and provides an explicit message regarding the violation ( V2 ) Server produces a generic error message ( V3 ) Server apparently ignores the invalid request and produces an appropriate response ( V4 ) Server apparently ignores the request completely
  25. 25. Results ICST 2008 © Jeff Offutt v
  26. 26. Research to Practice ICST 2008 © Jeff Offutt “ Knowing is not enough, we must apply. Willing is not enough, we must do.” Goethe They’re teaching a new way of plowing over at the Grange tonight - you going? Naw - I already don’t plow as good as I know how...
  27. 27. Industrial Case Study <ul><li>Inventions from scientists are slow to move into industrial practice </li></ul><ul><li>We wanted to investigate whether the obstacles are : </li></ul><ul><ul><li>Technical difficulties of applying to industrial use </li></ul></ul><ul><ul><li>Social barriers </li></ul></ul><ul><ul><li>Business constraints </li></ul></ul><ul><li>Tried to technology transition bypass testing to the research arm of a software company </li></ul>ICST 2008 © Jeff Offutt
  28. 28. Avaya’s NPP Technology <ul><li>Avaya Labs Research creates research prototypes of software systems, then turns successful prototypes over to product groups </li></ul><ul><li>NPP : Notification Preference Portal </li></ul><ul><ul><li>Users specify how and when they should be contacted </li></ul></ul><ul><ul><li>Types include phone, email and SMS </li></ul></ul><ul><ul><li>Contacts an be made in parallel or sequentially </li></ul></ul><ul><li>Used to notify users of events </li></ul><ul><li>This study was part of system testing </li></ul><ul><ul><li>NPP is now in production </li></ul></ul>ICST 2008 © Jeff Offutt
  29. 29. NPP Design and Implementation <ul><li>NPP is a highly user interactive web application </li></ul><ul><li>Uses many screens </li></ul><ul><li>Javascript is used on the client to : </li></ul><ul><ul><li>Validate inputs </li></ul></ul><ul><ul><li>Dynamically modify screen by manipulating DOM </li></ul></ul><ul><ul><li>Encode input data into XML before sending to the server </li></ul></ul><ul><li>The second two uses necessitated changes to how bypass testing was applied </li></ul>ICST 2008 © Jeff Offutt
  30. 30. NPP Bypass Tests <ul><li>Bypass testing analyzes HTML ( statically ) and generates inputs that violate input constraints </li></ul><ul><li>The extensive modification of the HTML DOM meant HTML could not be analyzed statically </li></ul><ul><li>Instead, the input requirements of the server software were identified, and tests encoded in XML </li></ul><ul><ul><li>A special-purpose tool was written to convert XML tests into HtmlUnit tests </li></ul></ul>ICST 2008 © Jeff Offutt
  31. 31. NPP Testing Results <ul><li>Six NPP screens were tested </li></ul><ul><li>Tests are invalid inputs – exceptions are expected </li></ul><ul><li>Effects on back-end were not checked </li></ul><ul><ul><li>Failure analysis just based on response screens </li></ul></ul>ICST 2008 © Jeff Offutt Web Screen Tests Failing Tests Unique Failures Points of Contact 42 23 12 Time Profile 53 23 23 Notification Profile 34 12 6 Notification Filter 26 16 7 Change PIN 5 1 1 Create Account 24 17 14 TOTAL 184 92 63
  32. 32. Types of Faults <ul><li>Invalid data saved into the database </li></ul><ul><ul><li>These lead to more visible failures later </li></ul></ul><ul><ul><li>Example : An invalid password was accepted, but the account could not subsequently be used </li></ul></ul><ul><ul><li>Example : Subsequent messages could not be sent to invalid contacts </li></ul></ul><ul><li>No response at all </li></ul><ul><ul><li>Probably a software component failed </li></ul></ul><ul><ul><li>Database or web server sometimes crashed </li></ul></ul><ul><li>Exposure errors </li></ul><ul><ul><li>Internal exception message sent in the response screen </li></ul></ul>ICST 2008 © Jeff Offutt
  33. 33. Conclusions <ul><li>Bypass testing worked very well in an industrial context </li></ul><ul><ul><li>There is no technical obstacle to adoption </li></ul></ul><ul><ul><li>Source is not needed </li></ul></ul><ul><li>Even hand generation of tests was quite cheap in comparison with other methods </li></ul><ul><ul><li>There is no valid business barrier </li></ul></ul><ul><li>Most problems are unlikely with non-malicious users and a correct implementation </li></ul><ul><ul><li>But client-side validation is notoriously error-prone </li></ul></ul>ICST 2008 © Jeff Offutt We conclude the primary obstacle is social
  34. 34. Future Work <ul><li>A major observability problem with web application testing is detecting invalid database values </li></ul><ul><ul><li>A comprehensive valid data model could allow database auditors to be developed </li></ul></ul><ul><li>Javascript needs to be fully parsed and analyzed </li></ul><ul><ul><li>Implement scripting violation rules </li></ul></ul><ul><li>Widen the scope of testing from a form/ to a site </li></ul><ul><ul><li>Test sequence of events </li></ul></ul><ul><ul><li>Application level Input Domain </li></ul></ul><ul><li>Explore possibilities for automated response evaluation </li></ul><ul><li>Ajax allows client-server messages to be sent asynchronously through message passing </li></ul><ul><ul><li>This introduces more controllability and observability problems </li></ul></ul>ICST 2008 © Jeff Offutt
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×