Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Web Bypass Testing


Published on

  • Be the first to comment

  • Be the first to like this

Web Bypass Testing

  1. 1. An Industrial Case Study of Bypass Testing on Web Applications Joint research Dr. Ye Wu , Xiaochen Du , Hong Huang , Vasileios Papadimitriou , Qingxiang Wang and Joann J. Ordille of Avaya Labs Research Based on papers in ISSRE 2004 and ICST 2008 Jeff Offutt Software Engineering George Mason University Fairfax, VA USA [email_address] Expanded version of a talk given at the first International Conference on Software Testing, Verification and Validation
  2. 2. Outline of Talk <ul><li>Motivation </li></ul><ul><li>Applying bypass testing – early results </li></ul><ul><li>Automating bypass testing </li></ul><ul><li>Real-world examples </li></ul><ul><li>Industrial case study </li></ul><ul><li>Conclusions and future work </li></ul>ICST 2008 © Jeff Offutt
  3. 3. Web Application Input Validation Sensitive Data Client Server <ul><li>Bad Data </li></ul><ul><li>Corrupts data base </li></ul><ul><li>Crashes server </li></ul><ul><li>Security violations </li></ul>Check data Check data Malicious Data Can “bypass” data checking
  4. 4. Deploying Software <ul><li>Bundled : Pre-installed on computer </li></ul><ul><li>Shrink-wrap : Bought and installed by end-users </li></ul><ul><li>Contract : Purchaser pays developer to develop and install, usually for a fixed price </li></ul><ul><li>Embedded : Installed on a hardware device, usually with no direct communication with user </li></ul>© Jeff Offutt, 2004 <ul><li>Web : Executed across the Internet through HTTP </li></ul>
  5. 5. Problem Parameters <ul><li>HTTP is a stateless protocol </li></ul><ul><ul><li>Each request is independent of previous request </li></ul></ul><ul><li>Servers have little information about where a request comes from </li></ul><ul><li>Web site software is extremely loosely coupled </li></ul><ul><ul><li>Coupled through the Internet – separated by space </li></ul></ul><ul><ul><li>Coupled to diverse hardware devices </li></ul></ul><ul><ul><li>Written in diverse software languages </li></ul></ul>© Jeff Offutt ICST 2008
  6. 6. Bypass Testing <ul><li>“ bypass ” client-side constraint enforcement </li></ul><ul><li>Bypass testing constructs tests to intentionally violate constraints : </li></ul><ul><ul><li>Eases test automation </li></ul></ul><ul><ul><li>Validates input validation </li></ul></ul><ul><ul><li>Checks robustness </li></ul></ul><ul><ul><li>Evaluates security </li></ul></ul>© Jeff Offutt ICST 2008
  7. 7. © Jeff Offutt User Name: Small $150 Version to purchase: Age: Large $500 Medium $250 Simple Example Web Page ICST 2008
  8. 8. © Jeff Offutt User Name: Small $150 Version to purchase: Age: Large $500 Medium $250 Proper Behavior ICST 2008 Username should be plain text only. Age should be between 18 and 150. Invalid data, please correct … Alan < Turing 500
  9. 9. Abbreviated HTML © Jeff Offutt <FORM > <INPUT Type=“text” Name=“username” Size=20> <INPUT Type=“text” Name=“age” Size=3 Maxlength=3> <P> Version to purchase: … <INPUT Type=“radio” Name=“version” Value=“150” Checked> <INPUT Type=“radio” Name=“version” Value=“250”> <INPUT Type=“radio” Name=“version” Value=“500”> <INPUT Type=&quot;submit&quot; onClick=&quot;return checkInfo(this.form)&quot;> <INPUT Type=“hidden” isLoggedIn=“no”> </FORM> ICST 2008
  10. 10. Bypass Behavior <ul><li>Extremely loose coupling … </li></ul><ul><li>combined with the stateless protocol … </li></ul><ul><li>allows users to easily bypass client-side checking : </li></ul>© Jeff Offutt Users can save and modify the HTML ICST 2008
  11. 11. Saved & Modified HTML © Jeff Offutt <FORM > <INPUT Type=“text” Name=“username” Size=20> <INPUT Type=“text” Name=“age” Size=3 Maxlength=3> <P> Version to purchase: … <INPUT Type=“radio” Name=“version” Value=“150”> <INPUT Type=“radio” Name=“version” Value=“250”> <INPUT Type=“radio” Name=“version” Value=“500” Checked> <INPUT Type=&quot;submit&quot; onClick=&quot;return checkInfo(this.form)&quot;> <INPUT Type=“hidden” isLoggedIn= “no” > </FORM> Allows an input with arbitrary age, no checking, cost=$25 … ‘ <‘ can crash an XML parser Text fields can have SQL statements 25 yes ICST 2008
  12. 12. SQL Injection © Jeff Offutt, 2004 User Name: Password: turing enigma Original SQL: SELECT username FROM adminuser WHERE username='turing' AND password ='enigma' “ injected” SQL: SELECT username FROM adminuser WHERE username='turing‘ OR ‘1’ = ‘1’ AND password ='enigma‘ OR ‘1’ = ‘1’ ’ OR ‘1’=‘1 ’ OR ‘1’=‘1
  13. 13. Applying Bypass Testing <ul><li>Analyze HTML to extract each form element </li></ul><ul><li>Model constraints imposed by HTML and JavaScript </li></ul><ul><li>Rules for data generation : </li></ul><ul><ul><li>From client-side constraints </li></ul></ul><ul><ul><li>Typical security violations </li></ul></ul><ul><ul><li>Common input mistakes </li></ul></ul>© Jeff Offutt ICST 2008 Validating input data on the client is like asking your opponent to hold your shield in a sword fight
  14. 14. Example Client-Side Constraint Rules <ul><li>Violate size restrictions on strings </li></ul><ul><li>Introduce values not included in static choices </li></ul><ul><ul><li>Radio boxes </li></ul></ul><ul><ul><li>Select ( drop-down ) lists </li></ul></ul><ul><li>Violate hard-coded values </li></ul><ul><li>Use values that JavaScripts flag as errors </li></ul><ul><li>Change “ transfer mode ” (get, post, …) </li></ul><ul><li>Change destination URL s </li></ul>© Jeff Offutt, 2004
  15. 15. Example Server-Side Constraint Rules <ul><li>Data type conversion </li></ul><ul><li>Data format validation </li></ul><ul><li>Inter-field constraint validation </li></ul><ul><li>Inter-request data fields (cookies, hidden) </li></ul>© Jeff Offutt, 2004
  16. 16. Example Security Violation Rules © Jeff Offutt, 2004 Potential Illegal Character Symbol Empty String Commas , Single and double quotes ’ or ” Tag symbols Tag symbols < and > Directory paths .. ../ Strings starting with forward slash / Strings starting with a period . Ampersands & Control character NIL, newline Characters with high bit set 254 and 255 Script symbols < javascript > or < vbscript >
  17. 17. First Example CyberChair <ul><li>CyberChair : Web-based conference management </li></ul><ul><ul><li> </li></ul></ul><ul><ul><li>ICSE , ICST, ISSRE, ICSM, … </li></ul></ul><ul><li>Bypass testing found 5 types of faults </li></ul><ul><ul><li>Submission without authentication </li></ul></ul><ul><ul><li>Unsafe use of hidden form field </li></ul></ul><ul><ul><li>Disclosing information (program crashes) </li></ul></ul><ul><ul><li>Lack of validation of file type </li></ul></ul><ul><ul><li>Allows papers of negative length </li></ul></ul>© Jeff Offutt, 2004
  18. 18. Automating Bypass Testing <ul><li>Autobypass : A web application that accepts a URL and generates input data for the HTML form fields </li></ul><ul><ul><li>Also accepts any needed login data </li></ul></ul><ul><ul><li>MS thesis by Vasileios Papadimitriou </li></ul></ul><ul><li>Built on top of HttpUnit </li></ul><ul><ul><li>Parses HMTL pages </li></ul></ul><ul><ul><li>Identifies forms and their fields </li></ul></ul><ul><ul><li>Creates bypass test cases </li></ul></ul><ul><ul><li>Submits test cases to the application’s server </li></ul></ul>ICST 2008 © Jeff Offutt
  19. 19. Types of Client Input Validation <ul><li>Client side input validation is performed by HTML form controls, their attributes, and client side scripts that access DOM </li></ul><ul><li>Validation types are categorized as HTML and scripting </li></ul><ul><ul><li>HTML supports syntactic validation </li></ul></ul><ul><ul><li>Client scripting can perform both syntactic and semantic validation </li></ul></ul>ICST 2008 © Jeff Offutt HTML Constraints Scripting Constraints <ul><li>Length (max input characters) </li></ul><ul><li>Value (preset values) </li></ul><ul><li>Transfer Mode (GET or POST) </li></ul><ul><li>Field Element (preset fields) </li></ul><ul><li>Target URL (links with values) </li></ul><ul><li>Data Type (e.g. integer check) </li></ul><ul><li>Data Format (e.g. ZIP code format) </li></ul><ul><li>Data Value (e.g. age value range) </li></ul><ul><li>Inter-Value (e.g. credit # + exp. date) </li></ul><ul><li>Invalid Characters (e.g. <,../,&) </li></ul>
  20. 20. Example Interface: yahoo registration form ICST 2008 © Jeff Offutt Limited Length (HTML) Preset Values (HTML) Preset Transfer Mode in form definition (HTML) Preset No of Fields (HTML) URL with preset Values (HTML) Data Value, Type, & Format validation (script) Inter Value validation (script)
  21. 21. Test Value Selection <ul><li>Challenge : </li></ul><ul><ul><li>How to automatically provide effective test values? </li></ul></ul><ul><li>Semantic Domain Problem (SDP) </li></ul><ul><ul><li>Values within the application domain are needed </li></ul></ul><ul><ul><li>Enumeration of all possible test values is inefficient </li></ul></ul><ul><li>Possible Solutions </li></ul><ul><ul><li>Random Values (ineffective – lots of junk ) </li></ul></ul><ul><ul><li>Automatically generated values (very hard ) </li></ul></ul><ul><ul><li>Taking values from session log files (feasible but incomplete ) </li></ul></ul><ul><ul><li>Tester input ( feasible ) </li></ul></ul><ul><li>AutoBypass uses an input domain created by parsing the interface and tester input </li></ul>ICST 2008 © Jeff Offutt
  22. 22. AutoBypass <ul><li>AutoBypass steps (the big picture) </li></ul>ICST 2008 © Jeff Offutt Parse Interface Set Default Values Generate Test Cases & Run Tests Review Results <ul><li>All HTML violation rules are used to generate test cases </li></ul><ul><li>First version of AutoBypass does NOT automatically violate scripting validation , but : </li></ul><ul><ul><li>AutoBypass behaves as a browser with scripts disabled </li></ul></ul><ul><ul><li>Tester can provide test inputs that will bypass scripting validation. </li></ul></ul>
  23. 23. Real-World Examples ICST 2008 © Jeff Offutt Atalker demo.joomla.or Poll, Users phpMyAdmin Main page, Set Theme, SQL Query, DB Stats Submit Request Info, New user Events & Music Search ATM locator, Site search Service availability Detail submit, Shopping cart control Froogle, Language tools Registration Quote search Us-markets Login form Notepad, Composer, Search reminder, Weather Search Cart manager, Book search/results Item dispatch, Handle buy Pure black-box testing means no source (or permission) needed !
  24. 24. Classifying Output Responses <ul><li>( V ) Valid Responses : invalid inputs are adequately processed by the server </li></ul><ul><li>( F ) Faults & Failures : invalid inputs that cause abnormal server behavior (typically caught by web server when application fails to handle the error) </li></ul><ul><li>( E ) Exposure : invalid input is not recognized by the server and abnormal software behavior is exposed to the users </li></ul>ICST 2008 © Jeff Offutt ( V1 ) Server acknowledges the invalid request and provides an explicit message regarding the violation ( V2 ) Server produces a generic error message ( V3 ) Server apparently ignores the invalid request and produces an appropriate response ( V4 ) Server apparently ignores the request completely
  25. 25. Results ICST 2008 © Jeff Offutt v
  26. 26. Research to Practice ICST 2008 © Jeff Offutt “ Knowing is not enough, we must apply. Willing is not enough, we must do.” Goethe They’re teaching a new way of plowing over at the Grange tonight - you going? Naw - I already don’t plow as good as I know how...
  27. 27. Industrial Case Study <ul><li>Inventions from scientists are slow to move into industrial practice </li></ul><ul><li>We wanted to investigate whether the obstacles are : </li></ul><ul><ul><li>Technical difficulties of applying to industrial use </li></ul></ul><ul><ul><li>Social barriers </li></ul></ul><ul><ul><li>Business constraints </li></ul></ul><ul><li>Tried to technology transition bypass testing to the research arm of a software company </li></ul>ICST 2008 © Jeff Offutt
  28. 28. Avaya’s NPP Technology <ul><li>Avaya Labs Research creates research prototypes of software systems, then turns successful prototypes over to product groups </li></ul><ul><li>NPP : Notification Preference Portal </li></ul><ul><ul><li>Users specify how and when they should be contacted </li></ul></ul><ul><ul><li>Types include phone, email and SMS </li></ul></ul><ul><ul><li>Contacts an be made in parallel or sequentially </li></ul></ul><ul><li>Used to notify users of events </li></ul><ul><li>This study was part of system testing </li></ul><ul><ul><li>NPP is now in production </li></ul></ul>ICST 2008 © Jeff Offutt
  29. 29. NPP Design and Implementation <ul><li>NPP is a highly user interactive web application </li></ul><ul><li>Uses many screens </li></ul><ul><li>Javascript is used on the client to : </li></ul><ul><ul><li>Validate inputs </li></ul></ul><ul><ul><li>Dynamically modify screen by manipulating DOM </li></ul></ul><ul><ul><li>Encode input data into XML before sending to the server </li></ul></ul><ul><li>The second two uses necessitated changes to how bypass testing was applied </li></ul>ICST 2008 © Jeff Offutt
  30. 30. NPP Bypass Tests <ul><li>Bypass testing analyzes HTML ( statically ) and generates inputs that violate input constraints </li></ul><ul><li>The extensive modification of the HTML DOM meant HTML could not be analyzed statically </li></ul><ul><li>Instead, the input requirements of the server software were identified, and tests encoded in XML </li></ul><ul><ul><li>A special-purpose tool was written to convert XML tests into HtmlUnit tests </li></ul></ul>ICST 2008 © Jeff Offutt
  31. 31. NPP Testing Results <ul><li>Six NPP screens were tested </li></ul><ul><li>Tests are invalid inputs – exceptions are expected </li></ul><ul><li>Effects on back-end were not checked </li></ul><ul><ul><li>Failure analysis just based on response screens </li></ul></ul>ICST 2008 © Jeff Offutt Web Screen Tests Failing Tests Unique Failures Points of Contact 42 23 12 Time Profile 53 23 23 Notification Profile 34 12 6 Notification Filter 26 16 7 Change PIN 5 1 1 Create Account 24 17 14 TOTAL 184 92 63
  32. 32. Types of Faults <ul><li>Invalid data saved into the database </li></ul><ul><ul><li>These lead to more visible failures later </li></ul></ul><ul><ul><li>Example : An invalid password was accepted, but the account could not subsequently be used </li></ul></ul><ul><ul><li>Example : Subsequent messages could not be sent to invalid contacts </li></ul></ul><ul><li>No response at all </li></ul><ul><ul><li>Probably a software component failed </li></ul></ul><ul><ul><li>Database or web server sometimes crashed </li></ul></ul><ul><li>Exposure errors </li></ul><ul><ul><li>Internal exception message sent in the response screen </li></ul></ul>ICST 2008 © Jeff Offutt
  33. 33. Conclusions <ul><li>Bypass testing worked very well in an industrial context </li></ul><ul><ul><li>There is no technical obstacle to adoption </li></ul></ul><ul><ul><li>Source is not needed </li></ul></ul><ul><li>Even hand generation of tests was quite cheap in comparison with other methods </li></ul><ul><ul><li>There is no valid business barrier </li></ul></ul><ul><li>Most problems are unlikely with non-malicious users and a correct implementation </li></ul><ul><ul><li>But client-side validation is notoriously error-prone </li></ul></ul>ICST 2008 © Jeff Offutt We conclude the primary obstacle is social
  34. 34. Future Work <ul><li>A major observability problem with web application testing is detecting invalid database values </li></ul><ul><ul><li>A comprehensive valid data model could allow database auditors to be developed </li></ul></ul><ul><li>Javascript needs to be fully parsed and analyzed </li></ul><ul><ul><li>Implement scripting violation rules </li></ul></ul><ul><li>Widen the scope of testing from a form/ to a site </li></ul><ul><ul><li>Test sequence of events </li></ul></ul><ul><ul><li>Application level Input Domain </li></ul></ul><ul><li>Explore possibilities for automated response evaluation </li></ul><ul><li>Ajax allows client-server messages to be sent asynchronously through message passing </li></ul><ul><ul><li>This introduces more controllability and observability problems </li></ul></ul>ICST 2008 © Jeff Offutt