Paul Thwaite – Java 8 Test LeadWednesday 3 October 2012Testing with Bytecode Instrumentation (BCI)JavaOne 2012 - CON3961  ...
Important DisclaimersTHE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.WHILST EFF...
Introduction to the speaker – Paul Thwaite■   11 years experience developing and deploying Java SDKs■   Recent work focus:...
What you will learn from this talk■   Why me and what I do■   What is BCI?■   Typical problems encountered during testing■...
Why me and what I do■   Test IBMs Java SDK     – I break Java■   Focus on system testing in QA     – Multi-threaded load t...
Bytecode Instrumentation6                              © 2012 IBM Corporation
What is BCI?■   Bytecode Instrumentation (BCI) is modifying the bytes of a class at runtime■   Classes can be modified as ...
ASM■   ASM is an all purpose Java bytecode manipulation framework■   It can be used to modify existing classes or dynamica...
Problems encountered in testing9                                     © 2012 IBM Corporation
Typical problems encountered during testing■    Test exception handling■    Test for memory exhaustion■    Test secure pat...
Problem 1: Testing all exception pathstry {     Socket aSocket = new Socket("www.example.com", 1234);     // more code...}...
java.net.Socket12                © 2012 IBM Corporation
Problem 1: Testing all exception pathstry {        Socket aSocket = new Socket("www.example.com", 1234);        // more co...
Problem 1: Testing all exception pathstry {        Socket aSocket = new Socket("www.example.com", 1234);        // more co...
Problem 1: Testing all exception pathstry {        Socket aSocket = new Socket("www.example.com", 1234);         // more c...
Problem 1: Testing all exception pathstry {        Socket aSocket = new Socket("www.example.com", 1234);         // more c...
Demo using BCI to help solve          these problems17                                  © 2012 IBM Corporation
Problem 1: Demo of UnknownHostException■    What does the test do?      – Uses java.net.Socket to create a connection to t...
Problem 1: Demo of UnknownHostException■    What does the test do?      – Uses java.net.Socket to create a connection to t...
Problem 1: Breakdown of the Java command        Append the ASM                        Append a Java        library to the ...
Problem 2: Testing Resource Exhaustion■    A large system, such as WebSphere Application Server, should be able to handle ...
Problem 2: Demo of OutOfMemoryError – Java command■    What does the test do?      – Fills up the heap with Intergers unti...
Problem 2: Demo of OutOfMemoryError – stdout■    What does the test do?      – Fills up the heap with Intergers until an O...
Problem 2: Breakdown of the Java command         Enable the Java agent         Pass in options to                         ...
Recap – why BCI?■    Some parts of Java applications are hard to test■    BCI provides a solution to this problem■    With...
Recap – why BCI?■    Some parts of Java applications are hard to test■    BCI provides a solution to this problem■    With...
Java agents27                 © 2012 IBM Corporation
Java agent overview■    A Java agent provides instrumentation capabilities to a Java application■    Java agents provide t...
How BCI works29                   © 2012 IBM Corporation
Using BCI to modify java.net.Socket     Socket                                            visitMethod(info)      Socket(In...
Using BCI to modify java.net.Socket     Socket                                            visitMethod(info)      Socket(In...
Using BCI to modify java.net.Socket     Socket                                            visitMethod(info)      Socket(In...
Using BCI to modify java.net.Socket     Socket                                            visitMethod(info)      Socket(In...
Using BCI to modify java.net.Socket     Socket                                            visitMethod(info)      Socket(In...
Using BCI to modify java.net.Socket     Socket                                            visitMethod(info)      Socket(In...
Using BCI to modify java.net.Socket     Socket                                                  visitMethod(info)      Soc...
if (hostname.equals(host)) throw new                      UnknownHostException();                                     ASM ...
if (hostname.equals(host)) throw new                   UnknownHostException();                                  Bytecodes ...
Limitations of BCI at runtime■    Restrictions when retransforming classes at runtime:      – Fields cannot be added or re...
BCI Options40                 © 2012 IBM Corporation
Replace a class using a pre-compiled version of your new class■    Pre-compile a new version of a class■    When the class...
Modify bytecodes with a BCI framework■    Use a framework      – Provides an abstraction layer      – Easier to learn (pro...
Use javap to create bytecodes■    Write some java code that does what you want■    Compile your class to byte-code■    Use...
ASMifier – Create ASM logic■    ASMifier      – Part of standard ASM distribution      – Turn a Java method into byte code...
What can go wrong? (Dealing with class verify problems)■    If the JVM is unhappy with your modified class, it may throw a...
What can go wrong? (Runtime Problems)■    No compile time checking■    Access control      – Calling a private method is v...
Conclusion■    BCI is very powerful■    BCI is used in many products■    JVM knowledge is required but...■    There are ma...
References■    Get Products and Technologies:      – IBM Java Runtimes and SDKs:          • https://www.ibm.com/developerw...
References■    Eclipse Bytecode Outline plugin (allows you to use asmifier in Eclipse)      – http://andrei.gmxhome.de/byt...
Copyright and Trademarks© IBM Corporation 2012. All Rights Reserved.IBM, the IBM logo, and ibm.com are trademarks or regis...
Upcoming SlideShare
Loading in...5
×

JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation

909

Published on

An introduction into how bytecode instrumentation can help with testing hard-to-reach areas of your Java code.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
909
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
39
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation

  1. 1. Paul Thwaite – Java 8 Test LeadWednesday 3 October 2012Testing with Bytecode Instrumentation (BCI)JavaOne 2012 - CON3961 © 2012 IBM Corporation
  2. 2. Important DisclaimersTHE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATIONCONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS ORIMPLIED.ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLEDENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTUREDIFFERENCES.ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCTPLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USEOF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIRSUPPLIERS AND/OR LICENSORS2 © 2012 IBM Corporation
  3. 3. Introduction to the speaker – Paul Thwaite■ 11 years experience developing and deploying Java SDKs■ Recent work focus: – Quality Assurance lead • Leading cross-continent teams • System Test on IBM SDKs – Test lead for Java 8 • Focus on testing in OpenJDK • OpenJDK contributor■ My contact information: – paul_thwaite@uk.ibm.com – linkedin.com/profile/view?id=263279693 © 2012 IBM Corporation
  4. 4. What you will learn from this talk■ Why me and what I do■ What is BCI?■ Typical problems encountered during testing■ Two demos showing how BCI can help with testing■ Introduction to Java agents■ How BCI works■ Limitations of BCI■ Overview of the different BCI approaches4 © 2012 IBM Corporation
  5. 5. Why me and what I do■ Test IBMs Java SDK – I break Java■ Focus on system testing in QA – Multi-threaded load testing – Third-party applications■ System test the Java SDK means writing Java■ Can the VM handle any application or Java code?5 © 2012 IBM Corporation
  6. 6. Bytecode Instrumentation6 © 2012 IBM Corporation
  7. 7. What is BCI?■ Bytecode Instrumentation (BCI) is modifying the bytes of a class at runtime■ Classes can be modified as the JVM loads them or at any point afterwards■ Application and SDK classes can be changed (with few restrictions)■ Abstraction frameworks exist to make BCI easier Disk Runtime Snippets Class JVM Code Class BCI Code Code Code7 © 2012 IBM Corporation
  8. 8. ASM■ ASM is an all purpose Java bytecode manipulation framework■ It can be used to modify existing classes or dynamically generate classes, directly in binary form■ Many products such as AspectJ and JRuby use ASM■ The examples in this presentation use ASM to abstract out the complexity of transforming classes8 © 2012 IBM Corporation
  9. 9. Problems encountered in testing9 © 2012 IBM Corporation
  10. 10. Typical problems encountered during testing■ Test exception handling■ Test for memory exhaustion■ Test secure paths through code■ Test specific code paths in very large systems such as WebSphere Application Server10 © 2012 IBM Corporation
  11. 11. Problem 1: Testing all exception pathstry { Socket aSocket = new Socket("www.example.com", 1234); // more code...} catch (UnknownHostException e) { //Handle correctly} catch (IOException e) { //Handle correctly} catch (SecurityException e) { //Handle correctly}11 © 2012 IBM Corporation
  12. 12. java.net.Socket12 © 2012 IBM Corporation
  13. 13. Problem 1: Testing all exception pathstry { Socket aSocket = new Socket("www.example.com", 1234); // more code...} catch (UnknownHostException e) { //Handle correctly} catch (IOException e) { //Handle correctly} catch (SecurityException e) { //Handle correctly}■ The test running normally13 © 2012 IBM Corporation
  14. 14. Problem 1: Testing all exception pathstry { Socket aSocket = new Socket("www.example.com", 1234); // more code...} catch (UnknownHostException e) { //Handle correctly} catch (IOException e) { //Handle correctly} catch (SecurityException e) { //Handle correctly}■ How do you force the exception? – Pull the network cable out of the test machine – Update /etc/hosts to force the exception■ Not practical – Manual, test-specific, other tests may run on the same machine14 © 2012 IBM Corporation
  15. 15. Problem 1: Testing all exception pathstry { Socket aSocket = new Socket("www.example.com", 1234); // more code...} catch (UnknownHostException e) { //Handle correctly} catch (IOException e) { //Handle correctly} catch (SecurityException e) { //Handle correctly}■ How do you force the exception? – Intercept the connection – IOException is a “catch all” and very hard to create – You would need to understand the implementation in order to test properly■ Not practical15 © 2012 IBM Corporation
  16. 16. Problem 1: Testing all exception pathstry { Socket aSocket = new Socket("www.example.com", 1234); // more code...} catch (UnknownHostException e) { //Handle correctly} catch (IOException e) { //Handle correctly} catch (SecurityException e) { //Handle correctly}■ How do you force the exception? – A SecurityManager would need to be used – Complicated to setup – Not practical especially if testing a complex system – Likely would need two versions of the test16 © 2012 IBM Corporation
  17. 17. Demo using BCI to help solve these problems17 © 2012 IBM Corporation
  18. 18. Problem 1: Demo of UnknownHostException■ What does the test do? – Uses java.net.Socket to create a connection to three hosts – Prints to stdout whether the connection was successful or not – BCI is used to trigger the UnknownHostException on www.oracle.com■ Test 1 – the test running normally java -cp testcase tests.NetworkTest■ Test 2 – the test running with BCI java -Xbootclasspath/a:lib/asm-all-4.0.jar:lib/networkTrans.jar -javaagent:lib/networkTrans.jar=www.oracle.com -cp testcase tests.NetworkTest Pass in the host on which to force the UnknownHostException18 © 2012 IBM Corporation
  19. 19. Problem 1: Demo of UnknownHostException■ What does the test do? – Uses java.net.Socket to create a connection to three hosts – Prints to stdout whether the connection was successful or not – BCI is used to trigger the UnknownHostException on www.oracle.com■ Test 1 – the test running normally Connected to host: www.example.com Connected to host: www.oracle.com Connected to host: www.ibm.com■ Test 2 – the test running with BCI Connected to host: www.example.com Couldnt find host: www.oracle.com UnknownHostException thrown Connected to host: www.ibm.com19 © 2012 IBM Corporation
  20. 20. Problem 1: Breakdown of the Java command Append the ASM Append a Java library to the agent library to bootclasspath the bootclasspath java -Xbootclasspath/a:lib/asm-all- 4.0.jar:lib/networkTrans.jar -javaagent:lib/networkTrans.jar=www.oracle.com -cp testcase tests.NetworkTest Enable the Java agent Pass in options to the agent Run the test as normal - the test does not change Note: The bootclasspath is used because the Socket class is being modified.20 © 2012 IBM Corporation
  21. 21. Problem 2: Testing Resource Exhaustion■ A large system, such as WebSphere Application Server, should be able to handle an OutOfMemoryError■ An OutOfMemoryError can occur at any time – It can also happen in different places within the system■ Hard and complicated to test without side effects – Changing the application servers environment is not a good idea – Using a reduced heap will not guarantee all cases are covered21 © 2012 IBM Corporation
  22. 22. Problem 2: Demo of OutOfMemoryError – Java command■ What does the test do? – Fills up the heap with Intergers until an OutOfMemoryError exception is thrown – It will take a very long time for the OutOfMemoryError exception to be thrown – BCI is used to set the trigger to just three Integers without having to change the test■ Test 1 – the test running normally java -cp testcase tests.AllocateTest Option to trigger OOME at just 3 Integers■ Test 2 – the test running with BCI java -javaagent:lib/resourceTrans.jar=3 -cp testcase:lib/asm-all-4.0.jar tests.AllocateTest22 © 2012 IBM Corporation
  23. 23. Problem 2: Demo of OutOfMemoryError – stdout■ What does the test do? – Fills up the heap with Intergers until an OutOfMemoryError exception is thrown – It will take a very long time for the OutOfMemoryError exception to be thrown – BCI is used to set the trigger to just three Integers without having to change the test■ Test 1 – the test running normally Adding new value 0 Adding new value 1 Adding new value 2... ...Adding new value 450...■ Test 2 – the test running with BCI Adding new value 0 Adding new value 1 Adding new value 2 Adding new value 3 Exception in thread "main" java.lang.OutOfMemoryError at transformer.ResourceTransformer.resourceHelper(ResourceTransformer.java:79) at tests.AllocateTest.increment(AllocateTest.java) at tests.AllocateTest.main(AllocateTest.java:12)23 © 2012 IBM Corporation
  24. 24. Problem 2: Breakdown of the Java command Enable the Java agent Pass in options to the agent java -javaagent:lib/resourceTrans.jar=3 -cp testcase:lib/asm-all-4.0.jar tests.AllocateTest Add the ASM library to Run the test as normal - the classpath the test does not change24 © 2012 IBM Corporation
  25. 25. Recap – why BCI?■ Some parts of Java applications are hard to test■ BCI provides a solution to this problem■ With BCI, you can: – Replace your method bodies at runtime – Replace your entire class – All without having to change your code■ BCI is performed by inserting new bytecodes into your application at runtime■ Using the UnknownHostException example, there are two ways to perform BCI – Change the testcase to use a mock version of java.net.Socket which calls a different class i.e. MySocketClass – Modify the java.net.Socket class■ Whichever you choose, you will be playing with bytecodes25 © 2012 IBM Corporation
  26. 26. Recap – why BCI?■ Some parts of Java applications are hard to test■ BCI provides a solution to this problem■ With BCI, you can: – Replace your method bodies at runtime – Replace your entire class – All without having to change your code■ BCI is performed by inserting new bytecodes into your application at runtime■ Using the UnknownHostException example, there are two ways to perform BCI – Change the testcase to use a mock version of java.net.Socket which calls a different class i.e. MySocketClass – Modify the java.net.Socket class■ Whichever you choose, you will be playing with bytecodes■ But dont fear – there are tools available which can help you!26 © 2012 IBM Corporation
  27. 27. Java agents27 © 2012 IBM Corporation
  28. 28. Java agent overview■ A Java agent provides instrumentation capabilities to a Java application■ Java agents provide the means to perform bytecode transformation■ An agent must have a premain method import java.lang.instrument.Instrumentation; public class MyJavaAgent { public static void premain(String agentArgument, Instrumentation instrumentation) {} } Manifest-Version: 1.0 Premain-Class: MyJavaAgent■ It is packaged inside a JAR file■ The JAR file manifest defines the agent class■ The agent has no modelling restrictions i.e. it can do anything any other java code can do – Start threads, listen on the network etc■ The Instrumention object is used to perform class transformations28 © 2012 IBM Corporation
  29. 29. How BCI works29 © 2012 IBM Corporation
  30. 30. Using BCI to modify java.net.Socket Socket visitMethod(info) Socket(InetAddress addr) isTargetMethod? ASM visitCode() Socket(String host,int port) insert new code30 © 2012 IBM Corporation
  31. 31. Using BCI to modify java.net.Socket Socket visitMethod(info) Socket(InetAddress addr) isTargetMethod? ASM visitCode() Socket(String host,int port) insert new code31 © 2012 IBM Corporation
  32. 32. Using BCI to modify java.net.Socket Socket visitMethod(info) Socket(InetAddress addr) isTargetMethod? ASM visitCode() Socket(String host,int port) insert new code32 © 2012 IBM Corporation
  33. 33. Using BCI to modify java.net.Socket Socket visitMethod(info) Socket(InetAddress addr) isTargetMethod? ASM visitCode() Socket(String host,int port) insert new code33 © 2012 IBM Corporation
  34. 34. Using BCI to modify java.net.Socket Socket visitMethod(info) Socket(InetAddress addr) isTargetMethod? ASM visitCode() Socket(String host,int port) insert new code34 © 2012 IBM Corporation
  35. 35. Using BCI to modify java.net.Socket Socket visitMethod(info) Socket(InetAddress addr) isTargetMethod? ASM visitCode() Socket(String host,int port) insert new code35 © 2012 IBM Corporation
  36. 36. Using BCI to modify java.net.Socket Socket visitMethod(info) Socket(InetAddress addr) isTargetMethod? ASM visitCode() Socket(String host,int port) insert new code if(hostname.equals(host)) throw new UnknownHostException();36 © 2012 IBM Corporation
  37. 37. if (hostname.equals(host)) throw new UnknownHostException(); ASM logic mv.visitVarInsn(Opcodes.ALOAD, 1); mv.visitMethodInsn(INVOKESPECIAL, "java/net/UnknownHostException", "<init>", "()V"); mv.visitCode();37 © 2012 IBM Corporation
  38. 38. if (hostname.equals(host)) throw new UnknownHostException(); Bytecodes 0: aload_1 1: ldc #2; // String host 3: invokevirtual #3; //Method java/lang/String.equals:(Ljava/lang/Object;)Z 6: ifeq 17 9: new #4; //class java/net/UnknownHostException 12: dup 13: invokespecial #5; //Method java/net/UnknownHostException."<init>":()V 16: athrow 17: return38 © 2012 IBM Corporation
  39. 39. Limitations of BCI at runtime■ Restrictions when retransforming classes at runtime: – Fields cannot be added or removed – Methods cannot be added or removed – Method signatures cannot be modified■ Effectively only method bodies can be modified■ Most classes can be modified (including most SDK classes)39 © 2012 IBM Corporation
  40. 40. BCI Options40 © 2012 IBM Corporation
  41. 41. Replace a class using a pre-compiled version of your new class■ Pre-compile a new version of a class■ When the class to be loaded next is the one you want to replace■ You can return the bytes for your new class and discard the old one■ No knowledge of bytecode is required■ Original source must be available and unchanged■ Need to match the correct compiler version41 © 2012 IBM Corporation
  42. 42. Modify bytecodes with a BCI framework■ Use a framework – Provides an abstraction layer – Easier to learn (provides symbolic constants, helper APIs) – Adds an additional dependency■ ASM – Widely used – Uder active development – Friendly license■ ASM gives you a good symbolic framework for writing bytecodes■ Knowledge of the JVM architecture is required42 © 2012 IBM Corporation
  43. 43. Use javap to create bytecodes■ Write some java code that does what you want■ Compile your class to byte-code■ Use a de-compiler to examine the class file■ JVM ships with javap■ verbose option (-v) decompiles method bytecodes void setupConnections(); Code: 0: ldc #2 // String www.example.com 2: sipush 1234 5: invokestatic #3 // Method MySocketFactory.newSocket: (Ljava/lang/String;I)Ljava/net/Socket; 8: astore_1 9: return43 © 2012 IBM Corporation
  44. 44. ASMifier – Create ASM logic■ ASMifier – Part of standard ASM distribution – Turn a Java method into byte code logic for insertion into an ASM class – An Eclipse plugin is available mv.visitLdcInsn("www.example.com"); mv.visitIntInsn(SIPUSH, 1234); mv.visitMethodInsn(INVOKESTATIC, "transformer/SocketHelper$MySocketFactory", "newSocket", "(Ljava/lang/String;I)Ljava/net/Socket;"); mv.visitVarInsn(ASTORE, 1); mv.visitInsn(RETURN);44 © 2012 IBM Corporation
  45. 45. What can go wrong? (Dealing with class verify problems)■ If the JVM is unhappy with your modified class, it may throw a java.lang.VerifyError■ stack shape inconsistent/stack underflow – Attempting to operate on the wrong type of variable/stack is empty – Often due to failure to push/pop all the correct variables■ Inconsistent stackmap frames – Stackmaps are method meta-data used by the JVM – A map of all the variables on the stack at the target of any branch/goto instruction – BCI cause targets to move, invalidating the stack maps – ASM will re-calculate them for you■ stack overflow – Maximum stack size specified in method meta data – Any operations added via BCI may increase required stack – ASM will calculate that for you too45 © 2012 IBM Corporation
  46. 46. What can go wrong? (Runtime Problems)■ No compile time checking■ Access control – Calling a private method is valid byte-code, fails at runtime■ Calling a non-existent method – Incorrect argument types – Incorrect argument ordering – Typos46 © 2012 IBM Corporation
  47. 47. Conclusion■ BCI is very powerful■ BCI is used in many products■ JVM knowledge is required but...■ There are many tools available to make it easier■ BCI allows testing of otherwise impossible or tricky situations■ Slides will be available on Slideshare after the conference■ Examples are available on request – paul_thwaite@uk.ibm.com47 © 2012 IBM Corporation
  48. 48. References■ Get Products and Technologies: – IBM Java Runtimes and SDKs: • https://www.ibm.com/developerworks/java/jdk/ – IBM Monitoring and Diagnostic Tools for Java: • https://www.ibm.com/developerworks/java/jdk/tools/■ Learn: – IBM Java InfoCenter: • http://publib.boulder.ibm.com/infocenter/java7sdk/v7r0/index.jsp■ Discuss: – IBM Java Runtimes and SDKs Forum: • http://www.ibm.com/developerworks/forums/forum.jspa?forumID=367&start=048 © 2012 IBM Corporation
  49. 49. References■ Eclipse Bytecode Outline plugin (allows you to use asmifier in Eclipse) – http://andrei.gmxhome.de/bytecode/index.html■ ASM – http://asm.ow2.org/index.html■ Java VM Spec – http://docs.oracle.com/javase/specs/jvms/se7/html/index.html■ java.lang.instrument package documentation – http://docs.oracle.com/javase/7/docs/api/index.html?java/lang/instrument/package-summary.html49 © 2012 IBM Corporation
  50. 50. Copyright and Trademarks© IBM Corporation 2012. All Rights Reserved.IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International BusinessMachines Corp., and registered in many jurisdictions worldwide.Other product and service names might be trademarks of IBM or other companies.A current list of IBM trademarks is available on the Web – see the IBM “Copyright and trademarkinformation” page at URL: www.ibm.com/legal/copytrade.shtml50 © 2012 IBM Corporation
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×