| ApacheConUS 2008 Conference | New Orleans, LA | Stormin ...


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

| ApacheConUS 2008 Conference | New Orleans, LA | Stormin ...

  1. 1. Tim Ellison Apache Harmony Project http://harmony.apache.org Stormin' the coffee cup with Apache Harmony
  2. 2. Join me during the next 45 mins to… <ul><ul><li>Learn about the motivations, current status, and future plans of the Apache Harmony project. </li></ul></ul>
  3. 3. Agenda <ul><li>Project History </li></ul><ul><li>Development Model </li></ul><ul><li>Code and Runtime Modularity </li></ul><ul><li>How Are We Doing? </li></ul><ul><li>Demo </li></ul><ul><li>Harmony and the JCK </li></ul><ul><li>Wrap-up </li></ul>
  4. 4. <ul><li>Primary goals </li></ul><ul><ul><li>Protect IP rights of the Java ecosystem. </li></ul></ul><ul><ul><li>Independent implementation of virtual machines , related class libraries , and other software commonly associated with Java platform implementations. </li></ul></ul><ul><ul><li>Community-developed , modular architecture allowing sharing and innovation . </li></ul></ul>Apache Harmony – a brief history 2005 2006 2007 2008 Harmony established in Apache incubator May 2005 IBM contributes core classes and binary VM license November 2005 Intel contributes VM/JIT/GC/classlib code January 2006… Established legal frame- work around Harmony No code Subset of 1.4 / 5.0 >99% 5.0, >90% 6.0 Java 6 work started March 2006 Open letter to Sun for JCK 5 April 2007 Harmony graduates as top level project October 2006 Day to day development correctness, completeness, performance, … ME code contribution April 2008 1 5 2 3 4
  5. 5. <ul><li>Contributors detail their prior access via a project questionnaire. </li></ul><ul><li>Developers can contribute in functional areas where they have not studied closed-source implementations (exceptions apply). </li></ul><ul><li>Existing code being contributed to the project must provide acceptable pedigree information. </li></ul>Enhanced IP-cleanliness See: http://harmony.apache.org/auth_cont_quest.html This is in addition to the standard Apache contribution processes
  6. 6. Bootstrapping the project <ul><li>Establish a common goal </li></ul><ul><ul><li>compliant and compatible Java SE 5 implementation </li></ul></ul><ul><li>JVM contributions </li></ul><ul><ul><li>initial VM implementations, … from Cobbs and Lydick </li></ul></ul><ul><ul><li>DRLVM, … from Intel </li></ul></ul><ul><li>Class library code contributions </li></ul><ul><ul><li>core lang, util, net, IO, beans, … from IBM </li></ul></ul><ul><ul><li>security, print, sound, awt, swing, … from Intel </li></ul></ul><ul><ul><li>math, rmi, … from National University of Cordoba, Argentina </li></ul></ul><ul><li>Production-quality VMs for build and test </li></ul><ul><ul><li>IBM J9 and BEA JRockit made available to project </li></ul></ul><ul><ul><ul><ul><li>Used for development </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Not under open source license, not a contribution </li></ul></ul></ul></ul>© javic flickr.com/photos/javic
  7. 7. Agenda <ul><li>Project History </li></ul><ul><li>Development Model </li></ul><ul><li>Code and Runtime Modularity </li></ul><ul><li>How Are We Doing? </li></ul><ul><li>Demo </li></ul><ul><li>Harmony and the JCK </li></ul><ul><li>Wrap-up </li></ul>
  8. 8. Spec-driven development <ul><li>Implementing the Java specifications as found in </li></ul><ul><ul><ul><li>Java SE platform Javadoc </li></ul></ul></ul><ul><ul><ul><li>Java Programming Language Specification </li></ul></ul></ul><ul><ul><ul><li>Java Virtual Machine (JVM) Specification, etc. </li></ul></ul></ul><ul><li>Ambiguities and omissions... </li></ul><ul><ul><li>resolved by consulting the Reference Implementation </li></ul></ul><ul><ul><li>determined by functional API testing </li></ul></ul>Producing a compliant implementation Writing Unit Tests Specification Reading Post intentions on the dev list Send patch and discuss problems on dev list Writing implementation iterative progress
  9. 9. Test-driven development <ul><li>Functional testing </li></ul><ul><ul><ul><li>API tests, internal interfaces tests, bug regression tests </li></ul></ul></ul><ul><ul><ul><li>Component oriented </li></ul></ul></ul><ul><li>Integration testing </li></ul><ul><ul><li>Build verification, test with different VMs </li></ul></ul><ul><ul><li>Component assembly oriented </li></ul></ul><ul><li>Application / System testing </li></ul><ul><ul><li>Running popular applications, ad hoc and test suites </li></ul></ul><ul><ul><li>End-product and compatibility oriented </li></ul></ul><ul><li>Platform, Performance and Stress testing </li></ul><ul><ul><li>Multi-platform continuous integration, performance / stress suites </li></ul></ul><ul><ul><li>Robustness and quality oriented </li></ul></ul>Producing a robust, compatible implementation until proven innocent all code is guilty ©B Tal, flickr.com
  10. 10. <ul><li>/classlib – the class library code </li></ul><ul><li>/drlvm – the VM, JIT, GC </li></ul><ul><li>/jdktools – Java technology development tools (javac, javah, javap, ...) </li></ul><ul><li>/trunk – federation point (builds everything together) </li></ul>Project repository organization Java 5 code base is >1.5 Million non-comment lines of code generated using David A. Wheeler's 'SLOCCount'
  11. 11. Federated workspace build <ul><li>Uses svn switch “trick” to assemble source code </li></ul><ul><li>Downloads project dependencies </li></ul><ul><li>Running “ant” results in complete HDK, Java Development Kit (JDK) and Java Runtime Environment (JRE) </li></ul><ul><li>Can work in any directory (svn commit does the Right Thing) </li></ul><ul><li>ant -Dsvn.revision=X builds any SVN revision </li></ul>Bringing it all together /trunk build.xml /working_classlib -> /classlib /working_vm -> /<your favorite vm> /working_jdktools -> /jdktools /current_resources -> /current_resources
  12. 12. Heterogeneous build and test farm across community Distributed build-test @ bug tracker committers GIT | Quilt | ? Check out Patches Continuous integration and test by community’s heterogeneous build farm Notify SVN Server contributors Check in/out E-mail Apache infrastructure
  13. 13. Agenda <ul><li>Project History </li></ul><ul><li>Development Model </li></ul><ul><li>Code and Runtime Modularity </li></ul><ul><li>How Are We Doing? </li></ul><ul><li>Demo </li></ul><ul><li>Harmony and the JCK </li></ul><ul><li>Wrap-up </li></ul>
  14. 14. Trace JIT compiler Interpreter Memory manager Thread manager VM XML NIO beans native code Java Native Interface (JNI™) Portability Layer Hardware / Operating System VMI (C) security Java SE class libraries Java SE Platform API Launcher Class library Kernel classes (Java class files) Everything is pluggable Architectural overview lang util net io
  15. 15. Harmony DRLVM – the Java Virtual Machine <ul><li>Core interpreter </li></ul><ul><ul><li>Bytecode verifier, class file loading / unloading, debug interface, JVMTI etc. </li></ul></ul><ul><li>Just in Time compilers </li></ul><ul><ul><li>profiling engine selects compiler based on typical code paths, data values, … </li></ul></ul><ul><ul><li>“ JET” fast compiler </li></ul></ul><ul><ul><ul><li>~14K NSLOC, ~10-20K methods/sec, resulting code 10x faster than interpreter </li></ul></ul></ul><ul><ul><li>“ OPT” optimizing compiler </li></ul></ul><ul><ul><ul><li>multiple code transformations, resulting code 2x faster than JET </li></ul></ul></ul><ul><li>Memory managers </li></ul><ul><ul><li>“ GCv4” stop the world, sequential copying compacting </li></ul></ul><ul><ul><li>“ GCv5” stop the world, parallel copying compacting generational </li></ul></ul><ul><ul><li>“ Tick” on the fly, concurrent mark, parallel, mark-sweep-compact generational </li></ul></ul>
  16. 16. Virtual Machine modularity <ul><li>Well-defined interfaces, consistent across platforms </li></ul><ul><li>Interfaces do not compromise runtime performance </li></ul><ul><li>Modules either build-time or runtime replaceable </li></ul><ul><li>Multiple implementations already exist for some modules </li></ul>Code execution and memory management
  17. 17. Harmony’s VMI <ul><li>VM-specific ‘kernel’ classes </li></ul><ul><ul><li>23 publicly defined Java SE types that the VM typically knows intimately, plus one helper </li></ul></ul><ul><li>Access to our C struct from JavaVM / JNIEnv pointers </li></ul><ul><ul><li>10 new C functions that augment standard JNI </li></ul></ul><ul><li>The VMI provides: </li></ul><ul><ul><li>Access to the operating system abstraction library ( port library ) </li></ul></ul><ul><ul><li>Access to per-VM storage functions ( VMLS ) which allows multiple VMs to exist in a single address space </li></ul></ul><ul><ul><li>Ability to get/set/iterate system properties </li></ul></ul><ul><li>The VMI does not : </li></ul><ul><ul><li>Require any enhanced VM/class library linkage </li></ul></ul><ul><ul><li>Prescribe object layout, garbage collection, synchronization, and so on </li></ul></ul>Virtual Machine Interface to the class libraries class libraries
  18. 18. Class library modularization <ul><li>A module… </li></ul><ul><ul><li>is related functionality scoped by Java packages </li></ul></ul><ul><ul><li>‘ exports’ user-API and internal-API, hides private internal implementation </li></ul></ul><ul><ul><li>defined by dependencies in the Java specification </li></ul></ul><ul><ul><li>minimizes coupling by explicit internal APIs </li></ul></ul><ul><ul><li>is delivered as a JAR file, a real OSGi bundle </li></ul></ul>Java SE platform implemented in ~30 components <ul><li>Observed benefits </li></ul><ul><ul><li>easier to manage prior exposure </li></ul></ul><ul><ul><li>freedom of assembly for module consumers </li></ul></ul><ul><ul><li>unit of replacement for fixes and updates </li></ul></ul><ul><ul><li>facilitates contributions </li></ul></ul>XML NIO Beans Class Library Native Code VMI (C) JNI API Kernel classes (Java class files) lang util net io Security Java SE Platform 5 API
  19. 19. Development time modularity <ul><li>Not a replacement for JDK software </li></ul><ul><ul><li>HDK  Harmony developers :: JDK software  Java developers </li></ul></ul><ul><ul><li>Contains all files necessary for Harmony development and testing </li></ul></ul><ul><li>Enables fast rebuild of individual modules – Java platform and native </li></ul><ul><li>Removes necessity to check out whole source tree </li></ul><ul><li>Supports separate or in-place development of HDK trees </li></ul>The Harmony Development Kit (“HDK”) $ svn co http://... modules/nio $ cd modules/nio $ ant -DHY.HDK= path/to/hdk -DHY.TARGET= path/to/target
  20. 20. Packaging Modularity rt.jar Software assemblies jsse.jar jce.jar … prefs.jar + prefs-src.jar + hyprefs.dll Harmony’s approach Conventional approach Applet AWT Beans LUNI Annotation Instrument Lang-mgmt Math NIO NIO_char Security SQL Text Concurrent Archive Logging Prefs
  21. 21. Multi-configuration support <ul><li>Harmony modules can be easily configured: </li></ul><ul><ul><li>Apache Harmony : builds all modules into Java 5 and Java 6 configurations </li></ul></ul><ul><ul><li>IBM Java 6 SE : a number of modules sourced from Harmony </li></ul></ul><ul><ul><li>Android : core libraries adapted to run on Dalvik VM </li></ul></ul>“ Right sized” application runtimes     x-net kXML   Xerces xml     text     swing     sql     sound     security     rmi     regex     print     prefs     pack200     nio_char     nio     math     luni     logging     lang-management     jndi     instrument     imageio     crypto     concurrent     beans     awt     auth     archive     applet     annotation     accessibility Android IBM Java 6 Harmony
  22. 22. Runtime modularity <ul><li>Single native launcher program </li></ul><ul><li>Runs command-line Java technology programs, including generic launcher and JDK software tools </li></ul><ul><li>Select VM provider based on command-line option </li></ul><ul><li>Select runtime specified modules (e.g. GC algorithm) </li></ul>Multi-VM and tools launcher jre/bin/ java.exe default/ drlvm-v1/ drlvm-v2/ ibm-j9/ $ java MyClass $ java –vmdir:ibm-j9 MyClass $ java –vmdir:drlvm-v1 MyClass
  23. 23. Agenda <ul><li>Project History </li></ul><ul><li>Development Model </li></ul><ul><li>Code and Runtime Modularity </li></ul><ul><li>How Are We Doing? </li></ul><ul><li>Demo </li></ul><ul><li>Harmony and the JCK </li></ul><ul><li>Wrap-up </li></ul>
  24. 24. <ul><li>Completeness “heat map” courtesy of JAPI tool </li></ul><ul><ul><li>compares API compatibility </li></ul></ul><ul><li>Over 99% 1.5 API complete </li></ul><ul><li>Missing functionality is now primarily in… </li></ul><ul><ul><li>providers </li></ul></ul><ul><ul><ul><ul><li>sound formats encoding / decoding </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Kerberos authentication </li></ul></ul></ul></ul><ul><ul><li>command-line tools </li></ul></ul><ul><ul><ul><ul><li>appletviewer, javaws, jstack, native2ascii, serialver, etc. </li></ul></ul></ul></ul>See: http://kaffe.org/~stuart/japi Our original goal - Java SE platform 5 API completeness October 2008
  25. 25. <ul><li>Harmony suites: </li></ul><ul><ul><li>Class library unit tests </li></ul></ul><ul><ul><li>DRL VM unit and regression tests </li></ul></ul><ul><ul><li>Functional test suite </li></ul></ul><ul><ul><li>JDKTools unit tests </li></ul></ul><ul><ul><li>Reliability test suite </li></ul></ul><ul><ul><li>Stress test suite </li></ul></ul><ul><ul><li>VM Validation test suite </li></ul></ul><ul><li>Application scenarios: </li></ul><ul><ul><li>Apache Ant scenario </li></ul></ul><ul><ul><li>Apache Axis2 client-server scenario </li></ul></ul><ul><ul><li>Apache Struts scenario </li></ul></ul><ul><ul><li>Apache Tomcat scenario </li></ul></ul><ul><ul><li>Dacapo benchmark </li></ul></ul><ul><ul><li>Eclipse hello world application </li></ul></ul><ul><ul><li>Eclipse Geronimo application </li></ul></ul><ul><ul><li>JEdit scenario </li></ul></ul><ul><ul><li>Jetty scenario </li></ul></ul><ul><ul><li>LDAP scenario </li></ul></ul><ul><ul><li>Scimark benchmark </li></ul></ul>testing, bug fixing, metrics, ... Code quality <ul><li>External test suites: </li></ul><ul><ul><li>Apache Ant 1.7.0 unit tests </li></ul></ul><ul><ul><li>Apache Maven unit tests </li></ul></ul><ul><ul><li>Eclipse unit tests. </li></ul></ul><ul><ul><li>Eclipse TPTP tests </li></ul></ul><ul><ul><li>Geronimo unit tests. </li></ul></ul><ul><ul><li>JUnit tests </li></ul></ul><ul><ul><li>Mauve test suite </li></ul></ul>
  26. 26. Performance Blackburn, S. M., et al The DaCapo Benchmarks: Java Benchmarking Development and Analysis , OOPSLA '06: Proceedings of the 21st annual ACM SIGPLAN conference on Object-Oriented Programming, Systems, Languages, and Applications , (Portland, OR, USA, October 22-26, 2006) Source: http://www.dacapobench.org 2008-11-01 dacapo-2006-10-MR2 Tracked by DaCapo performance benchmarks Key: worst result
  27. 27. <ul><li>Community builds, tests, and reports results on : </li></ul><ul><ul><li>x86 Windows 2000, Windows XP </li></ul></ul><ul><ul><li>x86 Debian, SUSE9, RHEL AS 4, Ubuntu 6 </li></ul></ul><ul><ul><li>x86_64 SUSE 9, RHEL AS 4, Ubuntu 6 </li></ul></ul><ul><li>Known work in progress on : </li></ul><ul><ul><li>Itanium Linux and Windows </li></ul></ul><ul><ul><li>PPC 32bit / 64bit Linux and AIX </li></ul></ul><ul><ul><li>x86 Mac OS X and FreeBSD </li></ul></ul><ul><ul><li>zSeries 31bit / 64 bit zLinux and zOS </li></ul></ul><ul><li>Interested in other platforms? Us too! </li></ul>Multi-platform support Write once, debug everywhere!
  28. 28. DEMO <ul><li>Harmony in action </li></ul>
  29. 29. Agenda <ul><li>Project History </li></ul><ul><li>Development Model </li></ul><ul><li>Code and Runtime Modularity </li></ul><ul><li>How Are We Doing? </li></ul><ul><li>Demo </li></ul><ul><li>Harmony and the JCK </li></ul><ul><li>Wrap-up </li></ul>
  30. 30. Apache’s JCK issue with Sun <ul><li>First request for Java SE JCK v5 - August 2006 </li></ul><ul><ul><li>As per changes to agreement in JSPA, the “Apache Compromise” </li></ul></ul><ul><li>Sun offer JCK with FoU restrictions – 3Q2006 / 1Q2007 </li></ul><ul><ul><li>Limits how users can use compliant software, ASF reject as not allowing distribution under ALv2. </li></ul></ul><ul><li>Open Letter from ASF to Sun – April 2007 </li></ul><ul><ul><li>Appeal directly to Schwartz gets no response. </li></ul></ul><ul><li>ASF position on the JCP – August 2007 </li></ul><ul><ul><li>The ASF will vote “no” on any proposal involving: </li></ul></ul><ul><ul><ul><li>JCP lead who is not complying with the JSPA </li></ul></ul></ul><ul><ul><ul><li>JSRs where the TCK-license contains FoU restrictions </li></ul></ul></ul><ul><li>Java ME & SE/EE Executive Committees </li></ul><ul><ul><li>Originally raised with EC back in February 2007 </li></ul></ul><ul><ul><li>In August 2007 Apache raised a motion… </li></ul></ul>History ©speakuplibrarian Used with permission +
  31. 31. What do the JCP ECs think about it? <ul><li>The chair reported the results of an electronic vote on a motion proposed by Apache and seconded by Intel addressing the use of Field Of Use clauses in TCK licenses. The text of the motion was: </li></ul><ul><li>&quot;TCK licenses must not be used to discriminate against or restrict compatible implementations of Java specifications by including field of use restrictions on the tested implementations or otherwise.  Licenses containing such limitations do not meet the requirements of the JSPA, the agreement under which the JCP operates, and violate the expectations of the Java community that JCP specs can be openly implemented.&quot; </li></ul><ul><li>The results were: </li></ul><ul><li>ME EC: 13 in favor, 1 against, 2 ineligible to vote SE/EE EC: 13 in favor, 1 against, 2 ineligible to vote </li></ul><ul><li>The Chair informed the EC that the PMO will take no further action on this motion. </li></ul>http://jcp.org/aboutJava/communityprocess/summaries/2007/September07-summary.html
  32. 32. Apache’s JCK issue with Sun <ul><li>Current position </li></ul><ul><ul><li>Sun have offered new terms for the JCK, that are still unacceptable. </li></ul></ul><ul><ul><li>Contains FoU limitations on ASF’s usage of the JCK (rather than downstream user restrictions). </li></ul></ul><ul><ul><ul><ul><li>This would allow us to redistribute under ALv2. </li></ul></ul></ul></ul><ul><ul><li>However, terms contain the requirement for an ‘official’ notice from Sun regarding IP notice and certification requirements – effectively a de facto augmentation of the Apache License, and therefore unacceptable. </li></ul></ul>Apache does not make statements regarding intellectual property other than facts we explicitly know to be true and are thereby disclosed in Notice files and the Apache License. +
  33. 33. Agenda <ul><li>Project History </li></ul><ul><li>Development Model </li></ul><ul><li>Code and Runtime Modularity </li></ul><ul><li>How Are We Doing? </li></ul><ul><li>Demo </li></ul><ul><li>Harmony and the JCK </li></ul><ul><li>Wrap-up </li></ul>
  34. 34. Summary <ul><li>We invested time up-front getting the IP infrastructure right. </li></ul><ul><li>Rapidly approaching a full compatible, open-source, implementation of Java SE. </li></ul><ul><li>The core code has been picked up and used in non-trivial commercial runtimes. </li></ul><ul><li>Our modularity story has given us flexibility and robustness to progress quickly and maintain stability. </li></ul><ul><li>Harmony is a fountain of innovation in Java technology. </li></ul>The best is yet to come... ©mary_gaston22 flickr.com
  35. 35. Active areas of effort <ul><li>Class library providers completion </li></ul><ul><li>VM performance work </li></ul><ul><li>Porting to new operating systems </li></ul><ul><li>Improved test case coverage </li></ul><ul><li>Real-world application testing </li></ul><ul><li>Community growth </li></ul>What we still need to do … help appreciated Apache Harmony
  36. 36. Q&A http://harmony.apache.org Tim Ellison [email_address]
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.