Optimizing Java Device Testing with Automatic Feature Discover y   Fyodor Romanov, Software Engineer
Agenda <ul><li>Java device testing challenges.
Testing framework structure.
Approach how to optimize and shorten device test-cycle using Diagnostic tests to explore device capabilities.
Demo showing Diagnostic tests usage and benefits </li></ul>
Problem description
Problem description <ul><li>Nearly 3 Billion devices run Java ME in the world today.
Before a manufacturer can make a device commercially available, they must perform a significant amount of testing to ensur...
Java Device quality testing helps Java technology wireless device manufacturers and network operators maximize product qua...
Problem description There are a lot of (more then 30) Java Platform, Micro Edition (Java ME platform) technologies (and co...
Problem description <ul><li>A lot of technologies, a lot of tests. More then 11,000 tests are developed currently in JDTS ...
Up to 3 weeks for guru to run all test on one device.
Challenge of managing and, if possible, lowering quality testing costs.
The most challenging task for test execution is to configure tests for particular device.
Express testing: evaluate device quality in 4 hours. </li></ul>
Problem description <ul><li>Not to test technologies which device does not support.  </li></ul><ul><li>Negative false case...
Problem statement Device manufacture knows, but testing house doesn't.
Before Automatic feature discovery.
Solution in nutshell
Solution implementation
Solution implementation Implementation example will be based on  JDTF test framework  however it should be applicable to a...
JDTF overall tests structure JDTF organizes tests into  hierarchical structure s called  test packs ,  a collection of tes...
JDTF testing structure Harness is using tests and different configuration files to execute test on device. Test execution ...
Tests execution workflow
How to distinguish diagnostic tests. JDTF provides keywords  to filter  (subset for execution) tests in  test packs. So tw...
Move them into special dedicated test pack. </li></ul>
Writing diagnostic test. Changes in MicroAgent. Diagnostic  tests find out whether the device supports an optional feature...
Writing diagnostic test <ul><li>Test name should clear indicate what optional feature is tested and placed in appropriate ...
Test should be marked with ‘diagnostic’ keyword
Upcoming SlideShare
Loading in …5
×

S313352 optimizing java device testing with automatic feature discovering

213 views

Published on

Presentation were given at JavaOne 2010

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
213
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • The growing complexity and diversity of devicebswith their varying operating systems, processors, and memory configurationbsincreases the need for thorough testing. At the same time, service providers and manufacturers face the challenge of managing and, if possible, lowering in-house costs. The Java Device testing helps device manufacturers and service providers ensure their reputation for quality, while building customer satisfaction and loyalty.
  • Java ME technology was originally created in order to deal with the constraints associated with building applications for small devices. For this purpose Sun defined the basics for Java ME technology to fit such a limited environment and make it possible to create Java applications running on small devices with limited memory, display and power capacity. Java ME platform is a collection of technologies and specifications that can be combined to construct a complete Java runtime environment specifically to fit the requirements of a particular device or market. The Java ME technology is based on three elements; * a configuration provides the most basic set of libraries and virtual machine capabilities for a broad range of devices, * a profile is a set of APIs that support a narrower range of devices, and * an optional package is a set of technology-specific APIs.
  • Failture analisys becomes complicated because even if a technology is supported, most of them have optional features which also can be unsupported. For example, JSR293 Location API 2.0 declares following optional features: Creating and deleting landmark stores, including private landmark stores Adding, removing and renaming landmark categories Adding and removing LandmarkStoreListener
  • Usually there are no proper available documentation for device and it’s manual and error-prone task to discover what technologies and optional features device supports and update special configuration file (template) accordingly.
  • Identifying what technologies device supports includes writing so-called Diagnostic tests that check particular optional feature by trying to use it and then analyze tests logs. Based on that user should set properties in device configuration file (template). I t is time-consuming to have to read through the log file and then interpret and w rite values generated there into configuration file. T hings that make it cumbersome are : Value key names may not match correctly: example Media Format names like MP4 Values are in different places, i.e. not grouped by feature/function/JSR.
  • Use Diagnostic tests to explore a device’s capabilities, such as : the presence of optional APIs, the presence of JSRs using Class.forName() canvas size, a number of colors ,… etc and report back to the harness. After running the automated diagnostic tests on a device, AUTOMATICALLY create a template that is customized for the device. In the generated template tests that cannot pass because of a missing feature, are filtered out. Using such a template saves testing time by running only tests that are relevant for a device.
  • When configuration is finished and test execution is not started, JDTF harness calls an optional test pack method for each test case. This method inspects the configuration and returns null if the test is relevant or a list of reasons describing why the test is irrelevant. An irrelevant test is one that is pointless to run because o ne or more properties in the configuration, which essentially represents the capabilities of the test device, makes a test irrelevant. For example, a property representing the presence of an optional device capability is set to false.
  • * Represents an entity that can convert device profile properties * to the test pack configuration values. * This interface intended to be implemented by the API clients. * This method maps Device Profile properties to test pack properties which * are put in the device profile template. * @param diagnosticsProperties a map which binds a test. pack id with a map * consisting of device profile properties and their values. * @return a map of test pack properties and their values.
  • Demo includes running JDTF with Automatic Feature Discovery feature implemented, running couple of diagnostic tests, generating template and demonstrating that appropriate tests are filtered out with proper reason.
  • S313352 optimizing java device testing with automatic feature discovering

    1. 2. Optimizing Java Device Testing with Automatic Feature Discover y Fyodor Romanov, Software Engineer
    2. 3. Agenda <ul><li>Java device testing challenges.
    3. 4. Testing framework structure.
    4. 5. Approach how to optimize and shorten device test-cycle using Diagnostic tests to explore device capabilities.
    5. 6. Demo showing Diagnostic tests usage and benefits </li></ul>
    6. 7. Problem description
    7. 8. Problem description <ul><li>Nearly 3 Billion devices run Java ME in the world today.
    8. 9. Before a manufacturer can make a device commercially available, they must perform a significant amount of testing to ensure a high-quality, reliable product that delivers customer satisfaction.
    9. 10. Java Device quality testing helps Java technology wireless device manufacturers and network operators maximize product quality and minimize time to market. </li></ul>
    10. 11. Problem description There are a lot of (more then 30) Java Platform, Micro Edition (Java ME platform) technologies (and combinations):
    11. 12. Problem description <ul><li>A lot of technologies, a lot of tests. More then 11,000 tests are developed currently in JDTS and number is growing.
    12. 13. Up to 3 weeks for guru to run all test on one device.
    13. 14. Challenge of managing and, if possible, lowering quality testing costs.
    14. 15. The most challenging task for test execution is to configure tests for particular device.
    15. 16. Express testing: evaluate device quality in 4 hours. </li></ul>
    16. 17. Problem description <ul><li>Not to test technologies which device does not support. </li></ul><ul><li>Negative false cases complicate failure analysis </li></ul>
    17. 18. Problem statement Device manufacture knows, but testing house doesn't.
    18. 19. Before Automatic feature discovery.
    19. 20. Solution in nutshell
    20. 21. Solution implementation
    21. 22. Solution implementation Implementation example will be based on JDTF test framework however it should be applicable to any Java device testing framework. The Java Device Test Framework (JDTF) is a test framework based on Oracle‘ s commercial Java Device Test Suite (JDTS) product. JDTF is a general purpose, fully-featured, flexible, and configurable test framework suited to assess various aspects of Java Platform, Micro Edition (Java ME) device implementation quality .
    22. 23. JDTF overall tests structure JDTF organizes tests into hierarchical structure s called test packs , a collection of tests that are functionally related and have common setup requirements :
    23. 24. JDTF testing structure Harness is using tests and different configuration files to execute test on device. Test execution on device is controlled by MicroAgent which starts/stops test and communicates with Harness (sends results, logs, etc.)
    24. 25. Tests execution workflow
    25. 26. How to distinguish diagnostic tests. JDTF provides keywords to filter (subset for execution) tests in test packs. So two options available to distinguish diagnostic tests: <ul><li>Mark them with special keyword, for example ‘diagnostic’
    26. 27. Move them into special dedicated test pack. </li></ul>
    27. 28. Writing diagnostic test. Changes in MicroAgent. Diagnostic tests find out whether the device supports an optional feature and export a property by using Logger class as follows. public class Logger { ... public void exportProperty(String name, String value) { LogRecord rec = new LogRecord (...); rec.setProperty(EXPORTED_PROPERTY_PREFIX + name, value); ... } }
    28. 29. Writing diagnostic test <ul><li>Test name should clear indicate what optional feature is tested and placed in appropriate package, for example: ConnectionTest.testHTTP
    29. 30. Test should be marked with ‘diagnostic’ keyword
    30. 31. Test should try to use optional feature and call Logger.exportProperty(String name, String value)
    31. 32. If feature is supported, return PASSED and if feature is not supported return FAILED. This helps to visualize test result.
    32. 33. Do nothing else </li></ul>
    33. 34. Writing diagnostic test. Code Sample. public class ConnectionTest implements AutomatedTest { ... public boolean testHTTP() { Connection c = null; try { ... c = Connector.open(url); logger.exportProperty(&quot;HTTPSupport&quot;, &quot;true&quot;); return true; } catch (IllegalArgumentException iae) { return false; } ... }
    34. 35. Developing Template Generator
    35. 36. Adding Template Builders to TestPacks
    36. 37. Changes in FW
    37. 38. Final touch. Adjusting Relevance Filters Relevance filter should filter out not relevant test providing right reason. Code sample: public String LapiDependencyProvider.isRelevant(final TestId id) throws TestDependencyProviderException { final boolean createDeleteLmsSupported = getBoolean(TestPackProperty.CreatingAndDeletingLandma rkStoresSupported, id); if (!createDeleteLmsSupported) { if (fqn.endsWith(&quot;#addLandmarkScenario2&quot;) || fqn.endsWith(&quot;#deleteLandmarkFromAnotherLMS&quot;) { return &quot;Test not relevant because Landmark Store create/delete operations configured to be not supported.&quot;; } }
    38. 39. Automatic Feature Discovery Efficiency. <ul><li>Proved to be very valuable.
    39. 40. Reduces 1 - 2 days work for one person (depending of person's experience) to 30 minutes.
    40. 41. Before Automatic Feature Discovery feature customers were taking a complete unknown device and painstakingly reading logs to find out what it supports . </li></ul>
    41. 42. DEMO
    42. 43. Useful links <ul><li>JDTF Project home page https://jdtf.dev.java.net with source code
    43. 44. JDTS home page with API detailed description
    44. 45. http://www.oracle.com/technetwork/java/javadevice-140362.html </li></ul>
    45. 46. Session k ey points <ul><li>How Diagnostic tests can help in Java </li></ul>device quality testing <ul><li>Tips how to develop such tests
    46. 47. How to process Diagnostic tests output in testing framework </li></ul>JDTF test framework was used for demonstration <Insert Picture Here>
    47. 48. Q&A Q&A

    ×