Samsung Test Criteria
Upcoming SlideShare
Loading in...5
×
 

Samsung Test Criteria

on

  • 1,234 views

 

Statistics

Views

Total Views
1,234
Views on SlideShare
1,234
Embed Views
0

Actions

Likes
0
Downloads
24
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Samsung Test Criteria Samsung Test Criteria Document Transcript

  • Samsung Test Criteria  2008.08.29 , Version 1.2 Copyright 2008 © Samsung Electronics Co. Ltd. Samsung Test Criteria 1
  • Contents Introduction ................................................................................................................ 3 Device Manufacturer Capabilities for Samsung Phones ............................................ 3 Summary of Test Criteria............................................................................................ 3 Tests for Applications requiring Manufacturer-approved Capabilities ......................... 4 STC-01 : Samsung Values .................................................................................................................. 4 STC-02 : SIS File Naming ................................................................................................................... 5 STC-03 : Installation into the Correct Subfolder.................................................................................. 6 STC-04*: Reserving MIME types on Installation or Startup ................................................................7 STC-05*: Memory Card Removal/Insertion while auto-starting an application Test ...........................8 STC-06 : Installation of an Application with Sensitive Capabilities......................................................9 STC-07*: Themes..............................................................................................................................10 STC-08 : Profiles ............................................................................................................................... 11 STC-09 : GSM Offline Profile ............................................................................................................12 STC-10* : WLAN Connectivity...........................................................................................................13 STC-11* : Application Interaction with WLAN Browsing ....................................................................14 STC-12* : Multilingual Applications ...................................................................................................15 STC-13* : Application functions according to Spec. ..........................................................................16 STC-14* : Backlight Usage................................................................................................................17 STC-15* : Camera Handling..............................................................................................................18 STC-16*: Music Player Handling.......................................................................................................19 STC-17* : DRM-Protected Content – Sending..................................................................................20 STC-18* : DRM-Protected Content – Block File Transfer .................................................................21 STC–19*: DRM-Protected Content - Backup/Restore ......................................................................22 STC-20*: DRM-Protected Content - Multimedia Playback................................................................23 STC–21* : UI and Application key .....................................................................................................24 Typical Application Errors ......................................................................................... 25 Glossary ................................................................................................................... 26 Test Cases marked with an asterisk (*) are identical to the standard S60 tests. Samsung Test Criteria 2
  • Introduction This document describes the test criteria against which applications will be tested when Device Manufacturer capabilities are requested from Samsung. It is also intended to guide developers using other Symbian Signed signing options for Series 60 applications that are intended to run on Samsung handsets. For background reference on Signing and Symbian Signed, visit //https://www.symbiansigned.com/ and view the guides that are available there. The document applies primarily to C++ developers creating native S60 applications, but is also relevant to developers creating applications using other languages, for example Python. This document does not apply to developers creating Java applications for S60 phones. Device Manufacturer Capabilities for Samsung Phones Samsung will grant Device Manufacturer capabilities to applications that: demonstrate a genuine requirement for the requested capabilities; and meet the Test Criteria defined in this document. For details of the request process, visit http://innovator.samsungmobile.com and select Support > Signing. Summary of Test Criteria Samsung test cases are prefixed STC (Samsung-specific Test Cases). In practice, they form a superset of the S60 test criteria. To meet the Samsung test criteria, applications should: Comply with the Symbian OS Test Criteria, available at https://developer.symbian.com/wiki/display/pub/Symbian+Signed+Test+Criteria; Comply with the Symbian OS Test Criteria, available at http://www.forum.nokia.com/document/Nokia_Test_Criteria_for_Symbian_Cpp_Applications); Comply with all Samsung-defined tests that replace any of the test exceptions specified in the above links. Comply with any other relevant additional Samsung-defined tests, for example tests to support specific features on particular Samsung models. All developers should be familiar with the style conventions for the S60 UI. Details can be found in the S60 UI Style Guide v1.3, which is available under Application UI and usability, also at http://www.forum.nokia.com/document/Nokia_Test_Criteria_for_Symbian_Cpp_Applications. Samsung Test Criteria 3
  • Tests for Applications requiring Manufacturer-approved Capabilities Applications requiring certification from Samsung must undergo all of the following tests. Those marked with an asterisk (*) are identical to the standard S60 tests. STC-01 : Samsung Values STC-01 : Samsung Values Test Description: Content that requires the granting of Device Manufacturer capabilities is expected to be consistent with Samsung’s philosophy and values, including its commitments to ethical standards and social responsibility. Steps to conduct the test: Expected test result: 1. Start the application. The application does not contain unsuitable content. 2. While using the application, check that there is no unsuitable content. In particular: Content should not depict or encourage extreme or excessive violence or cruelty against humans or animals. Content should not depict the use of weapons, including knives, axes, swords or guns in man-to-man combat. Content should not promote the abuse of alcohol, drugs, or of other substances, gambling for money, stealing, or other socially unacceptable behaviour. Content and applications should respect individual rights and privacy. Notes: Exceptions: None None Pass:____ Fail:____ Pass with exception: ____ Samsung Test Criteria 4
  • STC-02 : SIS File Naming STC-02 : SIS File Naming Test Description: The SIS file name must include the application name and correct version. Steps to conduct the test: Expected test result: 1. Check the name of the application's SIS file and Application's SIS file name is in the format: compare it against the available information in the About box and PKG file for the application. <application name>_<version>_<target platform version>.sis 2. The SIS file name must include the application name and correct version, should follow the The version number should be the same as the one format guidelines, and must be consistent with defined in the About box and PKG file for the the information in the application’s About box. application. The SIS file name should only include a device name if the application depends on specific hardware features on the device. A SIS file name that includes the name of a non- Samsung device will automatically fail this test, since it is assumed that it depends on hardware features not available in Samsung phones. Notes: Exceptions: None None Pass:____ Fail:____ Pass with exception: ____ Samsung Test Criteria 5
  • STC-03 : Installation into the Correct Subfolder STC-03 : Installation into the Correct Subfolder Test Description: The application creates and installs into the specified destination. Steps to conduct the test: Expected test result: 1. Install the application to a device. Either: 2. Check that the application is installed into the The application installs into the folder correct folder, as follows: defined in the application’s app_reg.rss file. If the defined folder does not already If the application installation folder is exist on the device, the application creates defined in the application’s this folder and installs into it. app_reg.rss file, the application should install into the specified location. Or Or: If no location is specified in the The application installs into the default app_reg.rss file, the application should location for installed applications on a given install into the default location on a given handset. handset. Notes: Exceptions: None None Pass:____ Fail:____ Pass with exception: ____ Samsung Test Criteria 6
  • STC-04 * : Reserving MIME types on Installation or Startup STC-04: Reserving MIME types on Installation or Startup Test Description: If the application supports changing MIME type associations, the default value should be No. The default values in the device must be maintained. Steps to conduct the test: Expected test result: 1. Start the application. The default value is No (i.e. no content types are automatically associated with the application). 2. Verify from the application settings that the default value for common content type The user is able to define which content types are associations is No. opened with the application. 3. Verify that the user is able to define which common content types are opened with the application. Notes: Exceptions: None None Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 7
  • STC-05 * : Memory Card Removal/Insertion while auto-starting an application Test STC-05: Memory Card Removal/Insertion while auto-starting an application Test Test Description: The removal or insertion of the memory card while the application is auto-starting shall not cause problems. Steps to conduct the test: Expected test result: Precondition: Application is installed to memory card. Test case A: In all test cases, check that the device switches on without problems and that removing/inserting the 1. Reboot the device after the application has been memory card does not cause anything unexpected installed. (device to reboot itself, jam, etc.). 2. Insert the memory card into the device before typing in the PIN code. Test case B: 1. Reboot the device after the application has been installed. 2. Insert the memory card into the device immediately after typing in the PIN code. Test case C: 1. Reboot the device after the application has been installed. 2. Remove the memory card from the device before typing in the PIN code. Test case D: 1. Reboot the device after the application has been installed. 2. Remove the memory card from the device immediately after typing in the PIN code. Test case E: 1. Reboot the device after the application has been installed. 2. Remove memory card. 3. Reboot the device again. Repeat the same test cases when the application is installed to C: (if supported by the application). Notes: Exceptions: This test applies only if the application supports auto- None start. Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 8
  • STC-06 : Installation of an Application with Sensitive Capabilities STC-06 : Installation of an Application with Sensitive Capabilities Test Description: An application with sensitive capabilities successfully installs. Steps to conduct the test: Expected test result: 1. Verify whether the application requires some of Applications that require TCB and/or DRM and/or the following sensitive capabilities: AllFiles, AllFiles have been certified and can be installed to DRM, or TCB. Samsung devices. 2. Open the PKG file and check which Product ID/Platform ID/Machine UID/Manufacturer ID has been defined. 3. Install the application to the target device(s). Notes: Exceptions: None None Pass:____ Fail:____ Pass with exception: ____ Samsung Test Criteria 9
  • STC-07 *: Themes STC-07: Themes Test Description: The application needs to work correctly with different themes. Steps to conduct the test: Expected test result: 1. Start the application. The application works correctly with different themes without any usability problems (i.e. information is 2. Change the current theme to another one. readable). 3. Use the application and see if the theme causes any problems in the application. 4. Change the theme to another device-specific theme. 5. Test the application for 5 minutes per theme. Notes: Exceptions: More themes to be used in this test are available at None http://www.s60.com/life. Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 10
  • STC-08 : Profiles STC-08 : Profiles Test Description: The application needs to follows the profiles correctly. Steps to conduct the test: Expected test result: 1. Go to Profiles and activate the ‘General’ profile. The application reads and follows the activated profiles. 2. Start the application. Sounds are not played in the ‘Silent’ profile. 3. Check that the application is using its standard sounds. 4. Go to Profiles and activate the ‘Silent’ profile (use the default device settings). 5. Check that no sounds are played. Notes: Exceptions: When sounds are played on an S60 device and a Sounds are played in ‘Silent’ mode if the event is notification is received (e.g. incoming triggered by the user. For example, when the user SMS/MMS/Bluetooth/IrDA), only a beep sound is selects a tone in the Player application, a sound is heard even though the ‘General’ profile is activated played. and a ring tone has been defined for notification. This behaviour is a designed feature in S60 3rd Edition devices. Pass:____ Fail:____ Pass with exception: ____ Samsung Test Criteria 11
  • STC-09 : GSM Offline Profile STC-09 : GSM Offline Profile Test Description: The application follows the ‘GSM Offline’ profile correctly when making connections. Steps to conduct the test: Expected test result: 1. Go to Profiles and activate the ‘GSM Offline’ The application is not able to make GSM/GPRS profile. connections or send an SMS/MMS. 2. Start the application. The application is able to make a WLAN connection but a note is shown to the user before the connection 3. Try to establish a connection (via GSM, GPRS or is made. WLAN) or send an SMS/MMS using the application. All the messages are saved to the Outbox to be sent later. 4. Check that the application handles ‘GSM Offline’ correctly, and that it displays a clear note to the The application indicates to the user that the user that the connection is not available. connection is not possible at the moment due to the ‘GSM Offline’ profile. By default, this also includes Bluetooth connections; however, the user can overwrite the default setting. IR connections are allowed. Notes: Exceptions: None None Pass:____ Fail:____ Pass with exception: ____ Samsung Test Criteria 12
  • STC-10 * : WLAN Connectivity STC-10 : WLAN Connectivity Test Description: The application must be able to make the correct use of WLAN connectivity. Steps to conduct the test: Expected test result: 1. Start the application. The application finds and connects to all available WLAN networks. 2. Check that the application is able to find and connect to all available WLAN networks. The application can handle the loss of WLAN coverage gracefully, i.e. 3. Take the device outside WLAN coverage. The application or device does not crash. 4. Check that the application behaves gracefully after WLAN coverage has been lost. The application or device does not jam. The application can change the connection automatically if the user has defined it. Notes: Exceptions: The test is applicable only to applications that utilise None WLAN connectivity. Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 13
  • STC-11 * : Application Interaction with WLAN Browsing STC-11 : Application Interaction with WLAN Browsing Test Description: The application does not affect the device’s ability to browse over a WLAN. Steps to conduct the test: Expected test result: 1. Start the application. The application runs correctly. 2. Start the Web browser and browse the Web using Connection is made and browsing works correctly. a WLAN connection. The application runs correctly after browsing over the 3. Switch back to the application. WLAN. 4. Switch back to the Web browser. Browsing is ended correctly. 5. End browsing by disconnecting. The application runs correctly after browsing over the WLAN has ended. 6. Switch back to the application. 7. Close the application. Notes: Exceptions: None None Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 14
  • STC-12 * : Multilingual Applications STC-12 : Multilingual Applications Test Description: The application selects the device language during startup. Steps to conduct the test: Expected test result: 1. Start the application. The application selects the device language during startup. 2. Check that the application uses the device language (if available – if not, it needs to use the default language, which is typically English). 3. Change the device language. 4. Start the application. 5. Check that the application uses the device language or the default language. 6. Repeat the test from step 3 until all languages have been tested. Usually the tests need to be done with two or three devices with different language packages. Notes: Exceptions: None None Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 15
  • STC-13 * : Application functions according to Spec. STC-13 : Application functions according to Spec. Test Description: The application must function as defined in the Help, User Guide, and/or functional specification (or other provided document). Steps to conduct the test: Expected test result: 1. Start the application. The application functions as specified in the Help and other supporting documentation. 2. Use all the features of the application. There are no hidden features. 3. Compare the application against the Help and other provided documentation, and verify that the There are no unexpected functions or problems. application behaves as defined. 4. Check that there are no hidden features. 5. Use the application and check whether anything unexpected happens. Notes: Exceptions: Much of the functionality testing will have been carried None out while performing other test cases. Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 16
  • STC-14 * : Backlight Usage STC-14 : Backlight Usage Test Description: The application must not leave the lights on. Steps to conduct the test: Expected test result: 1. Start the application. Lights are turned off after a while when the application is in the foreground. 2. Leave the application on in the foreground and check that the backlight is turned off after a while. Lights are switched on when you switch the application to the background (key press activity), but 3. Move the application to the background. they are turned off after a while when the application is in the background. 4. Switch to the application using a key press. The application is switched back to the foreground. 5. Close the application. The application is closed and lights are turned off after a while. Notes: Exceptions: None This test case can be ignored for applications that by nature require having the lights on, e.g. Maps Route Instructions Video TV Flash Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 17
  • STC-15 * : Camera Handling STC-15 : Camera Handling Test Description: The application must free the camera resource. Steps to conduct the test: Expected test result: 1. Start the application. The camera functionality of the application is activated. 2. Activate the camera functionality in the application. When the application is in the background, camera functionality is released. 3. Switch the application to the background. The system camera can be used. 4. Open the system camera. When the system camera is closed, the camera 5. Close the system camera. resource is released and the application being tested can use it. 6. Bring the application being tested to the foreground. When the application being tested is closed, the camera functionality is released and the system 7. Close the application. camera can be used. 8. Open the system camera. Notes: Exceptions: None None Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 18
  • STC-16 *: Music Player Handling STC-16 : Music Player Handling Test Description: The application must not affect the device’s ability to play music. Steps to conduct the test: Expected test result: 1. Start the application. The music player opens correctly and starts playing music. If the tested application is also a player and 2. Start the system music player and play some has a higher priority, then no disturbance should music. occur to the currently played track, and/or a warning should be displayed regarding the impossibility of 3. Switch back to the application. using the two players simultaneously. 4. Switch to the music player. The application runs correctly and music continues to play in the background. 5. Press the forward button in the music player. The music player functions work as specified. 6. Press the backward button in the music player. The volume button also works when the music player is in the background. 7. Switch back to the application. The music player closes without any notification or 8. Press the volume button. noise. 9. Close the music player. The application closes normally. 10. Close the application. Notes: Exceptions: None None Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 19
  • STC-17 * : DRM-Protected Content – Sending STC-17 : DRM-Protected Content – Sending Test Description: Ensure that DRM-protected content cannot be sent in decrypted form from an S60 system application or via a third-party application. Steps to conduct the test: Expected test result: 1. Download DRM-protected content. Content cannot be sent, or content was sent but cannot be used in the receiving device/computer. 2. Send the content via the following applications: Media, Messages, File Manager, Music Player, RealPlayer. 3. Send the content via the third party application. Notes: Exceptions: This test must be passed by any application that None provides content-sending facilities either as a separate application or as a plug-in to S60 applications. Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 20
  • STC-18 * : DRM-Protected Content – Block File Transfer STC- 18 : DRM-Protected Content – File Transfer Test Description: Ensure that content files handled by S60 system applications via a third-party plug-in, or directly via a third- party application, cannot be transferred in unprotected form. Ensure that content files handled by the third-party application cannot be transferred in unprotected form when accessed from a desktop PC. Steps to conduct the test: Expected test result: 1. Download DRM-protected content to the device’s Content cannot be copied, or if it can be copied, internal memory. content is still encrypted. 2. Copy the downloaded file from the internal memory to a memory card/device hard disk via the following applications: Media, File Manager. 3. Copy the downloaded file from the internal memory to a memory card/device hard disk via the third-party application. 4. Verify that the copied content file is in encrypted form, for example by trying to use it on a desktop PC. 5. Access the downloaded content via the third- party application from the desktop PC. 6. Verify that the accessed content file is in encrypted form, for example by trying to use it on a desktop PC. Notes: Exceptions: This test must be passed by any application that None provides file handling facilities (copying, backup/restore, access from a desktop PC) either as a separate application or as a plug-in to S60 applications. Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 21
  • STC–19 *: DRM-Protected Content - Backup/Restore STC-19 : DRM-Protected Content - Backup/Restore Test Description: A third-party application does not remove protection from content files in backup/restore operations. Steps to conduct the test: Expected test result: 1. Download DRM-protected content to the device’s Content is still encrypted. internal memory. 2. Take a backup using the third-party application. 3. Restore the backup to another device. 4. Verify that the copied content file is in encrypted form, for example by trying to use it on a desktop PC. Notes: Exceptions: This test must be passed by an application that None provides file handling facilities (copying, backup/restore, access from a desktop PC) either as a separate application or as a plug-in to S60 applications. Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 22
  • STC-20 *: DRM-Protected Content - Multimedia Playback STC-20 : DRM-Protected Content - Multimedia Playback Test Description: DRM-protected content cannot be played in S60 system applications via a third-party application or directly via a third-party application. Steps to conduct the test: Expected test result: 1. Download DRM-protected audio and video The content cannot be played. content. 2. Initiate playback in the following applications: Music Player, RealPlayer. 3. Initiate playback in the third-party application. Notes: Exceptions: This test must be passed by any application that None provides multimedia facilities either directly or as a plug-in to S60 applications. Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 23
  • STC–21 * : UI and Application key STC-21 : UI and Application Key Test Description: Ensure that the application is in line with the relevant UI style guide, and that application key functionality is in line with the S60 UI Style Guide. Steps to conduct the test: Expected test result: 1. Start the application. The application should stay in the background and should not launch when the Application key is 2. Move the application to the background and press pressed in the Application shell and ‘Idle’ screen. the Application key to open the Application shell. 3. Press the Application key again and check that the ‘Idle’ screen is displayed. 4. Repeat steps 2 and 3 a few times and check that the application is not launched. Notes: Exceptions: For more UI and user experience guidelines, see None http://wiki.forum.nokia.com/index.php/UI_test_criteria. Pass:____ Fail:____ Pass with exception: ____ * This test is identical to the standard S60 test. Samsung Test Criteria 24
  • Typical Application Errors This section identifies the ten most common reasons for applications failing to pass the required tests for certification: 1. Help is missing. 2. The About box is missing or does not include the required information (developer's name, application name, application version, end- user support contact information). 3. The application has inconsistent or faulty version numbering and naming. 4. The rapid pushing of certain buttons creates difficulties in the application, which leads to crashes or jams. 5. The application's dialogs, interaction, controls, UI layout, and so on, are inconsistent or not in accordance with the style guides. 6. Various difficulties arise when the application is working with low memory (this usually occurs when several applications are running simultaneously). Such situations often end in crashes or jams. 7. The application is unable to choose its destination (alias drive C: or D: or E:). 8. Startup takes too long. 9. The application does not pause during the interruption of operations, for example during an incoming phone call. (Of course, this is not necessary for all types of application.) 10. The application is so large that installation is extremely time-consuming (tens of minutes) or it allocates a gigantic portion of the memory. 11. The application's strings are longer than the allocated spaces, making parts of them invisible to the user. 12. The error notes are incomplete. Samsung Test Criteria 25
  • Glossary Term Definition AllFiles A sensitive capability along with DRM and TCB. DRM Digital Rights Management. A sensitive capability along with AllFiles and TCB. Functional A document that describes how the application is expected to function. It may include Specification Release notes, Help, a User's Guide, and/or a Requirements specification. MMS Multimedia Messaging Service. OS Operating System. Package List (.pkg). This file contains the specification of all files to be integrated by makesis, i.e. the application files as well as the other files (audio, images, XML, etc.) PKG needed by the application. The package list is usually written manually and contains the mapping of local files to the directory path on the destination Symbian device. The S60 user interface is a smartphone platform designed for Symbian OS by Nokia. S60 It supports mobile browsing, MMS, content downloading, as well as a host of PIM (Personal Information Management) and telephony applications. A device software platform for smartphones that Nokia licenses to other mobile S60 platform device manufacturers as a source-code product. Symbian Installation Script (.sis). The makesis reads the .pkg file and includes all SIS files listed. The resulting .sis file is an executable file that provides a standardised user interface for installing any Symbian application onto the particular device. SMS Short Message Service. TCB Trusted Computing Base. A sensitive capability along with DRM and AllFiles. Third party software An application that has been created by a third party company. In this document, all application references are to applications created for Samsung mobile phone platforms. Unique Identification. Each released Symbian platform application needs to have at least one UID number assigned to it. This UID is then written to each file created by UID the application to associate the files with that application. This guarantees that the correct application is used when the file is opened (the Symbian platform does not use ambiguous file extensions for this purpose like other systems). Samsung Test Criteria 26