Testing Rapidly Changing Applications With Self-Testing Object-Oriented Selenium Infrastructure

WRITINGWELL-ABSTRACTEDSELF-TESTINGAUTOMATIONON TOP OF JELL-O®DAN CUELLARLEAD SOFTWARE ENGINEERZOOSK - QA TEAM

TERMINOLOGYTest or Automation Scripta series of steps and expected outputHook or Elementa means of identifying a piece of UI (think the By class inWebDriver)Controllera program which executes basic commands (think WebDriver orSeleniumRC Server)

PHILOSOPHY• Selenium is a controller program, it exists to perform actions on a platform (web browsers) • Click a link • Read some text• Libraries exist to coordinate these small actions into larger logical actions on a particular platform • Sign In • Send A Message • Get All Error Messages• Interfaces exist to remove the platform from the context of libraries• Automation scripts should coordinate the highest level of abstraction whenever possible to maintain agility and robustness

A BIT ABOUT ZOOSK• Zoosk is a romantic social network with tens of millions of users that operates in 25 languages and in over 70 countries.• Products include: • web site • mobile web site • Facebook app • iPhone app • iPad app • Android app • Adobe Air app• The development pace is rapid, features are churned out in around 2 weeks time (including testing)• Product management is always running experiments and adjusting plans based on data from instrumentation• Developer to quality assurance ratio hovers somewhere around 10:1

SO WHAT DO YOUMEAN BY JELL-O®Jell-O [jel-oh]Site wide A/B tests where ½ of users will see a complete siteredesign and the other ½ will use the site as it was before.Facebook app users can also split between different A and Bdesigns at times.Each A & B may have its own JS, CSS, and business logic to goalong with it.Not to mention that different versions of the API were being usedby the web, Facebook app, and mobile apps.All the while, different features were being enabled and disabledfor different subsets of the A/B test groups

WHAT CAN GO WRONG?• General • Not enough information is retrieved to reproduce the failure• Elements • The type can change (e.g. dropdown becomes a checkbox) • An element can change location • An element can have multiple ways of identifying itself depending on an A/B test• Business Logic • Logic can change for one platform • Logic can change for some users and not others (A/B test) • Logic can change for all platforms

HOW DO WE COMBAT THESEPROBLEMS?• General • Not enough information is retrieved to reproduce the failure • Separate test actions from controller code, add abundant logging, and unify the patterns by which common operations are accomplished (e.g. selecting from dropdown) Don’t access selenium directly in any scripts. Create a wrapper that logs everything and performs basic checks. (e.g. is the choice available on the dropdown) • If libraries are used then you can log just about everything possible every time without having to re-paste all the logging code

HOW DO WE COMBAT THESEPROBLEMS? (CONT.)• Elements • The type can change • Abstracting away all uses of the item into a library • An element can change location • Centralize all hooks into the UI, you can change it once and it will be fixed for all • Using reflection, you can verify that all of your hooks are functional • An element can have multiple ways of identifying itself depending on an A/B test • ID’s are great, but you can also use an XPath Or expression in your centralized hook

HOW DO WE COMBAT THESEPROBLEMS? (CONT.)• Business Logic • Logic can change for one platform • Update the library for that platform • Logic can change for some users and not others (A/B test) • Detect the A/B test in the library • Logic can change for all platforms • Generic Libraries or test scripts which coordinate platform specific libraries are a great place to address this

HOW DO YOU DEALWITH THAT AGAIN?• Separate logical actions From calls to controller (in this case Selenium) code into libraries• Centralize your site model and logic, so that all automation runs through one place. When you have to fix something, you can fix it once for everything.• Write code that can test the test automation. Investigating test script failures can be costly, but if you test code can test itself you will be able to pick out certain kinds of problems easily.• Write platform-agnostic test scripts when possible. This leaves you with one copy of site-logic and test-logic for all platforms. When things change, you’ll be happy its in one place.

SEPARATION OF CONTROLFROM SCRIPTING• An extra layer is great to log information and perform additional actions that Selenium can’t assume it needs to do• We use a layer between that wraps logging, error handling, and some other things on top of most WebDriver commands Test Script •IsSignedIn() •Log(“Retrieving Cookie Auth”); Action •var authCookie = GetCookie(auth); •Log(authCookie); Library •return authCookie != null; •Log (“Getting Cookies”); •var c = Driver.Manage.().Cookies; WebDriver •Log(c); •var a = Driver.Manage().Cookies[“auth”]; Wrapper •Log(“Cookie value: “ + a.ToString()); return a;

CENTRALIZING YOUR SITEMODEL FOR SELF -TESTING• All hooks (means by which we retrieve WebElements) are stored in a single class, namespace, or something similar• Elements are stored lazily; the means to retrieve them are stored, but the actual WebElement is not retrieved until run-time. We use classes that wrap calls to By or methods which will return a WebElements• Hooks are grouped relevantly and each group has private self-test setup methods which will navigate the client (e.g. selenium) to where the hooks exist• Reflection is used to iterate through all of these groups and run the private self-test setup methods and then validate the functionality of the elements• Annotations (Java) or Attributes (C#) are used to exclude elements from the tests

ABOUT INTROSPECTION& REFLECTIONIntrospection (Java) and Reflection (C#) is the process by which acomputer program can observe, do type introspection, and modifyits own structure and behavior at runtime.In the example to follow we store information about how to usevarious hooks (references to pieces of user interface) in the codeso that later at runtime we can use the information to determinehow to test our library of hooks.

CENTRALIZED HOOKS[C#]namespace Automation.Hooks.Web{ public static class LoginPage { public static WebHook EmailField = By.Id(“email”); public static WebHook PasswordField = By.Id(“password”); public static WebHook SubmitButton = By.Id(“submit”); [OmitFromSelfTest] public static WebHook Spinner = By.Xpath(“//div[@class=’wait’]”); private static void _StartSelfTest() { _SelfTestSeleniumInterface.Open(“http://www.zoosk.com”) } }}

SIMPLE SELF-TEST[PseudoCode]// iterate in depth-first-search orderforeach(class c in Automation.Hooks.Web){ if c.hasMethod(“StartSelfTest”) c.executeMethod(“StartSelfTest”); foreach(WebHook h in c) Log.Verify(IsElementPresent(h)); if c.hasMethod(“StopSelfTest”) c.executeMethod(“StopSelfTest”);}

MORE ADVANCEDTECHNIQUES• Subclass WebHook to things like TextBoxHook and have the test check for reading and writing from the text field using Selenium• Do the same for Links, Images, etc. and write custom test methods• Randomize the order of which elements are tested to check for robustness• Add random data to the tests, e.g. type a random string into the text box.• Write custom test methods for structures that require it

CONSOLIDATING AND ABSTRACTINGPRODUCT LEVEL ACTIONS• Product Level Actions (e.g. Sign In To The Site) are stored in libraries, so that there exists one good copy that everyone can use• The libraries implement generic interfaces so the platform can be abstracted away (i.e. the native iPhone app and the website both have libraries with a signup function that conforms to an interface that can be used in any test script)• If the process changes for a particular platform, you can update it in a single place• If the process change across all platforms, you can use platform agnostic (generic) libraries and update the logic in a single place

ABSTRACT LIBRARIES[C#]namespace Automation.Libraries{ public SeleniumActionLibrary WebSignUpLib : ISignUpLib { public void SignIn(string email, string password) { if (IsSignedIn()) return; if (!Driver.Url.contains(“login.php”);Driver.Navigate().GoToUrl(“https://www.zoosk.com/login.php”); Hooks.Web.LoginPage.EmailField.type(email); Hooks.Web.LoginPage.PasswordField.type(password); Hooks.Web.LoginPage.SubmitButton.click(); WaitForElementToNotExist(Hooks.Web.LoginPage.Spinner); } }}

LIBRARY INTERFACE[C#]namespace Automation.Libraries{ interface ISignUpLib { void SignIn(string email, string password); bool IsSignedIn(string email, string password); void SignOut(); }}

PLATFORM AGNOSTICAUTOMATION SCRIPTS• Many tests that need to be run are logically identical across all platforms. • Deactivate an account, attempt to sign in, expect failure • Send a message, verify it shows up in sent items • Take down some web service, verify an error is produced• Implemented these tests at an abstract level allows you to have one test for all platforms • When the logic changes, the logic only exists one place in your automation code

AN ABSTRACTEDTEST[C#]namespace Automation.Tests{ public static class SignInWithDeactivatedAccountTest: Test { public static void Run() { AutomationClient [] clients = new AutomationClient[] { new WebClient(), new iPhoneClient() } var account = APIUtility.Accounts.CreateTestAccount(); APIUtility.Accounts.Deactivate(account); foreach(AutomationClient client in clients) { client.SignUpInterface.SignIn(account.email, acount.password); Log.VerifyFalse(!client.IsSignedIn()); }}}}

CONCLUSIONDevelopers and Designers frequently have to be able to adapt torapidly changing code, automation writers can be just as nimble.If you design and abstract your code well, you can buildsmaller, more robust and agile automation scripts.If anyone is interested in taking up work on bringing iOS supportto Selenium… Call Me.

MORE INFODan CuellarLead Software Engineer – QA Teamdanc@zoosk.com(Email me to retrieve related source code available for sharing)See AlsoSelf-Generating Test Artifacts for Selenium WebDriverTue 12:20PM – Track B – Marcus MerrellBuilding Quality on Foundations of MudTue 2:40PM – Track A – Kristian RosenwoldThe Restless Are Going NativeWed 2:20PM – Track A – Dante Briones

http://lanyrd.com	43
http://microchipatwork.com	11
http://198.199.79.89	5

Testing Rapidly Changing Applications With Self-Testing Object-Oriented Selenium Infrastructure

by seleniumconf on Apr 17, 2012

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 59

Statistics

Testing Rapidly Changing Applications With Self-Testing Object-Oriented Selenium Infrastructure Presentation Transcript