automatictester

my thoughts on test automation

SBT project in IntelliJ IDEA for Selenium WebDriver testing – tutorial

In this step-by-step tutorial you will learn how to create brand new project in IntelliJ IDEA with SBT as a build tool. This project will contain WebDriver, TestNG and Hamcrest libraries, which you will use to create Selenium tests. It is assumed you have prior Selenium knowledge and worked with IDE before. No prior experiences with SBT are required. This tutorial is based on IntelliJ IDEA 13.1.4 Community Edition and SBT 0.13.5, on OS X platform.

  • In IntelliJ, install Scala plugin and restart IDE
  • Create New Project / Scala / SBT
  • In Preferences / SBT, make sure you selected  Use auto-import  option
  • Add following dependencies to  build.sbt  file:
    libraryDependencies ++= Seq(
       "org.seleniumhq.selenium" % "selenium-java" % "2.42.2",
       "org.testng" % "testng" % "6.8.8",
       "org.hamcrest" % "hamcrest-all" % "1.3"
    )
  • Wait for IntelliJ to rebuild the project
  • Create your sample test:
    package com.wordpress.automatictester.examples;
    
    import org.openqa.selenium.By;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.firefox.FirefoxDriver;
    import org.testng.annotations.AfterTest;
    import org.testng.annotations.BeforeTest;
    import org.testng.annotations.Test;
    
    import static org.hamcrest.CoreMatchers.is;
    import static org.hamcrest.MatcherAssert.assertThat;
    
    public class FirefoxTest {
    
        private WebDriver driver;
    
        private static final By ABOUT_TAB = By.linkText("About");
        private static final By ROADMAP_LINK = By.xpath("//a[text() = 'Roadmap']");
        private static final By HELP_LINK = By.xpath("//a[text() = 'Help']");
    
        @BeforeClass
        private void initialize() {
            driver = new FirefoxDriver();
            driver.navigate().to("http://www.seleniumhq.org/");
            driver.manage().window().maximize();
        }
    
        @Test
        public void verifySomething() {
            driver.findElement(ABOUT_TAB).click();
            assertThat(driver.findElement(ROADMAP_LINK).isDisplayed(), is(true));
            assertThat(driver.findElements(HELP_LINK).isEmpty(), is(true));
        }
    
        @AfterClass
        private void quit() {
            driver.quit();
        }
    }
  • If everything went fine, you should now be able to right-click class name and select  Run ‘FirefoxTest’  to start your test

That’s it, SBT-backed project is ready to go!


If you need more control over the build process, you can get SBT executable and work with command line.

  • Download  sbt-launch.jar
  • Copy this JAR to your project root directory
  • Create sbt text file in your project root directory with following content:
    SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
    java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@"
  • In root project directory run  chmod +x sbt  to make it executable
  • Add following plugin to  plugins.sbt  file:
    addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")

    This will enable sbt gen-idea

  • Go to project root directory and run:
    ./sbt clean compile test:compile gen-idea

This will take care of dependencies and regenerate IntelliJ project. You should be prompted to reload project when you get back to IntelliJ.

Selenium PageFactory design pattern

According to some people, PageFactory design pattern helps to reduce boilerplate code from Page Objects. I don’t use PageFactory and never thought my Page Objects contain boilerplate code, however I decided to make some research: I’ve created a couple of classes using Page Object design pattern, and then rewritten them using PageFactory.

There is a single, fundamental difference between Page Objects and PageFactory: first pattern relies on By locators to identify WebElements, while second makes direct use of WebElements. This have a number of implications which will be discussed below.

ParentPage – Page Objects

package com.wordpress.automatictester.framework.classic.core;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;

public class ParentPage {

    protected WebDriver driver;

    public ParentPage(WebDriver driver) {
        this.driver = driver;
    }

    public void click(By locator) {
        driver.findElement(locator).click();
    }

    public boolean isDisplayed(By locator) {
        return !driver.findElements(locator).isEmpty();
    }

}

Parent Page Object contains a number of wrapper methods for page interaction. They are defined in parent class and are available through inheritance in child all classes, enforcing code reuse. For isDisplayed() method, we use approach suggested in WebDriver Javadoc: “findElement should not be used to look for non-present elements, use findElements(By) and assert zero length response instead”.

ParentPage – PageFactory

package com.wordpress.automatictester.framework.pagefactory.core;

import org.openqa.selenium.NoSuchElementException;
import org.openqa.selenium.WebElement;

public class ParentPage {

    public void click(WebElement element) {
        element.click();
    }

    public boolean isDisplayed(WebElement element) {
        try {
            return element.isDisplayed();
        } catch (NoSuchElementException e) {
            return false;
        }
    }
}

With PageFactory, we don’t need to access driver to find element, as element is already defined (see classes below). For that reason we don’t also need explicit constructor and driver initialization. However, as we don’t have an access to the driver, we can’t execute findElements() method, and therefore can’t use preferred approach to verify if element is displayed. There is a workaround for that, we can use a list of WebElements (both in page object class and in parent page class):

public boolean isDisplayed(List elements) {
    return elements.size() > 0;
}

However, in that case we may need to keep two locators of same element (e.g. WebElement and list of WebElements) when we need to perform some actions on that element and check if it is displayed. Of course we can keep only the list and access its first element to perform any actions besides verification if it is displayed, but this is a workaround for workaround and has nothing to do with removal of boilerplate code, which was supposed to be primary benefit of using PageFactory pattern.

In total: some code wasn’t needed, while some other was necessary.

ParentTest – Page Objects

package com.wordpress.automatictester.framework.classic.core;

import com.wordpress.automatictester.framework.classic.pages.AboutPage;
import com.wordpress.automatictester.framework.classic.pages.MainPage;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;

public class ParentTest {

    protected WebDriver driver;

    protected MainPage mainPage;
    protected AboutPage aboutPage;

    public void initializeTest() {
        driver = new FirefoxDriver();

        mainPage = new MainPage(driver);
        aboutPage = new AboutPage(driver);

        driver.navigate().to("http://www.seleniumhq.org/");
        driver.manage().window().maximize();
    }

    public void quit() {
        driver.quit();
    }
}

ParentTest – PageFactory

package com.wordpress.automatictester.framework.pagefactory.core;

import com.wordpress.automatictester.framework.pagefactory.pages.AboutPage;
import com.wordpress.automatictester.framework.pagefactory.pages.MainPage;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.support.PageFactory;

public class ParentTest {

    protected WebDriver driver;

    protected MainPage mainPage;
    protected AboutPage aboutPage;

    public void initializeTest() {
        driver = new FirefoxDriver();

        mainPage = PageFactory.initElements(driver, MainPage.class);
        aboutPage = PageFactory.initElements(driver, AboutPage.class);

        driver.navigate().to("http://www.seleniumhq.org/");
        driver.manage().window().maximize();
    }

    public void quit() {
        driver.quit();
    }
}

Almost no difference, except how we initialize page object instances.

MainPage – Page Objects

package com.wordpress.automatictester.framework.classic.pages;

import com.wordpress.automatictester.framework.classic.core.ParentPage;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;

public class MainPage extends ParentPage {

    private static final By ABOUT_TAB = By.linkText("About");

    public MainPage(WebDriver driver) {
        super(driver);
    }

    public void clickAboutTab() {
        click(ABOUT_TAB);
    }
}

MainPage – PageFactory

package com.wordpress.automatictester.framework.pagefactory.pages;

import com.wordpress.automatictester.framework.pagefactory.core.ParentPage;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.FindBy;

public class MainPage extends ParentPage {

    @FindBy(linkText = "About")
    private static WebElement aboutTab;

    public void clickAboutTab() {
        click(aboutTab);
    }
}

If we follow PageFactory pattern, we don’t need to handle driver at all, which is good. On the other side, element locators are usually coded in two lines, because of annotations. It’s hard to say we achived boilerplate code reduction here.

AboutPage – Page Objects

package com.wordpress.automatictester.framework.classic.pages;

import com.wordpress.automatictester.framework.classic.core.ParentPage;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;

public class AboutPage extends ParentPage {

    private static final By ROADMAP_LINK = By.linkText("Roadmap");
    private static final By HELP_LINK = By.linkText("Help");

    public AboutPage(WebDriver driver) {
        super(driver);
    }

    public boolean isRoadmapLinkDisplayed() {
        return isDisplayed(ROADMAP_LINK);
    }

    public boolean isHelpLinkDisplayed() {
        return isDisplayed(HELP_LINK);
    }
}

AboutPage – PageFactory

package com.wordpress.automatictester.framework.pagefactory.pages;

import com.wordpress.automatictester.framework.pagefactory.core.ParentPage;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.FindBy;

public class AboutPage extends ParentPage {

    @FindBy(linkText = "Roadmap")
    private static WebElement roadMapLink;

    @FindBy(linkText = "Help")
    private static WebElement helpLink;

    public boolean isRoadmapLinkDisplayed() {
        return isDisplayed(roadMapLink);
    }

    public boolean isHelpLinkDisplayed() {
        return isDisplayed(helpLink);
    }
}

Same here.

AboutTest – Page Objects

package com.wordpress.automatictester.framework.classic.tests;

import com.wordpress.automatictester.framework.classic.core.ParentTest;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.MatcherAssert.assertThat;

public class AboutTest extends ParentTest {

    @BeforeClass
    public void start() {
        super.initializeTest();
    }

    @Test
    public void verifyAboutPageLinks() {
        mainPage.clickAboutTab();
        assertThat(aboutPage.isRoadmapLinkDisplayed(), is(true));
        assertThat(aboutPage.isHelpLinkDisplayed(), is(false));
    }

    @AfterClass
    public void quit() {
        super.quit();
    }
}

AboutTest – PageFactory

package com.wordpress.automatictester.framework.pagefactory.tests;

import com.wordpress.automatictester.framework.pagefactory.core.ParentTest;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.MatcherAssert.assertThat;

public class AboutTest extends ParentTest {

    @BeforeClass
    public void start() {
        super.initializeTest();
    }

    @Test
    public void verifyAboutPageLinks() {
        mainPage.clickAboutTab();
        assertThat(aboutPage.isRoadmapLinkDisplayed(), is(true));
        assertThat(aboutPage.isHelpLinkDisplayed(), is(false));
    }

    @AfterClass
    public void quit() {
        super.quit();
    }
}

In case of test class, there is no difference between those two approaches.

Other PageFactory implications

There are also some other implications of not working with driver directly. With Page Objects, you can access sub-element of particular element:

driver.findElement(By.id("someId")).findElement(By.id("someOtherId"));

With PageFactory you can’t – or actually you can, but then you no longer make use of PageFactory, as you need to explicitly call By locator and follow Page Objects pattern:

someWebElement.findElement(By.id("someId"));

Conclusion

PageFactory design pattern was an attempt to resolve problem of boilerplate code of Page Objects. However, as shown in above examples, PageFactory can reduce number of lines of code in one place, but increase in another.

Weakness of PageFactory is that it relies on WebElements, while Page Objects design pattern relies on By locators. If you have By locator, you can always get a WebElement. If you have WebElement, you can’t get its By locator. In more advanced scenarios this may lead to storing both WebElements and By locators in your code.

TestNG and unresolved dependencies

TestNG gives you the power to manage dependencies between tests in your test set. However, with great power comes great responsibility. Let’s consider two different examples.

In first example, there is an unresolved dependency defined using dependsOnMethods:

package com.wordpress.automatictester.tests.dependencies;

import org.testng.annotations.Test;

public class DependentTests {
   @Test()
   public void d1() {}

   @Test(dependsOnMethods = "d0")
   public void d2() {}
}

When you attempt to run test set with such test, an exception will be thrown and none of your tests will execute:

org.testng.TestNGException:
com.wordpress.automatictester.tests.dependencies.DependentTests.d2() depends on nonexistent method d0

Similar story for unresolved dependency defined using dependsOnGroups:

package com.wordpress.automatictester.tests.dependencies;

import org.testng.annotations.Test;

public class DependentTests {
   @Test(dependsOnGroups = "NONEXISTENT")
   public void d3() {}
}

Again, when you attempt to run test set with such test, an exception will be thrown and none of your tests will execute:

org.testng.TestNGException:
DependencyMap::Method "DependentTests.d3()[pri:0, instance:com.wordpress.automatictester.tests.dependencies.DependentTests@65693ccc]" depends on nonexistent group "NONEXISTENT"

You will never get a compilation error because of unresolved TestNG dependencies, only exception will be thrown when you attempt to execute your test set. There is no built-in TestNG mechanism to verify if test set is runnable or not, without actually running it.

TestNG limitations: dependsOnGroups/Methods and DataProviders working together

TestNG is great: with  DataProviders  you can do proper Data-Driven Testing, and with  dependsOnGroups / dependsOnMethods  you can manage dependencies within your test set. However, cooperation between DataProviders and dependencies is not brilliant.
Let’s consider an example. Test method  testMeUsingDataProvier()  can utilize multiple test data sets provided by DataProvider. Test group  MULTIPLE_INVOCATION_TEST  can be assigned to that test method, and other tests may depend on this group. However, there is no option to assign test group to single test method invocation, per single test data set provided by DataProvider. In other words, if  testMeUsingDataProvier()  is executed 20 times with different data sets from DataProvider and there are also other test methods which depend on  MULTIPLE_INVOCATION_TEST  group, all invocations of  testMeUsingDataProvier()  must pass or none of dependent tests will kick off.
In that case usage of DataProviders coupled with test dependencies to populate system under test with test data before actual test executes may not be a sound option. In most cases you’d prefer to run all the tests you can, and omit only those for which test data couldn’t be generated. But this is not possible, as you cannot assign separate test groups in DataProvider on per-record basis. I look forward to seeing this limitation bypassed in future TestNG release.

Handling SSL certificate errors in Selenium

Handling SSL certificate errors with Selenium is easy, that’s true. Handling SSL certificate errors with Selenium is no easy, that’s also true. Everything depends on particular case. Generally speaking, if you run your tests only in Firefox or Chrome, setting it up shouldn’t be too daunting. Things are getting more complicated when you need to support also other browsers, as interaction between lines of code utilising Selenium libraries in your tests and particular browsers is not consistent. If you manage VMs you use to run Selenium tests and can install additional certificates, then you are almost there. However, in many cases you can’t. Many people advise not to use SSL at all while doing test automation, this includes, but is not limited to Selenium. Agreed, switching to pure HTTP should help get rid of additional complexity, but this is often also not possible due to the context of what you are doing. All in all, proper solution will depend on the browser you run your tests in. In case of IE which displays warning page, this page is actually an HTML page. Widely accepted workaround is to use JavascriptExecutor to click one of the links on this page. This will add some browser-specific code to your framework which you usually should try to avoid, but this time that can be a must. Things are not so simple for Safari, which in case of SSL errors displays popup, and there is no popup handling in SafariDriver. Unlike IE, there is no trivial workaround for this problem in that browser. Don’t forget to check this link if above solutions were not satisfactory for you.

Internal or external test execution environment?

For many organizations, outsourcing Selenium WebDriver test execution environment can be a sound option. Before you decide whether to maintain your test execution environment in-house or use external provider, you should answer yourself a couple of questions. Some of them are not trivial, however do it carefully if you want to avoid further challenges.

  • Can you provide public URL for your test environment? If no, does your external supplier provide proxy which can be used to access application running within your company’s intranet? Does this tool have any known limitations?
  • Heavy client-side processing with JavaScript is becoming more and more common practice in web development. If your application makes extensive use of JavaScript, compare specifications of in-house and externally supplied VMs. How do they refer to recommended hardware specification for OS that they run? Comparing JavaScript benchmark results can be a good option.
  • Do you need to execute your tests on desktop editions of Windows OS? See this post.
  • Do you plan to access your application over HTTPS? If so, can your external provider support mechanisms to handle possible issues with SSL certificates – e.g. installation of custom certificates on execution VMs or disabling warnings using proxy? Currently WebDriver is completely unable to handle SSL alert pop ups in Safari. Workaround for IE exists, however this is still a workaround.
  • Do you need your test execution VMs to run in particular time zone? If so, can you manage that with your external provider?
  • Do you need run your tests from specific IP address, e.g. due to IP whitelisting? If so, can your external provider supply you with such static address? Is it just a single address, a couple of them, or entire range?
  • Do you need to modify default registry settings on execution VMs? WebDriver support for IE11 is based on registry setting. Can this be done by your external provider?
  • Check if external provider can offer support during your working hours. Time zone difference can be challenging.

All in all, switching to external provider can be a good choice, if planned carefully.

Switching from Safari 6 to 7 and phishing site warnings

WebDriver users: always be careful when upgrading browser version which you use in your automated tests. When I was researching upgrade from Safari 6 to 7, I came across something that I didn’t even think of: HTTP basic authentication credentials included in application URL caused Safari 7 to display a phishing warning, which wasn’t displayed by Safari 6.

safari_phishing

This caused all the tests to fail, and there was no simple workaround, as SafariDriver doesn’t support handling browser pop ups. If you have same problem, double-check if you really need to store HTTP authentication credentials in your application URL – perhaps they can be safely removed.

External Test Execution Environment Providers and Windows Desktop OS

When you decide whether to build test execution environment in-house or use external test execution environment provider, you take some things for granted. If they offer you Windows 7 or 8, you assume it is Windows 7 or 8. But is it? Let’s have a look at some of the most popular providers – Sauce Labs and BrowserStack.

Here you can find a list of platforms provided by Sauce Labs. Please scroll to the bottom and see this: “For licensing reasons, Sauce Labs uses server editions of Windows which correspond closely to the desktop editions” followed by exact list of counterparts.

Here you can find similar information about BrowserStack: “The license agreement of Windows Desktop editions (XP, 7, 8, 8.1) doesn’t allow us to provide remote access to our customers. Therefore we use Windows Server edition with Desktop Experience, and use the versions of Internet Explorer made for Windows Desktop editions“.

From my experience, same IE version running on Windows desktop OS and its server counterpart are in 99% of cases alike. However, there is also 1% of cases where they differ.

It is good to be aware of what your tests actually run on.

Selecting Tests To Be Run Via testng.xml File

TestNG tests are usually started with testng.xml file. Below you can find examples how to select tests to be executed.

Tests from package:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Test runner">
    <test name="Package without subpackages">
        <packages>
            <package name="com.wordpress.automatictester.tests"/>
        </packages>
    </test>
</suite>

Tests from package and all its subpackages:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Test runner">
    <test name="Package with subpackages">
        <packages>
            <package name="com.wordpress.automatictester.tests.*"/>
        </packages>
    </test>
</suite>

Tests from classes:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Test runner">
    <test name="Classes">
        <classes>
            <class name="com.wordpress.automatictester.tests.login.LoginTests"/>
            <class name="com.wordpress.automatictester.tests.api.ApiTests"/>
        </classes>
    </test>
</suite>

Test methods from class:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Test runner">
    <test name="Methods">
        <classes>
            <class name="com.wordpress.automatictester.tests.api.ApiTests">
                <methods>
                    <include name="apiTestA"/>
                    <include name="apiTestB"/>
                </methods>
            </class>
        </classes>
    </test>
</suite>

Tests from package and all its sub-packages, but excluding those which belong to specified group:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Test runner">
    <test name="Groups - exclude">
        <groups>
            <run>
                <exclude name="AUXILIARY"/>
            </run>
        </groups>
        <packages>
            <package name="com.wordpress.automatictester.tests.*"/>
        </packages>
    </test>
</suite>

Tests from package and all its sub-packages, but only those which belong to specified group:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Test runner">
    <test name="Groups - include">
        <groups>
            <run>
                <include name="AUXILIARY"/>
            </run>
        </groups>
        <packages>
            <package name="com.wordpress.automatictester.tests.*"/>
        </packages>
    </test>
</suite>

IE8 and nth-child() CSS Selector

If you need to support IE8, using nth-child() CSS selector is not an option – it is not supported by IE8.

For example, this locator won’t work in IE8:

table#timeTable tbody tr:nth-child(4) td:nth-child(3)

nth-child-ie8

Happily, in that case you can achieve the same with XPath:

//table[@id='timeTable']/tbody/tr[4]/td[3]

If for some reasons you want to use CSS selectors, there is an ugly workaround with first-child and adjacent sibling selectors:

table#timeTable tbody tr:first-child + tr + tr + tr td:first-child + td + td

It works in IE8. I hope this post will save you time spent on investigations!