automatictester

my thoughts on test automation

Integrating TestNG with SBT

TestNG is a great unit test framework. In many work environments, it is highly desirable (or just most natural) to execute tests using build tool. To makes most of TestNG, one needs to utilise its configurability via testng.xml file. However, SBT doesn’t work with testng.xml files out of the box. In this blog post, I will show you how to combine both to work together. That should save you a few moments of your own time. In this tutorial I will create SBT project from scratch, without using any IDE. However, if you want to use IDE of your choice, feel free to do so.

    • Create sample-sbt-project directory
    • In project root directory, create build.sbt file:
      name := "sample-sbt-project"
      version := "1.0"
      scalaVersion := "2.10.4"
      libraryDependencies ++= Seq(
        "org.testng" % "testng" % "6.8.21",
        "org.scalatest" % "scalatest_2.10" % "2.2.4"
      )
    • Download  sbt-launch.jar and copy project root directory
    • Create sbt text file in your project root directory:
      SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
      java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@“
    • In root project directory run  chmod +x sbt  to make it executable
    • If you plan to use IntelliJ, create project/plugins.sbt file:
      addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0”)

      This will enable sbt gen-idea.

    • Create folder structure:
      mkdir -p src/test/java/com/wordpress/automatictester/tests
      mkdir -p src/test/scala
      mkdir -p src/test/resources
    • Create test src/test/java/com/wordpress/automatictester/tests/SampleTestA.java:
      package com.wordpress.automatictester.tests;
      
      import org.testng.annotations.Test;
      
      public class SampleTestA {
      
         @Test
         public void sampleTest() throws InterruptedException {
            System.out.println("Starting " + this.getClass().getName());
            Thread.sleep(5000);
            System.out.println("Finishing " + this.getClass().getName());
         }
      
      }

      You can create more classes like this in the same folder, if you want to see your tests running in parallel.

    • Create src/test/resources/testng.xml:
      <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
      <suite name="Test runner" parallel="classes" thread-count="10">
          <test name="Package with subpackages">
              <packages>
                  <package name="com.wordpress.automatictester.tests.*"/>
              </packages>
          </test>
      </suite>
      
    • Create src/test/scala/TestRunner.scala:
      import org.scalatest.testng.TestNGWrapperSuite
      
      class TestRunner extends TestNGWrapperSuite (
        List("src/test/resources/testng.xml")
      )

      This file will act as a wrapper for xml file.

    • Take care of dependencies and compile your project:
      ./sbt clean compile test:compile
    • Now you can run your test(s) via created Scala wrapper from command line:
      ./sbt "test-only TestRunner"

Timeouts in Selenium Webdriver

If you’re chasing timeout-related issue with your Selenium tests, it’s good to know what types of timeouts you can play with.


Timeouts set within your code

There are 3 timeouts which can be set under driver.manage().timeouts():

  • Selenium implicit waits are managed by implicitlyWait(). This setting will manage how long Selenium waits for element to appear before throwing NoSuchElementException.
  • Those of you who make use of JavascriptExecutor, can be interested in setScriptTimeout(). If script execution takes longer than specified, TimeoutException is thrown.
  • You can specify non-default value for pageLoadTimeout(). This parameter manages wait times for loading page by browser before throwing TimeoutException.

Timeouts set in Selenium Grid environment

If you run your tests in a Grid, there are a few timeouts which can be of your interest. Those are specified as command line parameters when running selenium-server-standalone-.jar:

  • newSessionWaitTimeout, which defaults to no timeout. If specified, new test must be allocated an execution node within that time. If it’s not, an exception is thrown: org.openqa.selenium.WebDriverException: Error forwarding the new session Request timed out waiting for a node to become available.
  • timeout, which specifies how long particular test which started execution can remain idle, before terminating client session. With this timeout, test which hangs without any communication with browser will not utilise execution node for ever. This is the exception you can find in your stack trace: org.openqa.selenium.WebDriverException: Session [] was terminated due to TIMEOUT.
  • browserTimeout, which is considered to be a safety net for timeout above.

Vendor-specific timeouts

If you make use of Sauce Labs or BrowserStack, it’s good to know if there is anything specific you should know about.

Selenium and new OperaChromiumDriver

Three months ago I’ve posted an article about how Opera and Selenium work together. OperaChromiumDriver is still in early beta, and this clearly impacts what can be achieved. Nevertheless I’ve managed to provide basic usage examples so that you can start working with it on your own. First of all, you will need the binaries, which can be downloaded here.

Local WebDriver

New Opera driver is based on ChromeDriver, which will let us do some hacks. Just pretend you are using ChromeDriver and only replace path to ChromeDriver binary with path to OperaChromiumDriver:

package com.wordpress.automatictester.opera.local;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

public class OperaDriverExample {

    private WebDriver driver;

    @BeforeClass
    public void setup() {
        System.setProperty("webdriver.chrome.driver", "/path/to/operadriver");
        driver = new ChromeDriver();
    }

    @Test
    public void testMethod() {
        driver.get("http://www.google.com");
        driver.findElement(By.name("q")).sendKeys("OperaDriver");
        driver.findElement(By.name("q")).submit();
    }

    @AfterClass
    public void quit() {
        driver.quit();
    }
}

RemoteWebDriver

Start OperaDriver:

./operadriver
Starting OperaDriver beta (v 0.1.498e0af382ca0283abd01832a17569991f2ecd52) on port 9515
Only local connections are allowed

Now you can run your test in Opera:

package com.wordpress.automatictester.opera.remote;

import org.openqa.selenium.By;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import java.net.MalformedURLException;
import java.net.URL;

public class OperaDriverExample {

    private RemoteWebDriver driver;

    @BeforeClass
    public void setup() throws MalformedURLException {
        URL hubUrl = new URL("http://localhost:9515");
        DesiredCapabilities capabilities = new DesiredCapabilities();
        driver = new RemoteWebDriver(hubUrl, capabilities);
    }

    @Test
    public void testMethod() {
        driver.get("http://www.google.com");
        driver.findElement(By.name("q")).sendKeys("OperaDriver");
        driver.findElement(By.name("q")).submit();
    }

    @AfterClass
    public void quit() {
        driver.quit();
    }
}

Please note URL format is different from typical RemoteWebDriver setup. DesiredCapabilities don’t need to specify browser.

Service

You can also run ChromeDriverService and do yet another hack – specify path to OperaDriver:

package com.wordpress.automatictester.opera.service;

import org.openqa.selenium.By;
import org.openqa.selenium.chrome.ChromeDriverService;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import java.io.File;
import java.io.IOException;

public class OperaDriverExample {

    private ChromeDriverService service;
    private RemoteWebDriver driver;

    @BeforeClass
    public void setup() throws IOException {
        service = new ChromeDriverService.Builder()
                .usingDriverExecutable(new File("/path/to/operadriver"))
                .usingAnyFreePort()
                .build();
        service.start();

        DesiredCapabilities capabilities = new DesiredCapabilities();
        driver = new RemoteWebDriver(service.getUrl(), capabilities);
    }

    @Test
    public void testMethod() {
        driver.get("http://www.google.com");
        driver.findElement(By.name("q")).sendKeys("OperaDriver");
        driver.findElement(By.name("q")).submit();
    }

    @AfterClass
    public void quit() {
        driver.quit();
        service.stop();
    }
}

Also here DesiredCapabilities are irrelevant, although required by constructor.

Selenium Java bindings 2.44.0, DesiredCapabilities, Platform and Sauce Labs

Those who use Selenium + Java + Sauce Labs combo can run into troubles when upgrading Java bindings from 2.43.1 to 2.44.0. This works fine before the upgrade:

capabilities.setCapability("platform", "Windows 7");

But throws an exception after upgrade:

Exception in thread "main" java.lang.IllegalArgumentException:
   No enum constant org.openqa.selenium.Platform.Windows 7

This is due to implemented matching platform with enum. There is an already fixed Selenium defect for that, but new version of language bindings has not been released yet. Those of you who are affected should revert back from 2.44.0 and wait for subsequent release.

Switching from one external Selenium Grid test execution provider to another

Some companies outsourced their Selenium Grid test execution environments to external companies. It has a number of advantages as well as disadvantages, as I wrote in one of my former posts. Some companies which are using external test environments, consider switching to competition. The question is – is there a competition? Yes, there is. However it is more limited, than you can assume. First of all, you need to be aware of what you are looking for. There are many cross-browser testing solutions available, but most of them are not what you are looking for. If you outsourced Selenium Grid (i.e. hub and nodes), this is what you want to replace. Any online solution offering screenshot capturing of given URL, or providing VMs for manual test execution is not what you are looking for. So, what are the options? At the moment, there are a few I am aware of. Sauce Labs is probably the most popular option. BrowserStack is another one. TestingBot is yet another one. There is one more, Gridlastic, which is based on Amazon EC2. A few to pick from.

Selenium WebDriver and Opera browser

Do you think WebDriver and Opera are going to work together? Of course they are, there is OperaDriver! Really? No, not really… Things are quite complex and there is a lot of miscommunication with regards to WebDriver support for Opera in test automation community. To get better understanding why things got so complex, I’d suggest to get familiar with Opera version history, in particular version 12 and 15. Opera browser was based on Presto engine up to version 12. With next version (not 13, but 15!), Opera switched to Blink engine.

OK, but how does it relate to WebDriver? Well known OperaDriver supports only browser versions 12 and older. You can read more about this OperaDriver (which was renamed to OperaPrestoDriver less than 2 weeks ago) on GitHub. I don’t like making up new names for software which is effectively being discontinued, as this can cause massive information chaos. Anyway, with current Opera version being 25, there is a massive gap!

Less than month ago, OperaChromiumDriver kicked off, which promises hassle-free WebDriver support for Opera 26 and above. This is a brand new project so I’d say it is very important to be realistic about expectations. I wish them best of luck and hope to see Selenium WebDriver & Opera working together soon!

In the meantime, if you try to run your Selenium tests using most recent Opera version and old OperaDriver (aka OperaPrestoDriver), expect  Invalid service list received  error. Don’t go this way, those two are incompatible.

Selenium WebDriver can’t run tests in Safari after upgrade to 7.1

If you (a) do Selenium test automation targeting OS X platform, and (b) recently upgraded your Safari browser to 7.1 version, you must have noticed it no longer works.

Once test starts to execute in a browser, error message is displayed: Unable to establish a connection with the SafariDriver extension

safari_driver

There is already a Selenium issue #7933 for that. No fix so far, however a manual workaround is available:

  • download SafariDriver jar, most recent version can be found in Maven repo, currently this is 2.43.1
  • extract content of the jar: jar xf selenium-safari-driver-2.43.1.jar
  • go to org/openqa/selenium and double-click SafariDriver.safariextz

Problem solved. You will need to do that once on every machine you run your tests in Safari.

What is interesting in this situation, is that there is no clear communication between those who develop the browser, and those who develop browser-specific WebDriver. This is very similar to problems with IE11 once it was released (that problem is already sorted).

Selenium WebDriver and supported browser versions

Selenium is a fantastic project, however its documentation is far from perfect. Information on Selenium website is not always precise and up to date – see Supported Browsers and Platforms and Platforms Supported by Selenium. I will try to sort that for you.

Here are the browsers most frequently used with WebDriver: Firefox, Chrome, Internet Explorer and Safari.

Firefox

My personal favourite browser with plenty of useful add-ons. In theory, supported are following versions: the latest release, the previous release, the latest ESR release and the previous ESR release. In practice, list of supported versions is not clear to me. Once your Firefox was automatically updated, there is very high chance of running into troubles. I’d recommend using current version -1 or -2, disabling auto update feature and manual updates of both language bindings and browser version being used. Some information could be found in language bindings’ change logs. For Java, this can be found here.

Chrome

Supported versions are clearly specified in release notes on Download page. E.g. release notes for ChromeDriver v2.11 can be found here.

Internet Explorer

In theory, all versions from 6 to most recent one are supported, however some additional configuration may be needed. In practice, if you have to support version 8 and below, this will be an uphill struggle. Try to avoid supporting old IE versions as much as possible. Also don’t be too optimistic with regards to support for newly released IE versions. See history of some issue for more details.

Safari

This browser was born for Mac OS X and in the past was available for Windows as well. Currently version for Windows was abandoned and Safari is available only for Apple products. Supported version: 5.1 and above. Apple used to slightly change default behavior of its browser with every new version, so expect the unexpected, e.g. that.

Wait until element contains specified text in Selenium WebDriver

Your goal: wait until element identified by given locator contains expected text.

Assumptions: element may or may not be initially present in DOM, element may or may not have expected text once it’s present in DOM.

Conditions: if element initially is not present, or doesn’t contain expected text, wait up to specified timeout for the condition to be satisfied.

Solution: ExpectedCondition class! Its handy methods should work for you in most cases.

Example:

public boolean isRoadmapLinkDisplayed() {
    return new WebDriverWait(driver, 5).until(ExpectedConditions.
            textToBePresentInElementLocated(ROADMAP_LINK, "Roadmap"));
}

Selenium WebDriver, Grid 2, PhantomJS and Ghost Driver – tutorial

Selenium WebDriver lets you execute your tests in large number of browsers. With PhantomJS and Ghost Driver you can even execute them in headless mode! PhantomJS is a headless WebKit based browser, which you can interact with using JavaScript. Ghost Driver is a PhantomJS-backed JavaScript implemetation of WebDriver. Ghost Driver has been integrated with PhantomJS since PhantomJS 1.8 release. In this post I will show you how to switch from running your Selenium WebDriver tests in a standard browser to headless mode.

Selenium WebDriver

We will use test and SBT project from previous post and modify it to use PhantomJSDriver.

  • Download PhantomJS binary and make sure it is in your PATH
  • Assuming you use SBT as a build tool, add Ghost Driver library to your build.sbt file:
    "com.github.detro.ghostdriver" % "phantomjsdriver" % "1.1.0"
  • Create sample test class:
    package com.wordpress.automatictester.examples;
    
    import org.openqa.selenium.By;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.phantomjs.PhantomJSDriver;
    import org.testng.annotations.AfterClass;
    import org.testng.annotations.BeforeClass;
    import org.testng.annotations.Test;
    
    import static org.hamcrest.CoreMatchers.is;
    import static org.hamcrest.MatcherAssert.assertThat;
    
    public class PhantomJSTest {
    
        private WebDriver driver;
    
        private static final By ABOUT_TAB = By.linkText("About");
        private static final By ROADMAP_LINK = By.xpath("//a[text() = 'Roadmap']");
        private static final By HELP_LINK = By.xpath("//a[text() = 'Help']");
    
        @BeforeClass
        private void initialize() {
            driver = new PhantomJSDriver();
            driver.navigate().to("http://www.seleniumhq.org/");
        }
    
        @Test
        public void verifySomething() {
            driver.findElement(ABOUT_TAB).click();
            assertThat(driver.findElement(ROADMAP_LINK).isDisplayed(), is(true));
            assertThat(driver.findElements(HELP_LINK).isEmpty(), is(true));
        }
    
        @AfterClass
        private void quit() {
            driver.quit();
        }
    }

It’s done!


Selenium Grid 2

For this demo, we will use Selenium Grid 2 with both hub and PhantomJS node running on local machine. You can read more about Selenium Grid 2 in that post.

  • Modify above @BeforeClass method to:
       private void initialize() throws MalformedURLException {
          URL hubUrl = new URL("http://localhost:4444/wd/hub");
          DesiredCapabilities capabilities = new DesiredCapabilities();
          capabilities.setBrowserName("phantomjs");
    
          driver = new RemoteWebDriver(hubUrl, capabilities);
          driver.navigate().to("http://www.seleniumhq.org/");
       }
  • Download Selenium standalone JAR if you don’t have one
  • Start hub:
    java -jar selenium-server-standalone-2.42.2.jar -role hub
  • Start node:
    phantomjs --webdriver=8080 --webdriver-selenium-grid-hub=http://127.0.0.1:4444

Test is ready to be executed now.