automatictester

my thoughts on test automation

REST-assured Tutorial

In this short post I’ll show you how to create your very first tests with REST-assured. REST-assured is a Java library which provides extremely easy to use DSL for testing REST services. We will use online API, JSONPlaceholder.

In this ultra simple tutorial, we will use only two files: Maven pom.xml and Java test class.

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>uk.co.automatictester</groupId>
    <artifactId>restassured-tutorial</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>

        <dependency>
            <groupId>com.jayway.restassured</groupId>
            <artifactId>rest-assured</artifactId>
            <version>2.8.0</version>
            <scope>test</scope>
        </dependency>

    </dependencies>

</project>

Test class

import com.jayway.restassured.builder.RequestSpecBuilder;
import com.jayway.restassured.builder.ResponseSpecBuilder;
import com.jayway.restassured.specification.RequestSpecification;
import com.jayway.restassured.specification.ResponseSpecification;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import static com.jayway.restassured.RestAssured.given;
import static org.hamcrest.CoreMatchers.equalTo;

public class PostsTests {

    RequestSpecification reqSpec;
    ResponseSpecification respSpec;

    @BeforeClass
    public void setupSpec() {
        reqSpec = new RequestSpecBuilder().setBaseUri("http://jsonplaceholder.typicode.com").build();
        respSpec = new ResponseSpecBuilder().expectStatusCode(200).build();
    }

    @Test
    public void verifyId() {
        given().spec(reqSpec)
                .when().get("/posts/5")
                .then().spec(respSpec)
                .body("id", equalTo(5));
    }

    @Test
    public void verifyUserId() {
        given().spec(reqSpec)
                .when().get("/posts/5")
                .then().spec(respSpec)
                .body("userId", equalTo(1));
    }

    @Test
    public void verifyTitle() {
        given().spec(reqSpec)
                .when().get("/posts/5")
                .then().spec(respSpec)
                .body("title", equalTo("nesciunt quas odio"));
    }

}

In above example we use RequestSpecBuilder to set base URI for all requests. We also use ResponseSpecBuilder to set expected status code, which is also common for all requests.

Our tests focus on verification of response body of a following request:
http://jsonplaceholder.typicode.com/posts/5

Monitoring JMeter resource utilisation

In order to prevent JMeter to be a bottleneck itself and impact performance test results, it’s good to know what resources does it consume during test execution. I found it useful to monitor JMeter’s JVM parameter with jconsole.

To start jconsole, run:

jconsole

Select process you want to monitor:

jconsole_connection

This should give you access to useful metrics, including memory and CPU utilisation, number of threads etc.

jconsole

Distributed testing with JMeter

Once you move on from prototyping your performance tests to actual performance test execution, single JMeter instance may quickly become the bottleneck. In this tutorial I’ll show you how to configure distributed testing.

Infrastructure:

  • 192.168.56.1 – JMeter client (machine which controls test execution)
  • 192.168.56.101 – JMeter server I (machine which executes tests and send results back to the client)
  • 192.168.56.102 – JMeter server II (same as above)

All above machines are dual-homed: have 2 network interfaces. Once is used for accessing internet, another for communication between the nodes. In this scenario we don’t make use of VPN, proxy and don’t need to set up SSH port forwarding.

Client configuration

In jmeter.properties, set the following:

remote_hosts=192.168.56.101,192.168.56.102
client.rmi.localport=1234

Server configuration

In jmeter.properties, set the following:

client.rmi.localport=1234
server.rmi.localport=54321

In jmeter-server, set the following:

RMI_HOST_DEF=-Djava.rmi.server.hostname=192.168.56.101

Do the same for 2nd server, but specify 192.168.56.102 IP address. You may want to use different port numbers. As long as ports you pick are not in use and client-side configuration matches server-side configuration, this is fine.

Test Execution

Start JMeter server on both nodes by running ./jmeter-server.

In the above steps, you configured bi-directional communication between client and servers. Once this is done, you are ready do kick off test execution with Run / Remote Start All in JMeter GUI. Alternatively, you can run your tests in non-GUI mode.

Basic CucumberJVM + Selenium WebDriver test automation framework

Below you can find complete, working test automation framework, which can be used to support BDD in your organisation. It bundles together CucumberJVM and Selenium WebDriver, with TestNG behind the scenes. Key features and design concerns:

  • Build with Maven
  • TestNG is used to enable parallel execution on runner level
  • Feature and runner is duplicated, to demonstrate tests (i.e. runners) can run in parallel
  • Every new feature must be associated with runner to run with mvn test
  • Cucumber features are imperative (in real world this probably will not be the case)
  • Framework design is oversimplified, to focus solely on CucumberJVM/Selenium/TestNG integration
  • Advanced configurability is not implemented (see above)
  • Reporting is not fine tuned

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>uk.co.automatictester</groupId>
    <artifactId>jwebfwk</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>jwebfwk</name>

    <build>

        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.18.1</version>
                <configuration>
                    <suiteXmlFiles>
                        <suiteXmlFile>src/test/resources/testng/testng.xml</suiteXmlFile>
                    </suiteXmlFiles>
                </configuration>
            </plugin>

        </plugins>

    </build>

    <dependencies>

        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>18.0</version>
        </dependency>

        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>2.46.0</version>
        </dependency>

        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.8.21</version>
        </dependency>

        <dependency>
            <groupId>info.cukes</groupId>
            <artifactId>cucumber-testng</artifactId>
            <version>1.2.2</version>
        </dependency>

        <dependency>
            <groupId>info.cukes</groupId>
            <artifactId>cucumber-java</artifactId>
            <version>1.1.7</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.hamcrest</groupId>
            <artifactId>hamcrest-all</artifactId>
            <version>1.3</version>
        </dependency>

    </dependencies>

</project>

src/test/resources/testng/testng.xml

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Test runner" parallel="classes" thread-count="2">
    <test name="Package with subpackages">
        <packages>
            <package name="uk.co.automatictester.jwebfwk.runners.*"/>
        </packages>
    </test>
</suite>

src/test/resources/features/Download.feature

Feature: Download section
  In order to use Selenium in my project, I want to download Selenium language bindings

  Scenario: Java binding download link check
    Given I am on the Selenium homepage
    When I click "Download" tab
    Then I should see "Java" download link on Download page
    And I should see "C#" download link on Download page
    And I should see "Ruby" download link on Download page
    And I should see "Python" download link on Download page
    And I should see "Javascript (Node)" download link on Download page

src/test/resources/features/Download_Copy.feature

Feature: Download section
  In order to use Selenium in my project, I want to download Selenium language bindings

  Scenario: Java binding download link check
    Given I am on the Selenium homepage
    When I click "Download" tab
    Then I should see "Java" download link on Download page
    And I should see "C#" download link on Download page
    And I should see "Ruby" download link on Download page
    And I should see "Python" download link on Download page
    And I should see "Javascript (Node)" download link on Download page

src/test/java/uk/co/automatictester/jwebfwk/runners/DownloadFeatureRunner.java

package uk.co.automatictester.jwebfwk.runners;

import cucumber.api.CucumberOptions;
import cucumber.api.testng.AbstractTestNGCucumberTests;

@CucumberOptions(features = "src/test/resources/features/Download.feature",
        glue = "uk.co.automatictester.jwebfwk.glue",
        format = {"pretty"})
public class DownloadFeatureRunner extends AbstractTestNGCucumberTests {
}

src/test/java/uk/co/automatictester/jwebfwk/runners/DownloadFeatureRunner_Copy.java

package uk.co.automatictester.jwebfwk.runners;

import cucumber.api.CucumberOptions;
import cucumber.api.testng.AbstractTestNGCucumberTests;

@CucumberOptions(features = "src/test/resources/features/Download_Copy.feature",
        glue = "uk.co.automatictester.jwebfwk.glue",
        format = {"pretty"})
public class DownloadFeatureRunner_Copy extends AbstractTestNGCucumberTests {
}

src/test/java/uk/co/automatictester/jwebfwk/page/objects/DownloadPage.java

package uk.co.automatictester.jwebfwk.page.objects;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import uk.co.automatictester.jwebfwk.framework.ParentPage;

public class DownloadPage extends ParentPage {

    private static final String DOWNLOAD_LINK = "//div[@id='mainContent']//table[1]//tbody//td[text()='%s']//..//td[4]//a[text()='Download']";

    public DownloadPage(WebDriver driver) {
        super(driver);
    }

    public boolean hasDownloadLinkFor(String linkText) {
        By downloadLinkLocator = By.xpath(String.format(DOWNLOAD_LINK, linkText));
        return hasElement(downloadLinkLocator);
    }

}

src/test/java/uk/co/automatictester/jwebfwk/page/objects/MainPage.java

package uk.co.automatictester.jwebfwk.page.objects;

import org.openqa.selenium.WebDriver;
import uk.co.automatictester.jwebfwk.framework.ParentPage;

public class MainPage extends ParentPage {

    public MainPage(WebDriver driver) {
        super(driver);
    }

    public void clickTab(String tab) {
        click(tab);
    }

}

src/test/java/uk/co/automatictester/jwebfwk/glue/StepDefinitions.java

package uk.co.automatictester.jwebfwk.glue;

import cucumber.api.java.After;
import cucumber.api.java.Before;
import cucumber.api.java.en.Given;
import cucumber.api.java.en.Then;
import cucumber.api.java.en.When;
import uk.co.automatictester.jwebfwk.framework.ParentScenario;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.core.Is.is;

public class StepDefinitions extends ParentScenario {

    @Before
    public void beforeScenario() {
        startBrowser();
    }

    @Given("^I am on the Selenium homepage$")
    public void I_am_on_the_Selenium_homepage() {
        navigateTo();
    }

    @When("^I click \"([^\"]*)\" tab$")
    public void I_click_tab_on(String tab) {
        mainPage.clickTab(tab);
    }

    @Then("^I should see \"([^\"]*)\" download link on Download page$")
    public void I_should_see_download_link_on_download_page(String linkText) {
        assertThat(downloadPage.hasDownloadLinkFor(linkText), is(true));
    }

    @After
    public void afterScenario() {
        closeBrowser();
    }
}

src/test/java/uk/co/automatictester/jwebfwk/framework/ParentScenario.java

package uk.co.automatictester.jwebfwk.framework;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import uk.co.automatictester.jwebfwk.page.objects.DownloadPage;
import uk.co.automatictester.jwebfwk.page.objects.MainPage;

import java.util.concurrent.TimeUnit;

public class ParentScenario {

    private WebDriver driver;

    protected DownloadPage downloadPage;
    protected MainPage mainPage;

    protected void startBrowser() {

        driver = new FirefoxDriver();
        driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);

        downloadPage = new DownloadPage(driver);
        mainPage = new MainPage(driver);
    }

    protected void navigateTo() {
        driver.navigate().to("http://docs.seleniumhq.org/");
    }

    protected void closeBrowser() {
        driver.quit();
    }

}

src/test/java/uk/co/automatictester/jwebfwk/framework/ParentPage.java

package uk.co.automatictester.jwebfwk.framework;

import org.openqa.selenium.WebDriver;

public abstract class ParentPage extends DSL {

    public ParentPage(WebDriver driver) {
        super(driver);
    }

}

src/test/java/uk/co/automatictester/jwebfwk/framework/DSL.java

package uk.co.automatictester.jwebfwk.framework;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;

public abstract class DSL {

    private WebDriver driver;

    public DSL(WebDriver driver) {
        this.driver = driver;
    }

    public void click(String text) {
        click(By.linkText(text));
    }

    public void click(By by) {
        driver.findElement(by).click();
    }

    public boolean hasElement(By by) {
        return !driver.findElements(by).isEmpty();
    }

}

Your mvn test output should be similar to:

-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestSuite
Feature: Download section
In order to use Selenium in my project, I want to download Selenium language bindings
Feature: Download section
In order to use Selenium in my project, I want to download Selenium language bindings
Starting ChromeDriver (v2.9.248307) on port 46253
Starting ChromeDriver (v2.9.248307) on port 7664

Scenario: Java binding download link check                            # src/test/resources/features/Download.feature:4
Given I am on the Selenium homepage                                 # StepDefinitions.I_am_on_the_Selenium_homepage()
When I click “Download” tab                                         # StepDefinitions.I_click_tab_on(String)
Then I should see “Java” download link on Download page             # StepDefinitions.I_should_see_download_link_on_download_page(String)
And I should see “C#” download link on Download page                # StepDefinitions.I_should_see_download_link_on_download_page(String)
And I should see “Ruby” download link on Download page              # StepDefinitions.I_should_see_download_link_on_download_page(String)
And I should see “Python” download link on Download page            # StepDefinitions.I_should_see_download_link_on_download_page(String)
And I should see “Javascript (Node)” download link on Download page # StepDefinitions.I_should_see_download_link_on_download_page(String)

1 Scenarios (1 passed)
7 Steps (7 passed)
0m6.882s

Scenario: Java binding download link check                            # src/test/resources/features/Download_Copy.feature:4
Given I am on the Selenium homepage                                 # StepDefinitions.I_am_on_the_Selenium_homepage()
When I click “Download” tab                                         # StepDefinitions.I_click_tab_on(String)
Then I should see “Java” download link on Download page             # StepDefinitions.I_should_see_download_link_on_download_page(String)
And I should see “C#” download link on Download page                # StepDefinitions.I_should_see_download_link_on_download_page(String)
And I should see “Ruby” download link on Download page              # StepDefinitions.I_should_see_download_link_on_download_page(String)
And I should see “Python” download link on Download page            # StepDefinitions.I_should_see_download_link_on_download_page(String)
And I should see “Javascript (Node)” download link on Download page # StepDefinitions.I_should_see_download_link_on_download_page(String)

1 Scenarios (1 passed)
7 Steps (7 passed)
0m7.340s

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.189 sec - in TestSuite

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.780 s
[INFO] Finished at: 2015-06-11T12:03:30+02:00
[INFO] Final Memory: 7M/81M
[INFO] ------------------------------------------------------------------------

If copy & paste doesn’t work for you in JMeter on OS X

If copy & paste doesn’t work for you in JMeter running on OS X, check if you didn’t corrupt .properties files in <jmeter_dir>/bin folder. If you override those with command line parameters, ensure you override the right ones. It is important to load certain files with certain command line switches, as described by ./jmeter.sh -help output. If you mix them, you can face strange issues.

Anyway, if copy & paste still doesn’t work for you, check not only CMD+C/V, but also CTRL+C/V and right-click menu. There is a chance one of them will be working for you 🙂

(for those who are not familiar with OS X: in most cases CMD button is used instead of CTRL on this platform)

Empty TestPlan or error reading test plan – see log file

JMeter was recently installed on new machine, and it was unable to open certain JMX files:

Empty TestPlan or error reading test plan – see log file

If that’s what have just happened to you, do what the message says – se the log file. In my case that JMX file was utilising additional JMeter plugins which were installed on other machines, but weren’t installed on the new one. Copying extracted content to <jmeter_dir>/lib/ext folder sorted the problem.

SSLHandshakeException while recording JMeter scripts

When I recently tried to record JMeter tests for web app front end, I came across SSLHandshakeException. Page was perfectly accessible for web browser using default network settings, but browser couldn’t get through while proxied via JMeter’s HTTP(S) Test Script Recorder.

Investigation took a while. Key thing to understand this problem is to be aware every web browser supports a particular list of ciphers which can be used when establishing secure connection with web app server. There must be some intersection between ciphers supported by browser and required by web server. If there isn’t any, secure connection will not be established. This will make your application inaccessible.

When you proxy web browser requests via JMeter test script recorder, this is now up to JMeter (or actually Java installed on your local machine) to provide cipher supported by web server. Ciphers supported by web browser are no longer important, as this is now JMeter proxy which establishes connection with remote web server. That’s why application could work for me in web browser, but not when proxied through JMeter.

If you encounter SSLHandshakeException, first upgrade Java version on machine you try to record your tests. It can fix the problem. If it won’t, list of ciphers supported by your Java version can be determined using this simple utility. Talk to your WebOps Engineer, compare it with web app server config (it was Nginx in my case) and amend configuration.

JMeter – org.apache.http.NoHttpResponseException: the target server failed to respond

I’ve been executing same set of tests using JMeter 2.12 against same test environment, but from different locations. Depending on network location (and network latency), I found a number of tests to fail without a good reason, with following exception:

org.apache.http.NoHttpResponseException: the target server failed to respond

I tried a couple of tricks and finally found a solution. In HTTP Request Defaults, I selected HttpClient4. In user.properties, I set httpclient4.idletimeout=120. This finally got rid of network-dependent execution errors.

Parallel execution on method level in Selenium + TestNG framework

Most Selenium test automation frameworks based on TestNG implement parallel test execution on class level. This can be easily achieved and is implicitly supported by default for Page Object design pattern class hierarchy:

traditional-page-object-framework-class-hierarchy

You can find example of such design in one of my previous posts – for both pure Page Object and PageFactory design patterns. This default solution works for most of us as long as we don’t have to restart browser after every test method. If we do, we can end up with artificially large number of classes with single test methods.

To enable parallelism per test method, not class, we need to redesign class hierarchy. Test classes can’t extend ParentTest. We need an App object, witch which each test method interacts separately. Below is complete example of such design:

ParentPage

package uk.co.automatictester.session.per.method;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;

public class ParentPage {

    protected WebDriver driver;

    public ParentPage(WebDriver driver) {
        this.driver = driver;
    }

    public void click(By locator) {
        driver.findElement(locator).click();
    }

    public boolean isDisplayed(By locator) {
        return !driver.findElements(locator).isEmpty();
    }

}

MainPage

package uk.co.automatictester.session.per.method;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;

public class MainPage extends ParentPage {

    private static final By ABOUT_TAB = By.linkText("About");

    public MainPage(WebDriver driver) {
        super(driver);
    }

    public void clickAboutTab() {
        click(ABOUT_TAB);
    }
}

AboutPage

package uk.co.automatictester.session.per.method;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;

public class AboutPage extends ParentPage {

    private static final By ROADMAP_LINK = By.linkText("Roadmap");
    private static final By HELP_LINK = By.linkText("Help");

    public AboutPage(WebDriver driver) {
        super(driver);
    }

    public boolean isRoadmapLinkDisplayed() {
        return isDisplayed(ROADMAP_LINK);
    }

    public boolean isHelpLinkDisplayed() {
        return isDisplayed(HELP_LINK);
    }
}

App

package uk.co.automatictester.session.per.method;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

public class App {

    protected WebDriver driver;

    protected MainPage mainPage;
    protected AboutPage aboutPage;

    public void startBrowser() {
        driver = new ChromeDriver();

        mainPage = new MainPage(driver);
        aboutPage = new AboutPage(driver);

        driver.navigate().to("http://www.seleniumhq.org/");
        driver.manage().window().maximize();
    }

    public void quit() {
        driver.quit();
    }
}

AboutTest

package uk.co.automatictester.session.per.method;

import org.testng.annotations.Test;

import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.MatcherAssert.assertThat;

public class AboutTest {

    @Test
    public void verifyAboutPageRoadmapLink() {
        App app = new App();
        app.startBrowser();

        app.mainPage.clickAboutTab();
        assertThat(app.aboutPage.isRoadmapLinkDisplayed(), is(true));

        app.quit();
    }

    @Test
    public void verifyAboutPageHelpLink() {
        App app = new App();
        app.startBrowser();

        app.mainPage.clickAboutTab();
        assertThat(app.aboutPage.isHelpLinkDisplayed(), is(false));

        app.quit();
    }
}

testng.xml

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Test runner" parallel="methods" thread-count="2">
    <test name="Thread per method">
        <packages>
            <package name="uk.co.automatictester.session.per.method.*"/>
        </packages>
    </test>
</suite>

Integrating TestNG with Maven

This article is similar to the one on SBT and TestNG integration. If you are new to Maven and/or Maven + TestNG integration, it can save you a good few moments.

  • Set up Maven project. You can do it with your favourite IDE or just use Maven archetype:
    mvn -B archetype:generate \
       -DgroupId=com.wordpress.automatictester.projects.maven \
       -DartifactId=sample-maven-project

    (It is assumed you have Maven set up correctly on your machine)

  • Get rid of automatically created app and test classes
  • Remove JUnit dependency from pom.xml file, add TestNG dependency instead:
    <dependencies>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.8.21</version>
        </dependency>
    </dependencies>
  • Create src/test/resources/testng.xml:
    <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
    <suite name="Test runner" parallel="classes" thread-count="10">
        <test name="Package with subpackages">
            <packages>
                <package name="com.wordpress.automatictester.projects.maven.*"/>
            </packages>
        </test>
    </suite>
  • Add TestNG configuration to pom.xml:
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.18.1</version>
                <configuration>
                    <suiteXmlFiles>
                        <suiteXmlFile>${filename}</suiteXmlFile>
                    </suiteXmlFiles>
                </configuration>
            </plugin>
        </plugins>
    </build>
  • Create src/test/java/com/wordpress/automatictester/tests/SampleTestA.class:
    package com.wordpress.automatictester.projects.maven;
    
    import org.testng.annotations.Test;
    
    public class SampleTestA {
    
        @Test
        public void sampleTest() throws InterruptedException {
            System.out.println("Starting " + this.getClass().getName());
            Thread.sleep(5000);
            System.out.println("Finishing " + this.getClass().getName());
        }
    
    }

    You can create more sample tests, if you want to see how they run in parallel.

  • If you are using Intellij, add TestNG to classpath as suggested by IDE.
  • Now you should be ready to run your tests with Maven:
    mvn test -Dfilename=src/test/resources/testng.xml

    You should now see your test(s) passing.