Jun 10, 2013

Multiple test configurations with TestNG

Multiple test configurations with TestNG

I use TestNG for automated testing as it has a number of features that JUnit doesn’t have. One of its strengths is the power and flexibility of configuring tests and supplying parameters to them, for which there are two methods:
  1. Using the testng.xml file to define both the list of tests to run and parameters.
  2. Using data providers within your test code to load parameters.
Using the testng.xml file is very simple, but not as powerful as using data providers. However, suppose you’d like to run the same set of tests with different parameters, but without having to write data providers – how could you achieve that?
One mechanism is to allow test writers to specify parameters in two files which are merged by Ant into a single testng.xml file before running the tests:
  • A testng-default.xml file with the list of tests and default parameters.
  • A .properties to supply overrides or additional parameters.
Merging files like this is a task very suited to a scripting language. Here is an implementation in groovy:
println "Merging testng-default.xml with .properties file into testng.xml"

if (args.size() != 3) {
    println "Script requires three arguments:"
    println "Path to testng template file"
    println "Path to .properties file"
    println "Path for saving merged testng file"
}

String testNGTemplate = args[0]
String propertiesFile = args[1]
String mergedTestNGFile = args[2]

// get the properties
File propsFile = new File(propertiesFile)
FileInputStream input = new FileInputStream(propsFile)
Properties props = new Properties()
props.load(input)

// Start copying the testng file
File oldTestNG = new File(testNGTemplate)
File newTestNG = new File(mergedTestNGFile)

// Get the set of parameters. Read in using Groovy XML parser.
Node xml = new XmlParser().parse(oldTestNG)
NodeList parameterList = xml.test.parameters.parameter
// iterate over parameter list and replace any that have been overridden
parameterList.each{p ->
    println p.@name
    if (props.keySet().contains(p.@name)) {
        println "Replacing ${p.@name} with value from .properties file"
        // update the value to the one set in .properties
        p.@value = props.get(p.@name)
    }
}

// Now add any properties that weren't listed in the original
// testng-default.xml file
Set<String> parameterNames = xml.test.parameters.parameter.@name
Node parameters = xml.test.parameters.getAt(0)

props.each{ key, value ->
  if (!parameterNames.contains(key)) {
    println "Adding property from .properties. Key: ${key}, value: ${value}"
    parameters.appendNode("parameter",[name: "${key}", value: "${value}"])
  }    
}

// Write out the file
new XmlNodePrinter(new PrintWriter(newTestNG)).print(xml)
Note that this script is written assuming that all of your test classes are part of a single “test”, with the parameters inside that test, like the following:
<suite name="regression-tests">
    <test name="regression-basic">
        <parameters>
            <parameter name="param1" value="value1"/>
            <parameter name="param2" value="value2"/>
            <parameter name="param3" value="value3"/>
        </parameters>
        <classes>
            <class name="uk.co.somecorp.Test1" />
            <class name="uk.co.somecorp.Test2" />      
            <class name="uk.co.somecorp.Test3" />
        </classes>
    </test>
</suite>
This is reflected in the lines in the Groovy script that navigate to the parameters, such as:
NodeList parameterList = xml.test.parameters.parameter
Obviously if you want to organise your default testng file differently you’ll need to adjust these parts of the script.
You can invoke the script from Ant with:
<taskdef name="groovy" classname="org.codehaus.groovy.ant.Groovy">
   <classpath>
      <fileset file="/path_to_groovy/groovy-all.jar"/>
   </classpath>
</taskdef>    

<!-- Target to merge the testng-default.xml and .properties files -->
  <target name="merge-files">
     <groovy src="/path_to_script/MergeTestNGAndProperties.groovy">
        <arg value="/some_path/testng-default.xml"/>
        <arg value="/some_path/my.properties"/>
        <arg value="/output_path/testng.xml"/>
     </groovy>
</target>
 
 
Resoure:
 

Automating Selenium testing with TestNG Ant and CruiseControl

Automating Selenium testing with TestNG, Ant and CruiseControl

If you are using Selenium for web testing, most likely you’ll want to make your tests as automated as possible so that they can be run automatically on a regular basis. You can do this by using a few additional technologies:

1. Integrate with a Java test framework

For any serious testing, your tests will need to be written in Java, rather than the basic Selenium html format. You can save the tests in Java from the Selenium IDE by going to Options -> Format. You have a choice of either TestNG or JUnit format. Personally I prefer TestNG as it is a more powerful framework. I’m not going to give a tutorial on how to using TestNG as there is plenty of information on the web for this, but the key features are:
  • Easy to define what tests you want to run and their parameters by using a testng.xml file.
  • Parameters can be automatically passed to tests using the @Parameters annotation.
  • For data driven tests, data provider methods can be used to invoke the tests multiple times with different sets of input and expected output.
  • Assertion mechanism for the checks in your tests.
  • HTML reporting.

2. Write an Ant script to compile and run your tests

The main things your Ant script needs to do:
  • Compile your tests.
  • Start the Selenium RC server (assuming you are using Selenium 1).
  • Run the tests.
  • Stop the Selenium server.
Here is a sample of what your script should look like:
?Download build.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
<project name="Selenium tests">
  <!-- The tests should only depend on three libraries: Selenium, TestNG and the MySQL Java Connector. -->
  <property name="ant.dir" location="/ant-1.7.0" />
  <echo message="Ant directory: ${ant.dir}" />
 
  <property name="lib.ant.dir" location="${ant.dir}/lib" />
  <echo message="Ant lib directory: ${lib.ant.dir}" />
 
  <!-- define TestNG location -->
  <path id="lib.testng">
    <fileset dir="${lib.ant.dir}" includes="testng-5.11-jdk15.jar"/>
  </path>
  <taskdef resource="testngtasks" classpathref="lib.testng" />
 
  <!-- define Java connector location -->
  <property name="lib.mysql.connector" location="/mysql_connector" />
 
  <!-- use the Antelope if task rather than the standard Ant one -->
  <taskdef name="iftest" classname="ise.antelope.tasks.IfTask"  
    classpath="${ant.dir}/antelope/AntelopeApp_3.1.9.jar" />
 
  <!-- define source location -->
  <property name="src.selenium-tests.dir" location="../src/selenium" />
 
  <!-- Compile tests -->
  <target name="compile-selenium-tests">
    <!-- Create the classes directory if it doesn't exist -->
    <iftest>
      <bool>
        <available file="classes" type="dir" />
      </bool>
      <echo message="Classes directory already exists" />
      <else>
        <echo message="Classes directory does not exist, creating..." />
        <mkdir dir="classes" />  
      </else>
    </iftest>
    <!-- CruiseControl seems to have problems picking up a Windows environment -->
    <!-- variable to say where the javac compiler is, -->
    <!-- so if we set fork="yes" then we can specify the location with -->
    <!-- the executable parameter. -->
    <javac srcdir="${src.selenium-tests.dir}" destdir="classes" fork="yes" 
	  executable="C:\usr\java\jdk1.6.0_06\bin\javac.exe" >
      <classpath>  
        <fileset dir="${lib.ant.dir}" includes="selenium-server-standalone-2.0b3.jar"/>
        <fileset dir="${lib.ant.dir}" includes="testng-5.11-jdk15.jar"/>
        <fileset dir="${lib.mysql.connector}" includes="mysql-connector-java-5.0.8-bin.jar" />
      </classpath>
    </javac>
  </target>
 
  <!-- Targets to start and stop the Selenium RC server. Taken from the Selenium documentation -->
  <!-- http://wiki.openqa.org/display/SRC/Selenium-RC+and+Continuous+Integration -->  
  <target name="start-selenium-server">
    <echo message="Starting Selenium RC server (runs as a separate process)" />
    <!-- To make it easier to see if the Selenium RC server is running, we copy the -->
    <!-- normal java.exe to create a different executable called -->
    <!-- javaSeleniumRCServer.exe. That way, if you list the processes Windows is running -->
    <!-- you can see if the Selenium RC server is running, rather than just seeing a -->
    <!-- process called "java". -->
    <java jar="${lib.ant.dir}/selenium-server-standalone-2.0b3.jar" fork="true" 
	  spawn="true" jvm="javaSeleniumRCServer">
      <!-- Enable logging for the Selenium server -->
      <arg line="-log selenium.log -browserSideLog"/>
    </java>
  </target>
 
  <target name="stop-selenium-server">
    <echo message="Stopping Selenium RC server" />
      <get taskname="selenium-shutdown" 
	    src="http://localhost:4444/selenium-server/driver/?cmd=shutDownSeleniumServer"
        dest="result.txt" ignoreerrors="true" />
      <echo taskname="selenium-shutdown" message="DGF Errors during shutdown are expected" />
  </target>
 
  <!-- Run tests. -->
  <target name="run-selenium-tests" depends="compile-selenium-tests>
 
      <property name="build.classes.dir" location="classes" />
 
      <!-- Start the Selenium RC server -->
      <antcall target="start-selenium-server" />
 
      <echo message="Running Selenium Tests." />
      <!-- Put the output in the same place as our other tests -->
      <!-- We don't want to immediately fail the build if the tests fail, as we -->
      <!-- need to stop the Selenium server, so we set the property testsFailed -->
      <!-- and check that AFTER we have stopped the Selenium RC server. -->
      <testng outputdir="test-output" haltonfailure="false" 
	    haltonskipped="false" failureProperty="testsFailed">
        <!-- Compiled classes are in the build/classes directory -->
        <classpath>
           <pathelement location="${build.classes.dir}" />
          <fileset dir="${lib.ant.dir}" includes="selenium-server-standalone-2.0b3.jar"/>
          <fileset dir="${lib.ant.dir}" includes="testng-5.11-jdk15.jar"/>
          <fileset dir="${lib.mysql.connector}" includes="mysql-connector-java-5.0.8-bin.jar" />
        </classpath>
        <!-- List of tests to be run is in the testng.xml file in the src/system_tests/selenium directory -->
        <xmlfileset dir="${src.selenium-tests.dir}" includes="testng.xml"/>
      </testng>
 
      <!-- Stop the Selenium RC server -->
      <antcall target="stop-selenium-server" />
 
      <!-- CruiseControl needs the test results in JUnit format, it can't read TestNG output. -->
      <!-- However, TestNG comes with an Ant command called junitreport that will conver the output. -->
      <mkdir dir="${base_dir}/unit_tests"/>
      <junitreport todir="${base_dir}/unit_tests">
        <fileset dir="${base_dir}/build/test-output">
          <include name="*/*.xml"/>
          <exclude name="*/testng-failed.xml"/>        
        </fileset>        
        <report format="noframes"  todir="${base_dir}/unit_tests"/>
      </junitreport>
 
      <!-- Check the flag set by TestNG and fail the build if we had test failures. -->
      <!-- Unfortunately using this mechanism it looks like we can't check for -->
	  <!-- skips, only failures. -->
      <iftest name="testsFailed" exists="true">
        <fail message="Test failures."/>
      </iftest>
  </target>
</project>

3. Check your tests into a source control system

Although you can get away without using source control, for any serious testing it’s a must. It will make it easier to keep track of changes to the tests and maintain multiple versions of them if you need to. The most commonly used free source control systems are Subversion and Git, although my example below uses Perforce.

4. Set up CruiseControl to run the tests

Although I’d recommend using the CruiseControl Java GUI (CC-Config) for configuring CruiseControl, under the covers it is just making changes to the config.xml file, and to explain the settings it is easiest just to give an example of that file:
?Download config.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<cruisecontrol>
  <dashboard />
  <project name="Selenium tests" requiremodification="false">
    <modificationset>
      <p4 View="//Selenium_dp/..." Port="9.20.123.456:1666" 
	    Client="Selenium_workspace" User="auto" Passwd="password" />
    </modificationset>
    <schedule Interval="86400">
      <ant Target="run-selenium-tests" 
	    AntHome="C:\Program Files\CruiseControl\apache-ant-1.7.0" 
	    BuildFile="C:\Selenium_tests\build.xml" UseDebug="true" />
    </schedule>
    <log />
    <listeners>
      <currentbuildstatuslistener 
	    File="C:\Program Files\CruiseControl\logs\status.html" />
    </listeners>
    <bootstrappers>
      <p4bootstrapper Port="9.20.123.456:1666" View="//Selenium_dp/..." 
	    User="auto" Passwd="password" Client="Selenium_workspace" />
    </bootstrappers>
  </project>
</cruisecontrol>
I’ll explain the various parts of the configuration:
  • project – requiremodification is set to false. If set to true, CruiseControl will only run the build if changes have been made to the code. For building a product, this makes sense, but not for running tests. The product you are testing could have changed, so you still want to rerun the tests even if they haven’t changed themselves.
  • modificationset – tells CruiseControl how to check if any of your tests have been updated in your source control system. This example shows the syntax for connecting to Perforce.
  • schedule – tells CruiseControl how to run your build, and how often. The time interval is measured in seconds, so an interval of 86400 seconds means the tests will be run once a day.
  • listeners – defines how to see the current progress of the build. I’ve just set this to the default log location.
  • bootstrappers – bootstrapping is what occurs before every build. In this configuration, I’ve set the bootstrapper to get all of the test files from source control. For large projects, this wouldn’t make sense, as the bootstrapper will get all of the files regardless of whether any of them have actually been updated. In that situation, you would usually write your Ant script so that it contains targets that can extract your files from source control. The bootstrapper would just get the Ant script, and only if a modification was detected in the files would a build run and the modified files be extracted from source control. However, for a small or moderately sized set of test files, the current configuration is fine.

Useful links

TestNG: http://testng.org/doc/index.html
Apache Ant manual: http://ant.apache.org/manual/index.html
CruiseControl config.xml: http://cruisecontrol.sourceforge.net/main/configxml.html

Resoure:
Headly proctor

Writing XPath selectors for Selenium tests

Tutorial: Writing XPath selectors for Selenium tests

Selenium is the most commonly used web testing framework. It allows you to write tests in Java that can perform standard “web” actions such as filling in forms, clicking buttons etc. Usually, a test will take the following form:
  • Open a web page.
  • Locate an html element.
  • Interact with it in some way. i.e. click it, type into it, select an option
  • Locate another html element.
  • Interact with it.
  • etc
Locating elements is obviously key to writing a test, and Selenium offers several ways you can do this:
  • By name.
  • By id.
  • Using XPath.
Using the name or id is okay for simple tests, but for more complex situations, you need to know how to write XPath queries. Even though you will typically start off by using the Selenium IDE Firefox plugin to record a test, which will automatically generate selectors for you, in practice you will usually find that you need to change some of them. This will often happen when you are dealing with a web page that is dynamically generated. For example, suppose you are searching a website for a shirt to buy. If there are various sizes and colours available for the shirt, you could get back multiple “Add to bag” buttons. When you click the button for the white shirt in collar size regular, the Selenium IDE won’t know that this is what you are doing. It will do its best to write an expression that picks out the button, but it will probably be incorrect. For example, if the button you click is the second one on the page, the Selenium IDE could use that to pick out the button. However, if you rerun the test and some of the sizes or colours are out of stock, the button you want might not be the second one. What you really want to do is write a test that knows to “click the button for the white shirt in regular size”.  XPath allows you to do this, but XPath queries can be quite complex to begin with, so the purpose of this tutorial is to guide you through writing a query, step by step.

Example

Step 1 – Navigate to the problematic web page

The first step is to manually navigate to the web page which your test is having a problem with. I’ve recently been working on a test that purchases a product by selecting a customer, then a product, then entering address details, then finally credit card details. I used the Selenium IDE to record the test and then saved it as Java code for playback. However, when running, it failed at the stage of entering the credit card details, specifically the card number. Have a look at the following screenshot and see if you can tell why:

Did you spot it? There are two “card number” fields on the page. The recorded Selenium test was picking up the one for the gift card. You can see that we need the one in the credit card section, so how do we go about changing the selector to pick this out?

Step 2 – Write and test your XPath expression

There are a number of ways to write and test an XPath expression:
  1. Using the Selenium IDE itself – it has a “test” option.
  2. Using Firebug and a separate XPath tester.
  3. Using Firebug and the FirePath tester, which is a separate add-on but integrates with Firebug.
All these methods work, but the third method is probably the most powerful, so it is the one I will explain here. The first thing to do is to identify the html element that you are interested in, and see how it fits into the structure of the page. If I open up Firebug, I’ll see something like this:

Firebug has the “selector” tool,  which allows you to select an html element on the page and see it in the HTML view. It is the arrow icon in the top left of the FireBug panel:

Once you have added the FirePath extension, you’ll get a separate FirePath view and you’ll also be able to use the selector tool to bring up an element in that view. So, if I switch to the FirePath view and then use the selector tool to pick out the card number input box, I’ll see the following:

An XPath expression that picks out the element will automatically be generated and displayed at the top of the panel. However, this is the expression that needs to be rewritten. I find the easiest way to write an XPath expression is to gradually build it up. What is the simplest thing to start with? Well, I’m looking for an input, so a good start might be to write something that will select all inputs:
//input
Sure enough, FirePath highlights all of the inputs on the page:

I know that I need the card number input within the “Credit Card” area of the page, but how do I know what XPath expression to write to pick this out? Well, you can use Firebug to inspect the DOM to understand the structure:

Looking at this structure, you can see that conceptually I’m looking for the input with the id of cardNumber that is within the div that has an h3 heading of “Credit Card”. To build up an expression for this, you can start by just trying to pick out that div. For example:
//div[h3/text()='Credit Card']
This correctly picks out the div, but how do you navigate down to the input? Well, in XPath you use axes to navigate up and down the DOM. The input is within the div, so we’ll need to use an axis like “child” or “descendant”. You could test that with:
//div[h3/text()='Credit Card']/descendant::*
Sure enough, that picks out all of the nodes beneath the div. I know that I want an input with an id of “cardNumber”, so I change that to:
//div[h3/text()='Credit Card']/descendant::input[@id='cardNumber']
This works, but actually it is overspecified. What if the heading was changed to an h2 or a legend? It is more flexible for the expression just to specify that the div must have some kind of child node with the text “Credit Card”, rather than specifically an h3 heading. Hence the final XPath expression is:
//div[*/text()='Credit Card']/descendant::input[@id='cardNumber']

Step 3 – put the XPath expression back into your test and rerun

Once you have worked out the XPath expression, you can put it back into your test and rerun it. I’m using the Selenium 2 API for my test, so the code becomes:
webDriver.findElement(
  By.xpath(
    "//div[*/text()='Credit Card']/descendant::input[@id='cardNumber']"
  )
).sendKeys("1234123412341234");
If you’re using Selenium 1, your syntax will be more like:
selenium.type(
  "//div[*/text()='Credit Card']/descendant::input[@id='cardNumber']",
  "1234123412341234"
);

Hopefully you’ll find that by using the above technique with Firebug and FirePath, you can build up and test even complex XPath expressions easily. If you want to read up on XPath, a good place is the w3schools tutorial:
http://www.w3schools.com/xpath/default.asp
Firebug can be downloaded from:
http://getfirebug.com/
and FirePath from:
https://addons.mozilla.org/en-us/firefox/addon/firepath/

Soure:
Hedley Proctor