Jul 12, 2013

Why Automated Tests Didn’t Find That Bug?

It is widely accepted that the purpose of automated tests is not to find new defects but rather find regression bugs as new features are developed. But even so, there are many occasions where regression bugs slip through that end up in production and really should have been caught by the automated regression tests.

Let’s examine the reasons why automated tests failed to find the regression bugs:
1 – The Scenario Was Not Though Of
Automated Tests are as good as the person who designed the tests. Automated Tests are a set of instructions that are executed against the target application. First, test analysts design test cases and come up with a list of scenarios. The scenarios are then scripted in a programming language to execute the scenarios automatically. Therefore, if a particular scenario was not thought of, it wouldn’t have been scripted to run as automated test.

2 – The Scenario Was Thought Of But Was Not Scripted
It takes time to automate tests. Depending on the complexity of the tests, test engineers coding skills, flexibility of test automation tools and frameworks, some tests take a long time to automate and hence miss the chance to find regression bugs as new features are developed.

3 – There is a Bug in the Test Code Itself
There are situations where the automated tests do not really run against the tester’s intentions or assumptions. In other words the automation engineers made a mistake in coding the tests or did not insert verification points in correct places.

4 – The Automated Tests Couldn’t Execute due to Environment Issues
This is particularly true when running System Tests via application UI, e.g. launching the application in browser. In such cases there are many dependencies on other applications, 3rd party or downstream systems, and if any of these are down or not responding or respond intermittently, the automated tests could not execute successfully and hence could not verify correctness of a particular test.

5 – Poor Analysis of Test Reports
After automated tests have executed, when there are failed tests, analysis is required to see the reason for the failed cases. This can take quite some time to analyze all the failed cases (many of which can be false positive). The analysis part is normally done manually and if the analysis is not done correctly, there could be genuine failures that are overlooked or masked by other issues.
Can you think of other issues why automated tests miss defects?

Source: Testing excellence

Jun 10, 2013

Multiple test configurations with TestNG

I use TestNG for automated testing as it has a number of features that JUnit doesn’t have. One of its strengths is the power and flexibility of configuring tests and supplying parameters to them, for which there are two methods:

Using the testng.xml file to define both the list of tests to run and parameters.
Using data providers within your test code to load parameters.

Using the testng.xml file is very simple, but not as powerful as using data providers. However, suppose you’d like to run the same set of tests with different parameters, but without having to write data providers – how could you achieve that?
One mechanism is to allow test writers to specify parameters in two files which are merged by Ant into a single testng.xml file before running the tests:

A testng-default.xml file with the list of tests and default parameters.
A .properties to supply overrides or additional parameters.

Merging files like this is a task very suited to a scripting language. Here is an implementation in groovy:

println "Merging testng-default.xml with .properties file into testng.xml"

if (args.size() != 3) {
    println "Script requires three arguments:"
    println "Path to testng template file"
    println "Path to .properties file"
    println "Path for saving merged testng file"
}

String testNGTemplate = args[0]
String propertiesFile = args[1]
String mergedTestNGFile = args[2]

// get the properties
File propsFile = new File(propertiesFile)
FileInputStream input = new FileInputStream(propsFile)
Properties props = new Properties()
props.load(input)

// Start copying the testng file
File oldTestNG = new File(testNGTemplate)
File newTestNG = new File(mergedTestNGFile)

// Get the set of parameters. Read in using Groovy XML parser.
Node xml = new XmlParser().parse(oldTestNG)
NodeList parameterList = xml.test.parameters.parameter
// iterate over parameter list and replace any that have been overridden
parameterList.each{p ->
    println p.@name
    if (props.keySet().contains(p.@name)) {
        println "Replacing ${p.@name} with value from .properties file"
        // update the value to the one set in .properties
        p.@value = props.get(p.@name)
    }
}

// Now add any properties that weren't listed in the original
// testng-default.xml file
Set<String> parameterNames = xml.test.parameters.parameter.@name
Node parameters = xml.test.parameters.getAt(0)

props.each{ key, value ->
  if (!parameterNames.contains(key)) {
    println "Adding property from .properties. Key: ${key}, value: ${value}"
    parameters.appendNode("parameter",[name: "${key}", value: "${value}"])
  }    
}

// Write out the file
new XmlNodePrinter(new PrintWriter(newTestNG)).print(xml)

Note that this script is written assuming that all of your test classes are part of a single “test”, with the parameters inside that test, like the following:

<suite name="regression-tests">
    <test name="regression-basic">
        <parameters>
            <parameter name="param1" value="value1"/>
            <parameter name="param2" value="value2"/>
            <parameter name="param3" value="value3"/>
        </parameters>
        <classes>
            <class name="uk.co.somecorp.Test1" />
            <class name="uk.co.somecorp.Test2" />      
            <class name="uk.co.somecorp.Test3" />
        </classes>
    </test>
</suite>

This is reflected in the lines in the Groovy script that navigate to the parameters, such as:

NodeList parameterList = xml.test.parameters.parameter

Obviously if you want to organise your default testng file differently you’ll need to adjust these parts of the script.
You can invoke the script from Ant with:

<taskdef name="groovy" classname="org.codehaus.groovy.ant.Groovy">
   <classpath>
      <fileset file="/path_to_groovy/groovy-all.jar"/>
   </classpath>
</taskdef>    

<!-- Target to merge the testng-default.xml and .properties files -->
  <target name="merge-files">
     <groovy src="/path_to_script/MergeTestNGAndProperties.groovy">
        <arg value="/some_path/testng-default.xml"/>
        <arg value="/some_path/my.properties"/>
        <arg value="/output_path/testng.xml"/>
     </groovy>
</target>

Resoure:

hedleyproctor

Automating Selenium testing with TestNG Ant and CruiseControl

Automating Selenium testing with TestNG, Ant and CruiseControl

If you are using Selenium for web testing, most likely you’ll want to make your tests as automated as possible so that they can be run automatically on a regular basis. You can do this by using a few additional technologies:

1. Integrate with a Java test framework

For any serious testing, your tests will need to be written in Java, rather than the basic Selenium html format. You can save the tests in Java from the Selenium IDE by going to Options -> Format. You have a choice of either TestNG or JUnit format. Personally I prefer TestNG as it is a more powerful framework. I’m not going to give a tutorial on how to using TestNG as there is plenty of information on the web for this, but the key features are:

Easy to define what tests you want to run and their parameters by using a testng.xml file.
Parameters can be automatically passed to tests using the @Parameters annotation.
For data driven tests, data provider methods can be used to invoke the tests multiple times with different sets of input and expected output.
Assertion mechanism for the checks in your tests.
HTML reporting.

2. Write an Ant script to compile and run your tests

The main things your Ant script needs to do:

Compile your tests.
Start the Selenium RC server (assuming you are using Selenium 1).
Run the tests.
Stop the Selenium server.

Here is a sample of what your script should look like:

^?Download build.xml

<project name="Selenium tests">
  <!-- The tests should only depend on three libraries: Selenium, TestNG and the MySQL Java Connector. -->
  <property name="ant.dir" location="/ant-1.7.0" />
  <echo message="Ant directory: ${ant.dir}" />
 
  <property name="lib.ant.dir" location="${ant.dir}/lib" />
  <echo message="Ant lib directory: ${lib.ant.dir}" />
 
  <!-- define TestNG location -->
  <path id="lib.testng">
    <fileset dir="${lib.ant.dir}" includes="testng-5.11-jdk15.jar"/>
  </path>
  <taskdef resource="testngtasks" classpathref="lib.testng" />
 
  <!-- define Java connector location -->
  <property name="lib.mysql.connector" location="/mysql_connector" />
 
  <!-- use the Antelope if task rather than the standard Ant one -->
  <taskdef name="iftest" classname="ise.antelope.tasks.IfTask"  
    classpath="${ant.dir}/antelope/AntelopeApp_3.1.9.jar" />
 
  <!-- define source location -->
  <property name="src.selenium-tests.dir" location="../src/selenium" />
 
  <!-- Compile tests -->
  <target name="compile-selenium-tests">
    <!-- Create the classes directory if it doesn't exist -->
    <iftest>
      <bool>
        <available file="classes" type="dir" />
      </bool>
      <echo message="Classes directory already exists" />
      <else>
        <echo message="Classes directory does not exist, creating..." />
        <mkdir dir="classes" />  
      </else>
    </iftest>
    <!-- CruiseControl seems to have problems picking up a Windows environment -->
    <!-- variable to say where the javac compiler is, -->
    <!-- so if we set fork="yes" then we can specify the location with -->
    <!-- the executable parameter. -->
    <javac srcdir="${src.selenium-tests.dir}" destdir="classes" fork="yes" 
	  executable="C:\usr\java\jdk1.6.0_06\bin\javac.exe" >
      <classpath>  
        <fileset dir="${lib.ant.dir}" includes="selenium-server-standalone-2.0b3.jar"/>
        <fileset dir="${lib.ant.dir}" includes="testng-5.11-jdk15.jar"/>
        <fileset dir="${lib.mysql.connector}" includes="mysql-connector-java-5.0.8-bin.jar" />
      </classpath>
    </javac>
  </target>
 
  <!-- Targets to start and stop the Selenium RC server. Taken from the Selenium documentation -->
  <!-- http://wiki.openqa.org/display/SRC/Selenium-RC+and+Continuous+Integration -->  
  <target name="start-selenium-server">
    <echo message="Starting Selenium RC server (runs as a separate process)" />
    <!-- To make it easier to see if the Selenium RC server is running, we copy the -->
    <!-- normal java.exe to create a different executable called -->
    <!-- javaSeleniumRCServer.exe. That way, if you list the processes Windows is running -->
    <!-- you can see if the Selenium RC server is running, rather than just seeing a -->
    <!-- process called "java". -->
    <java jar="${lib.ant.dir}/selenium-server-standalone-2.0b3.jar" fork="true" 
	  spawn="true" jvm="javaSeleniumRCServer">
      <!-- Enable logging for the Selenium server -->
      <arg line="-log selenium.log -browserSideLog"/>
    </java>
  </target>
 
  <target name="stop-selenium-server">
    <echo message="Stopping Selenium RC server" />
      <get taskname="selenium-shutdown" 
	    src="http://localhost:4444/selenium-server/driver/?cmd=shutDownSeleniumServer"
        dest="result.txt" ignoreerrors="true" />
      <echo taskname="selenium-shutdown" message="DGF Errors during shutdown are expected" />
  </target>
 
  <!-- Run tests. -->
  <target name="run-selenium-tests" depends="compile-selenium-tests>
 
      <property name="build.classes.dir" location="classes" />
 
      <!-- Start the Selenium RC server -->
      <antcall target="start-selenium-server" />
 
      <echo message="Running Selenium Tests." />
      <!-- Put the output in the same place as our other tests -->
      <!-- We don't want to immediately fail the build if the tests fail, as we -->
      <!-- need to stop the Selenium server, so we set the property testsFailed -->
      <!-- and check that AFTER we have stopped the Selenium RC server. -->
      <testng outputdir="test-output" haltonfailure="false" 
	    haltonskipped="false" failureProperty="testsFailed">
        <!-- Compiled classes are in the build/classes directory -->
        <classpath>
           <pathelement location="${build.classes.dir}" />
          <fileset dir="${lib.ant.dir}" includes="selenium-server-standalone-2.0b3.jar"/>
          <fileset dir="${lib.ant.dir}" includes="testng-5.11-jdk15.jar"/>
          <fileset dir="${lib.mysql.connector}" includes="mysql-connector-java-5.0.8-bin.jar" />
        </classpath>
        <!-- List of tests to be run is in the testng.xml file in the src/system_tests/selenium directory -->
        <xmlfileset dir="${src.selenium-tests.dir}" includes="testng.xml"/>
      </testng>
 
      <!-- Stop the Selenium RC server -->
      <antcall target="stop-selenium-server" />
 
      <!-- CruiseControl needs the test results in JUnit format, it can't read TestNG output. -->
      <!-- However, TestNG comes with an Ant command called junitreport that will conver the output. -->
      <mkdir dir="${base_dir}/unit_tests"/>
      <junitreport todir="${base_dir}/unit_tests">
        <fileset dir="${base_dir}/build/test-output">
          <include name="*/*.xml"/>
          <exclude name="*/testng-failed.xml"/>        
        </fileset>        
        <report format="noframes"  todir="${base_dir}/unit_tests"/>
      </junitreport>
 
      <!-- Check the flag set by TestNG and fail the build if we had test failures. -->
      <!-- Unfortunately using this mechanism it looks like we can't check for -->
	  <!-- skips, only failures. -->
      <iftest name="testsFailed" exists="true">
        <fail message="Test failures."/>
      </iftest>
  </target>
</project>

3. Check your tests into a source control system

Although you can get away without using source control, for any serious testing it’s a must. It will make it easier to keep track of changes to the tests and maintain multiple versions of them if you need to. The most commonly used free source control systems are Subversion and Git, although my example below uses Perforce.

4. Set up CruiseControl to run the tests

Although I’d recommend using the CruiseControl Java GUI (CC-Config) for configuring CruiseControl, under the covers it is just making changes to the config.xml file, and to explain the settings it is easiest just to give an example of that file:

^?Download config.xml

<cruisecontrol>
  <dashboard />
  <project name="Selenium tests" requiremodification="false">
    <modificationset>
      <p4 View="//Selenium_dp/..." Port="9.20.123.456:1666" 
	    Client="Selenium_workspace" User="auto" Passwd="password" />
    </modificationset>
    <schedule Interval="86400">
      <ant Target="run-selenium-tests" 
	    AntHome="C:\Program Files\CruiseControl\apache-ant-1.7.0" 
	    BuildFile="C:\Selenium_tests\build.xml" UseDebug="true" />
    </schedule>
    <log />
    <listeners>
      <currentbuildstatuslistener 
	    File="C:\Program Files\CruiseControl\logs\status.html" />
    </listeners>
    <bootstrappers>
      <p4bootstrapper Port="9.20.123.456:1666" View="//Selenium_dp/..." 
	    User="auto" Passwd="password" Client="Selenium_workspace" />
    </bootstrappers>
  </project>
</cruisecontrol>

I’ll explain the various parts of the configuration:

project – requiremodification is set to false. If set to true, CruiseControl will only run the build if changes have been made to the code. For building a product, this makes sense, but not for running tests. The product you are testing could have changed, so you still want to rerun the tests even if they haven’t changed themselves.
modificationset – tells CruiseControl how to check if any of your tests have been updated in your source control system. This example shows the syntax for connecting to Perforce.
schedule – tells CruiseControl how to run your build, and how often. The time interval is measured in seconds, so an interval of 86400 seconds means the tests will be run once a day.
listeners – defines how to see the current progress of the build. I’ve just set this to the default log location.
bootstrappers – bootstrapping is what occurs before every build. In this configuration, I’ve set the bootstrapper to get all of the test files from source control. For large projects, this wouldn’t make sense, as the bootstrapper will get all of the files regardless of whether any of them have actually been updated. In that situation, you would usually write your Ant script so that it contains targets that can extract your files from source control. The bootstrapper would just get the Ant script, and only if a modification was detected in the files would a build run and the modified files be extracted from source control. However, for a small or moderately sized set of test files, the current configuration is fine.

Useful links

TestNG: http://testng.org/doc/index.html
Apache Ant manual: http://ant.apache.org/manual/index.html
CruiseControl config.xml: http://cruisecontrol.sourceforge.net/main/configxml.html

Resoure:
Headly proctor

Pages

Jul 12, 2013

Why Automated Tests Didn’t Find That Bug?

Jun 10, 2013

Multiple test configurations with TestNG

Multiple test configurations with TestNG

Automating Selenium testing with TestNG Ant and CruiseControl

Automating Selenium testing with TestNG, Ant and CruiseControl

1. Integrate with a Java test framework

2. Write an Ant script to compile and run your tests

3. Check your tests into a source control system

4. Set up CruiseControl to run the tests

Useful links