Introduction to Blackbox testing of Web Applications using Selenium

After my visit tot JavaPolis, in December 2005, I wrote a brief article on Selenium, a tool for testing Web Applications. For one of our internal Knowledge Center Sessions (next Tuesday), I have created a presentation and accompanying workshop on Selenium. It turns out to be a very handy tool for functional , blackbox tests on HTML applications, without any dependency on the implementation of the Web Application: PHP, .NET, Java/J2EE, static HTML – it does not matter, since Selenium tests from the browser end. In this article, I will share some of my findings.

Selenium (http://www.openqa.org/selenium/) is a test tool for web applications. Selenium tests run directly in a browser, just as real users do. And they run in Internet Explorer, Mozilla and Firefox on Windows, Linux, and Macintosh. No other test tool covers such a wide array of platforms.

• Browser compatability testing. Test your application to see if it works correctly on different browsers and operating systems. The same script can run on any Selenium platform.
• System functional testing. Create regression tests to verify application functionality and user acceptance.

Selenium Blackbox Tests are defined in (static) HTML documents as a table of command-records. Each table row has three columns (the last of which can be empty for some commands). Typically a command-record states which command should be performed (like click, fireEvent, type), on which element (link, button, document.images[2], the form input which ID=myNameInput) and with what parameter or context information. These commands are interpreted by the Selenium JavaScript command interpreter and executed in the application frame – a part of the browser window where the Application Under Test (AUT) is displayed. Through these commands, we can mimic the actions performed by a user as well as analyze the results of these actions. We can read form elements, titles, alert texts as well as attributes for any element in the document. We can also click on buttons, hyperlinks, checkboxes and radiobuttons as well as select from combo-boxes and select-elements.

 
Selenium gives us in the browser window a good overview of what the results of running a single TestCase or an entire TestSuite are. Through the View DOM and Show Log options, we can zoom in on individual steps and commands.

Note: it is also possible to have Selenium post the Test Results in an HTML Form to a URL where for example a Servlet processes the results. When using that post-mechanism, the Selenium test can be fully automated, as no human intervention is required.

See http://www.openqa.org/selenium/seleniumReference.html for an overview of all Selenium commands.

In this workshop, we will create a very simple Html “application” – with only static Html – and build the Selenium Test Suite. Through this exercise, you will get a good feel for how to get started with Selenium and how to use the Selenium commands for creating a TestCase. In a second step we will apply Selenium to a real three tier web application.

Set up the Web Application to be tested

Even though I applaud the concept of Test Driven Development, it will be quite hard to first create the Selenium Test Cases and afterwards build the application on which to apply them. So we will start by creating the HTML application. For this mini-workshop, we have a mini-application, a set of three HTML documents with two images and a stylesheet.

Download or Copy the file StaticHtmlAppForSelenium.zip and unzip it to a home directory of your choice.

The first page in the application is HomePage.html:

Clicking on the hyperlink takes us to the second page:

And clicking on the button initiates navigation to the third and most interesting page – a form:

Let’s assume this application is built according to the functional requirements. We now want to create a black box test that can be used to verify whether the main requirements are complied with in all to be supported browser platforms. That test can also be used as a regression test that can be used to verify whether after maintenance activities the application still adheres to all requirements.

Getting Started with Selenium

• Dowload Selenium (release 0.6) from http://www.openqa.org/selenium/ (3.8 Mb)
• Extract this zip-file to a SELENIUM_HOME directory, like c:\selenium or c:\apps\selenium
• Copy the SELENIUM_HOME\selenium-0.6.0\selenium directory to the root of your Web Application directory. That means that the selenium directory – with among others the TestRunner.html file – is at the same level as the static HTML files of the Web Application

• A Selenium Test session is typically started by running the TestRunner.html file in the selenium sub-directory. This file sets up the frames with the TestSuite, the current TestCase and the TestRunner console as well as the application frame.
  

The TestRunner.html file loads the upper-left frame with the file TestSuite.html in the selenium\tests directory. This is a file that is manually edited to contain a table with all the (application specific) TestCases. Let’s create our first TestCase. We can best do that by first specifying the TestCase in the TestSuite.html file.

• Open TestSuite.html in text editor. Remove all table records, except for the first two. Edit the first table record; set the contents to something like Test Suite voor Statische AMIS Website applicatie. The second table record has a reference to the first TestCase, the static file TestOpen.html . We’ll keep that reference for now. So the file should now look this:

<html>
<head>
  <meta content="text/html; charset=ISO-8859-1" http-equiv="content-type">
  <title>Test Suite</title>
</head>
<body>
  <table   cellpadding="1"
           cellspacing="1"
           border="1">
        <tbody>
            <tr><td><b>Test Suite voor Statische AMIS Website applicatie</b></td></tr>
            <tr><td><a href="./TestOpen.html">TestOpen</a></td></tr>
        </tbody>
    </table>
</body>
</html>

• And now the fun starts. We will flesh out that first TestCase. Open the file TestOpen.html. The <table> in this file contains table records; the first one is used as the table header – it is ignored by the Selenium Command Processor. All subsequent table records
  should contain a valid Selenium Command in the first t
  able cell and its parameters in the second and optionally the third table cell. We will do a few simple things in this first TestCase:

• Open the Web Application (in this case the static HTML document HomePage.html) (Selenium Command open)
• Check the title of the page: is it HomePage? (assertTitle)
• Check whether the document contains the text “Klik hier om” (verifyTextPresent)
• Check if the title attribute of the anchor element with id equal to url1 has a value of “Klik hier om te beginnen” (assertAttribute)
• Click on this same hyperlink (that will take us to the second page, Site1.html) (click)

See http://www.openqa.org/selenium/seleniumReference.html for the complete reference of all Selenium commands.

The HTML code in TestOpen.html should look like this:

<html>
<head>
  <meta content="text/html; charset=ISO-8859-1"
 http-equiv="content-type">
  <title>Test Open</title>
</head>
<body>
<table cellpadding="1" cellspacing="1" border="1">
  <tbody>
    <tr>
      <td rowspan="1" colspan="3">Test Open<br>
      </td>
    </tr>
    <tr>
      <td>open</td>
      <td>../HomePage.html</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>assertTitle</td>
      <td>HomePage<br>
      </td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>verifyTextPresent</td>
      <td>Klik hier om</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>assertAttribute</td>
      <td>//a[@id='url1']/@title</td>
      <td>Klik hier om te beginnen</td>
    </tr>
    <tr>
      <td>click</td>
      <td>//a[@id='url1']</td>
      <td>&nbsp;</td>
    </tr>
  </tbody>
</table>
</body>
</html>

When rendered, this is the display:

• Time to run our first Selenium TestSuite. Open the file selenium\TestRunner.html. This in turn will open TestSuite.html and TestOpen.html.

The browser with the TestRunner:

We can now run the test from the Control Panel. The results will be shown – red for failure, green for success. Sounds familiar, doesn’t it?

We can now run the test from the Control Panel. The results will be shown – red for failure, green for success. Sounds familiar, doesn’t it?

 

More advanced Test Operations

In this section, we will make our tests a little more involved. We will add two TestCases, one focusing on Site1.html and the other one dedicated to Site2.html. This last document is an HTML Form that looks very much like our typical dynamic Web Application.

• Let’s first update the TestSuite.html file with the two new TestCases:

<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="content-type">
<title>Test Suite</title>
</head>
<body>
<table     cellpadding="1"
           cellspacing="1"
           border="1">
        <tbody>
            <tr><td><b>Test Suite voor Statische AMIS Website applicatie</b></td></tr>
            <tr><td><a href="./TestOpen.html">TestOpen</a></td></tr>
            <tr><td><a href="./TestSite1.html">TestSite1</a></td></tr>
            <tr><td><a href="./TestSite2.html">TestSite2</a></td></tr>
        </tbody>
    </table>
</body>
</html>

• Now we create the TestSite1.html file by copying and renaming the TestOpen.html file. In this TestCase, we want the following steps:

• Open TestSite1.html
• assertTitle: check of the title for this page is equal to AMIS Site – pagina 1
• Check whether we have an IMG element that displays the AMISLibrary.jpg image
• Click on that image and verify whether we get an alert saying something like “kunt niet”
• Click on the button called “naarform” – that will takes us to Site2.html

Expressed in HTML, this set of Selenium instructions is as follows:

<html>
<head>
  <meta content="text/html; charset=ISO-8859-1"
 http-equiv="content-type">
  <title>Test Open</title>
</head>
<body>
<table cellpadding="1" cellspacing="1" border="1">
  <tbody>
    <tr>
      <td rowspan="1" colspan="3">Test Open<br>
      </td>
    </tr>
    <tr>
      <td>open</td>
      <td>../Site1.html</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>assertTitle</td>
      <td>AMIS Site - pagina 1<br>
      </td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>assertElementPresent</td>
      <td>//img[@src='AMISLibrary.jpg']</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>click</td>
      <td>document.images[0]</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>assertAlert</td>
      <td>*kunt niet*</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>click</td>
      <td>name=naarform</td>
      <td>focus</td>
    </tr>
  </tbody>
</table>
</body>
</html>

The browser displays this in the following manner:

Note: one thing that may easily cause confusion – it sure confused me – is the distinction between verify and assert. All Selenium Checks can be used in 2 modes, “assert” and “verify”. These behave identically, except that when an “assert” check fails, the test is aborted. When a “verify” check fails, the test will continue execution. This allows a single “assert” to ensure that the application is on the correct page, followed by a bunch of “verify” checks to test form field values, labels, etc.

Now we have that out of the way, let us run this second TestCase from the TestRunner.<
br /> hspace=”0″ border=”0″ align=”bottom” src=”https://technology.amis.nl/wp-content/uploads/images/selenium14.jpg” />

Testing interaction with Html Form Elements

The third TestCase – TestSite2.html – will prove the most interesting. This is where we start interacting with Form elements. The Site2.html page is this one:

It has a number of form elements, different types like Text input, Radio Buttons and Select (Combobox). In addition, there is a piece of client side logic that will transform the Voornaam item to lowercase as soon as the field loses focus. When the form is submitted, there is client side validation on all required items (voornaam en achternaam). There might be – I know there isn’t – validation on the format of the string in the Geboortedatum field.

We will perform the following actions in this TestCase:

1. Open Site2.html
2. Focus on the voornaam item
3. Set the value of Voornaam to “Mijn Voornaam”
4. Leave the Voornaam item (fire the blur event)
5. Wait for three seconds
6. Verify whether the value in Voornaam has been ‘lower cased’ to “mijn voornaam”
7. Set the field Geboortedatum to “27-12-1969”
8. Set the Geslacht Radiogroup to Vrouw
9. Select Chinees for the Diner voorkeur
10. Submit the form and verify whether an alert is displayed (we have not provided a value for the required achternaam item)
11. Set the value of Achternaam to Jellema and submit again

 

• Create the TestSite2.html file by copying and renaming TestSite1.html. Note that we already added the TestCase to the TestSuite.html.
• Write the Selenium Test Command in the TestSite2.html file. The result should be something like:

<html>
<head>
  <meta content="text/html; charset=ISO-8859-1"
 http-equiv="content-type">
  <title>Test Open</title>
</head>
<body>
<table cellpadding="1" cellspacing="1" border="1">
  <tbody>
    <tr>
      <td rowspan="1" colspan="3">Test Open<br>
      </td>
    </tr>
    <tr>
      <td>open</td>
      <td>../Site2.html</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>fireEvent</td>
      <td>name=voornaam</td>
      <td>focus</td>
    </tr>
    <tr>
      <td>type</td>
      <td>name=voornaam</td>
      <td>Mijn Voornaam</td>
    </tr>
    <tr>
      <td>fireEvent</td>
      <td>name=voornaam</td>
      <td>blur</td>
    </tr>
    <tr>
      <td>pause</td>
      <td>3000</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>assertValue</td>
      <td>document.forms[0].voornaam</td>
      <td>mijn voornaam</td>
    </tr>
    <tr>
      <td>type</td>
      <td>name=geboortedatum</td>
      <td>27-12-1969</td>
    </tr>
    <tr>
      <td>click</td>
      <td>//input[@name='geslacht' and @value='v']</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>select</td>
      <td>name=dinerkeuze</td>
      <td>value=c</td>
    </tr>
    <tr>
      <td>click</td>
      <td>name=submitButton</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>assertAlert</td>
      <td>*verplicht*</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>fireEvent</td>
      <td>name=achternaam</td>
      <td>focus</td>
    </tr>
    <tr>
      <td>type</td>
      <td>name=achternaam</td>
      <td>Jellema</td>
    </tr>
    <tr>
      <td>click</td>
      <td>name=submitButton</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>fireEvent</td>
      <td>name=achternaam</td>
      <td>focus</td>
    </tr>
  </tbody>
</table>
</body>
</html>

And in the browser this is like this:

• Run this new TestCase in the TestRunner:

• At this moment, we do not test whether the application enforces the format dd-mm-jjjj format on the Geboortedatum field. Let us add test commands to set the Geboortedatum to an erroneous value, blur the field and test for an alert. Add the following table records in the table in TestSite2.html:

    <tr>
      <td>type</td>
      <td>name=geboortedatum</td>
      <td>27 december 1969</td>
    </tr>
    <tr>
      <td>type</td>
      <td>name=geboortedatum</td>
      <td>27 december 1969</td>
    </tr>
    <tr>
      <td>assertAlert</td>
      <td>*format*</td>
      <td>&nbsp;</td>
    </tr>
    <tr>
      <td>fireEvent</td>
      <td>name=geboortedatum</td>
      <td>focus</td>
    </tr>

We have a failed test! Let us do something about that.

<p>
   Geboortedatum 
   <input type="text" name="geboortedatum" maxlength="15" size="15" onBlur="if (!validateDateFormat( this.value )){alert('Valid format is dd-mm-yyyy')};" />
   (dd-mm-jjjj, bijvoorbeeld 15-09-1969)
</p>

And the new JavaScript fiunction:

<script type="text/javascript">

function validateDateFormat( strValue ) {
/************************************************
DESCRIPTION: Validates that a string contains only
    valid dates with 2 digit month, 2 digit day,
    4 digit year. Date separator must be -.
    Uses combination of regular expressions and
    string parsing to validate date.
    Ex. dd-mm-yyyy like 15-09-69 

PARAMETERS:
   strValue - String to be tested for validity

RETURNS:
   True if valid, otherwise false.

REMARKS:
   Avoids some of the limitations of the Date.parse()
   method such as the date separator character.
*************************************************/
  var objRegExp = /^\d{1,2}(\-)\d{1,2}\1\d{4}$/

  //check to see if in correct format
  if(!objRegExp.test(strValue))
    return false; //doesn't match pattern, bad date
  else{
    
    // var strSeparator = strValue.substring(2,3)
    // bug fix by Radovan Ra
dic to get the fir
st separator
    for (i=0; i<strValue.length; i++) {
      if (strValue.charAt(i)>'9') or (strValue.charAt(i)<'0')
        break;
    }
    var strSeparator="-";

    var arrayDate = strValue.split(strSeparator); //split date into month, day, year
    //create a lookup for months not equal to Feb.
    var arrayLookup = { '01' : 31,'03' : 31, '04' : 30,'05' : 31,'06' : 30,'07' : 31,
                        '08' : 31,'09' : 30,'10' : 31,'11' : 30,'12' : 31}
    var intDay = parseInt(arrayDate[0]);

    //check if month value and day value agree
    if(arrayLookup[arrayDate[1]] != null) {
      if(intDay <= arrayLookup[arrayDate[1]] && intDay != 0)
        return true; //found in lookup table, good date
    }

    //check for February (bugfix 20050322)
    var intMonth = parseInt(arrayDate[1]);
    if (intMonth == 2) { 
       var intYear = parseInt(arrayDate[2]);
       if( ((intYear % 4 == 0 && intDay <= 29) || (intYear % 4 != 0 && intDay <=28)) && intDay !=0)
          return true; //Feb. had valid number of days
       }
  }
  return false; //any other values, bad date
}
</script>

Now run the test again. This time, we should have an alert and therefore the test should succeed.

• Selenium Tests can be run from the command line, like this:

firefox.exe file:///W:/SeleniumWebAppTesting/StaticHtmlTest/public_html/selenium/TestRunner.html?auto=true

 

• The TestRunner also accepts a parameter called test. This parameter specifies which HTML source contains the TestSuite with TestCases. By default, this parameter is set to tests/TestSuite.html. We could run our very own TestSuite document using a URL like:

firefox.exe file:///W:/SeleniumWebAppTesting/StaticHtmlTest/public_html/selenium/TestRunner.html?test=../MyOwnTestSuite.html&auto=true.
The value of test is interpreted relative to the directory that holds the TestRunner.html document.

 

• We can also specify that the testresults should be posted to a url where a servlet or other post-handling component knows how to deal with the Selenium results. Selenium can be integrated with an automated build. When the parameter “auto=true” is added to the URL, Selenium will run the entire suite of tests, and then post the results to a handling URL. The default URL is “/postResults”, but an alternative handler location can be provided by specifying a “resultsUrl” parameter.

 

Also see: http://www.openqa.org/selenium/testrunner.html

 

In a subsequent article, I will be discussing the Selenium IDE for Firefox. It is implemented as a Firefox extension, and allows you to record, edit, and debug tests. Selenium IDE includes the entire Selenium Core, allowing you to easily and quickly record and play back tests in the actual environment that they will run.

Selenium IDE is not only recording tool: it is a complete IDE. You can choose to use it’s recording capability, or you may edit your scripts by hand. With autocomplete support and the ability to move commands around quickly, Selenium IDE is the ideal environment for creating Selenium tests no matter what style of tests you prefer

Using the Selenium IDE will make the creation of Selenium Tests a lot easier than we saw in this article.