Using Selenium for automated functional testing

We are now in an age when it is accepted by most IT managers that Software Quality Assurance – and Testing/Validation & Verification in particular – is not only a desirable element in any project, but is actually an essential aspect that cannot be ignored if you want to remain competitive towards more SQA-mature rivals.

The cost of cutting corners usually arrives later, and with heavy interest. In extreme cases, it can make you go bankrupt if the fault has enough impact, but in an age of fast social networking and real-time news – damaging your reputation when your products fail is the most common setback.

Fortunately, it is now possible to rely not only on enterprise-class tools but also on simple, open-source free suites, which help refine the art from management of the initial requirements to test management, automation, defect management, and related practices.


Record and play

Development cycles are getting shorter and the ability to test rapidly in an agile mode going forward is what companies want. At LeaseWeb’s Cloud unit, we have been attempting to find a balance between test costs and coverage, and towards that, there has been an increased effort in areas like unit testing, test automation, and continuous integration.

Shorter periods do not necessarily reduce the risk of new functionality affecting existing functionality; in fact, with frequent releases, there is less time for regression testing. In this post, we will present a small introduction on how an investment on the automation of functional tests, especially regression suites has helped us move faster, without added risk.

Regression cycles can be a drain on productivity and absorb time that would otherwise be spent in other activities, including improving the test management process. Manual regression testing can also be a mundane and boring activity for most test teams, requiring concentration even when following a clear test plan or checklist.

Our team has realized the necessity to automate functional tests and has taken some initial steps:

  • We looked at our test suites and identified core test cases in very high risk/high impact areas.
  • Of the flagged tests, we determined which were more time-consuming on each development cycle, and of those, which would be better performed via automation, and which would benefit the most from experienced-based, human testing, especially when involving exploratory testing techniques. It was determined that the most time-consuming test runs were related to regression on a Bootstrap-frontend-framework based web portal used by our internal teams to manage our Cloud offerings.
  • A subset of the test cases were then selected for automation scripting, allowing us an initial level of coverage of about 43%.
  • With that in mind, we looked at several possible tools for automation, some strongly GUI-dependent, others relying on headless browsers. Both have their strengths and weaknesses, usually trading between ease of first write/maintenance and customization power.

For now, we settled on using Selenium IDE (Se-IDE), a free Firefox extension created by Shinya Kasatani of Japan that can automate the browser usage through a record-and-playback feature. It is both a recording tool and a simple, accessible IDE.

NNE-article1-1The main advantage of Se-IDE is its ease of use when quickly recording test cases by test teams with low-programming skill sets, as well as future easy maintenance. At the same time, it retains the ability to export tests to formats that can be used on Selenium 2 (formerly called WebDriver). In the first case, most interactivity occurs visibly on the browser, while in the latter, extra power is possible via direct control of the browser at OS level.

Even with Se-IDE having limited native functionality, its original commands can be expanded by directly coding JavaScript in the IDE window, should the user need it. In this first article, we will provide an overview of how Se-IDE can be used, with later articles/tutorials focusing on specifics like Element Locators and advanced tests.

Setting up

As a starting point, install the Selenium IDE plugin in Firefox. Selenium IDE has a plugin system that allows for easy extension and customization, with a few additional browser extensions and Se-IDE plugins that can prove useful:

  • Firebug and FirePath: These two extensions provide various useful tools for the Selenium scripter, but object identification will probably be one of the most useful.
  • Highlight Elements: This Selenium plugin allows much easier visual identification of objects being targeted during the execution of a script.

NNE-article1-2

  • Stored Variables: Selenium does not natively provide a way to keep track of stored variables, so this plugin is quite useful.

NNE-article1-3

NNE-article1-4Running the main Selenium IDE plugin

After running Selenium IDE, you can also configure dedicated extensions for the IDE. That can be configured in Options>General>Selenium IDE Extensions. Multiple extensions can be added, comma-separated:

NNE-article1-5

Expanding Selenium IDE further via JavaScript user extensions

The following also prove useful:

  • Sideflow: Selenium does not natively provide a powerful way to control script flow. You can enhance that by adding labels, while cycles, GoTo jumps and so on, via the Sideflow plugin.

NNE-article1-6

Controlling script flow with simple labels, while, and GoToIf usage

  • Random: This library allows the creation of random numbers and strings according to certain parameters. It proves useful to randomize data in relevant test cases.

The usual recipe

After running the IDE window, you will be presented with a blank project. In most cases, the formula to prepare a test case revolves around three basic steps:

  1. Record an execution via manually running a test case.
  2. Fine-tune the recorded script to make it as generic and robust to change as possible, without compromising its ability to validate test conditions. This is the lengthier stage.
  3. Integrate your new test case in the corresponding suite, adding flow-control if necessary. In certain usages, you will want to limit the execution of a test case block depending on previous results and/or required coverage for a release.

Inevitably, you will have to revisit your script as the tested product evolves. Scope changes like new features might require updates. Similarly, the identification of faults in production might bring the need to expand coverage with new tests.

NNE-article1-7

(The Selenium IDE interface, including extra plugins)

Once a stable pool of tests is consolidated, they are executed considerably faster, especially when repeat runs are required. That does not mean, however, that the test cases will not need to be revised often; in our environment, the sprint cycles mean that new functionality is released every two weeks.

In some (most) cases, such new functionality does not affect older regression test cases, but there are occasions when a major interface change might require tweaks to the current scripts, or even a full rewrite. Every time you refactor a script, you might find new ways to make it more adaptable to future changes and updates.

Tracking variables

One way to make the test cases more robust is to use stored variables instead of hardcoded content as much as possible. You can do this with the Store command, later retrieving content with a $ wrapper for full flexibility.

For example, if you store “bar” as the variable my_string_text1, you can later use it in any command, e.g. a Type command with “foo${ my_string_text1}” as value would result in “foobar” being output anywhere during script execution.

If you installed the Stored-Vars plugin mentioned before, a new tab at the bottom of the interface will allow you to keep track of variables, useful during debug/step execution.

Extending the native commands with JavaScript usage

Se-IDE provides a limited number of native functions out-of-the-box. In case you require something that Se-IDE does not originally do, you can add your own JavaScript code. A simple example would be randomizing a username:

NNE-article1-8

Using this in the ‘Target’ field of a Store command, would store “TestUser” plus a random 0-99 number in the variable entered in the value field. While Se-IDE did not natively allow it, a simple code snipped added the feature.

Another simple example would be selecting one of three random locations for a web form, in that case you could do it by using something like this in the target field:

NNE-article1-9

Locating objects

Web test automation predominantly revolves around GUI objects i.e. various UI controls like pull-downs, input boxes, text fields and so on. These objects can be identified via Name, ID, Link, XPath, CSS, etc. and some might have changing properties during runtime.

Once a script is initially recorded, you might find necessary to adjust object identifiers, as they allow Se-IDE to identify targets for actions during runtime, and the accuracy of this process is vital.

Firebug can help identify objects precisely, something you can confirm with the FIND button in the “Target” area of Selenium IDE. You can use Firebug’s “Inspect” tool, select the element and then click FirePath to see its CSS or XPath identifiers.

NNE-article1-10

(Firebug and FirePath showing the XPath locator for an image)

By default, the Se-IDE frequently generates index-based XPath while recording – this is not the right approach and maintainability becomes an issue, as the likelihood of a script breaking simply because an object is later moved is high. For this reason, it might be beneficial to convert those locators to IDs or CSS.

Se-IDE locators’ work on single HTML references at a time, but often you need to work with a nested HTML structure with frames. Firebug can help analyze the HTML DOM to help you derive the best object identification locator.

Wrapping up

In our next posts, we are going to work on a short tutorial and later create advanced automated test cases using only Se-IDE installed as described, as well as delve deeper into locators, recording and editing, and custom JavaScript usage.

Even though the Se-IDE has a decent amount of functionality, it also has a few limitations, for example, it is restricted to only Firefox and it lacks the ability to scale well. To counteract that, we will show you later how to use the IDE to help write cases for the standalone, external Selenium Server.

Testing your project with PHPUnit and Selenium

PHPUnit is a very handy tool you can use to test and detect possible mistakes in your PHP project.

Functional testing is also possible with PHPUnit using default headless browser (crawler). It works perfect as long as you don’t need to test functionality which uses JavaScript. For this purpose, you can use PHPUnit and Selenium.

Selenium is a powerful test tool that allows you to perform tests for web applications in any programming language using any mainstream browser.

Installation requirements

Before you start using Selenium to test your PHP project, you need to install the following:

  1. Selenium Server
  2. PHPUnit_Selenium package

1. Installing Selenium Server

Perform the following steps to install the Selenium Server:

    1. Download a distribution archive of Selenium Server.
    2. Unzip the distribution archive and copy selenium-server-standalone-2.35.0.jar (check the version suffix) to a folder from where you will run the server (such as /usr/local/bin).
    3. Start the Selenium Server by running the following command:
java -jar /usr/local/bin/selenium-server-standalone-2.35.0.jar

2. Installing PHPUnit_Selenium package

      The PHPUnit_Selenium package is necessary for natively accessing the Selenium Server from PHPUnit.
      To install it, run the following command:
pear install phpunit/PHPUnit_Selenium

Using Selenium in PHPUnit tests

There are two Selenium test cases:

    • PHPUnit_Extensions_Selenium2TestCase
    • PHPUnit_Extensions_SeleniumTestCase

PHPUnit_Extensions_Selenium2TestCase test case allows you to use the WebDriver API (partially implemented).

<?php
class WebTest extends PHPUnit_Extensions_Selenium2TestCase
{
    protected function setUp()
    {
        $this->setBrowser('firefox');
        $this->setBrowserUrl('http://www.example.com/');
    }

    public function testTitle()
    {
        $this->url('http://www.example.com/');
        $this->assertEquals('Example WWW Page', $this->title());
    }

}
?>

PHPUnit_Extensions_SeleniumTestCase test case implements the client/server protocol to talk to Selenium Server as well as specialized assertion methods for web testing.

<?php
require_once 'PHPUnit/Extensions/SeleniumTestCase.php';

class WebTest extends PHPUnit_Extensions_SeleniumTestCase
{
    protected function setUp()
    {
        $this->setBrowser('*firefox');
        $this->setBrowserUrl('http://www.example.com/');
    }

    public function testTitle()
    {
        $this->open('http://www.example.com/');
        $this->assertTitle('Example WWW Page');
    }
}
?>

Using different browsers with WebDrivers

For running tests with different browsers you should have WebDriver of that browser. As an example, we’ll try to use the Google Chrome browser and its WebDriver.

  1. Download WebDriver.
  2. To make Selenium Server aware of this WebDriver, perform one of the following tasks:
    • Store the Chrome WebDriver binary in the system path
    • Start the Selenium Server with -Dwebdriver.chrome.driver=path/to/your/chromedriver

You can now set Chrome as a browser for your functional test:

...
$this->setBrowser('chrome');
...

Running headless Selenium Server with Xvfb

Sometimes you don’t want the browser to be launched at your desktop during testing, or you may not be using Xserver. To make it work, one of the easiest solutions is to use Xvfb.

Xvfb is an X11 server that performs various graphical operations in memory, without displaying any screen output.

So, let’s try it. At first ensure that you have Xvfb installed on your server. If not, you can install it from your OS repository. For example, to install it on Ubuntu, run the following command:

sudo apt-get install xvfb

Once installed, to run your Selenium Server in Xvfb, run the following command:

DISPLAY=:1 xvfb-run java -jar selenium-server-standalone-2.35.0.jar

Once the server starts, run any of the test case examples. The output should display as follows:

PHPUnit 3.8.0 by Sebastian Bergmann.

F

Time: 4 seconds, Memory: 6.25Mb

There was 1 failure:

1) WebTest::testTitle
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'Example WWW Page'
+'Example Domain'

/your/project/Tests/Functional/WebTest.php:14

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

That’s all for now. Enjoy testing!

Source references:

  1. PHPUnit and Selenium (official documentation)
  2. Selenium Server downloads list

Automated testing with Selenium part II

We are starting with test automation and after a selection of tools; we decided to go with Selenium. An open source tool for user interface testing for web applications.

Features that are important in the Selenium tool

  •  Record & playback interface for quick scripting and low learning curve.
  •  The record & playback interface is working as a plug-in in Firefox.
  •  There is a plug-in for Record and Playback to do loops in IDE scripts, but is limited
  •  There is a plug-in for data driven testing
  •  Selenium is usable with open source programming platforms (such as Ruby or Python)

Combination with Python

For now within LeaseWeb we experimented with the Python development platform. Python is also open Source software, with a low learning curve. When the scripts are programmed (or imported from Record and Playback IDE interface) it can be used for more sophisticated automated testing. For example read data from a database and check this against web software.

When this is setup with Selenium webdriver you can do cross browser platform automated checking.

  • Cross browser and cross-platform
  • Performance testing by using basic test scenarios

Install Selenium IDE (the record and playback tool)
Go to http://seleniumhq.org/ and follow the instructions on that page for installing Selenium IDE
Install the Selenium webdriver

Get Python

  • Download Python from www.python.org (please install Python 2.x, not Python 3.x, this new version has little support yet)

Add Python to your PATH

  • Right click on “My Computer”.
  • Select “Properties” from the context menu.
  • In the “System Properties” dialog box, click on the “Advanced” tab.
  • Click on the “Environment Variables” button.
  • Highlight the “Path” Variable in “System variables” section.
  • Click the “Edit” button.
  • Append the following lines to the text inside the “Variable value” text box semi-colon delimited.
  • C:\Python25\;C:\Python25\Scripts\ (where 25 is the version number of your downloaded Python version)
  • Click “OK” on the “Edit System Variable” dialog box then “OK” on the “Environment Variables” dialog box to commit the changes.

Install setuptools
Download setuptools via http://pypi.python.org/pypi/setuptools#downloads
The file “setuptools-x.x.win32-pyx.x.exe (md5)” is an executable that will self extract and setup the setuptools.

Install Python
For the next step you’ll need Python 2 installed, which you can get from http://www.python.org/getit/. You’ll also need to install setuptools from http://pypi.python.org/pypi/setuptools.
Once you have these, you can run the following to install the Selenium python client library:
– easy_install pip
– pip install selenium

Support for browsers
At the time of writing this only the Firefox and Chrome webdrivers work (1st of November 2011). This is because of the webdriver is a new platform for Selenium. The old Selenium platform (Selenium RC and Selenium Grid) are used now in lots of companies, but within Leaseweb we decided to use the new one. This is something for the near future to experiment with within Leaseweb.
Install Google Chrome driver and Internet Explorer
If you want to play around with the Selenium Webdriver and some other browsers than Firefox, you will have to install the drivers for these web browsers.

  • Google Chrome: you should download the chrome driver and place in a system path (you could for example put the driver in the c:\pythonxx folder)
  • For Internet Explorer you will have to turn on ALL security zones: Turn ON protected mode in ALL Internet Explorer Zones (Security Tab in IE settings)

Conclusion
The new version of Selenium is better and more user friendly than the old version, but the compatibility with web browsers is not good enough yet. But for learning and starting with this tool it is good enough. Reasonably sophisticated scripts can be made in combination with Python as a programming environment, what makes it very flexible. Selenium is updated a lot, so in near future I guess all the internet browsers will work.

Subversion revision information in the Symfony debug toolbar

At LeaseWeb we are very serious about software development and that is why we hire software testers. These people do risk analysis, think of new test cases, execute them and organize them in test suites using TestLink. We also use PHP and are great fans of the Symfony MVC framework. Being serious about Agile software development means that we do TDD using unit tests and functional test. For this we use the Symfony build-in unit and functional tests. We also use Selenium for creating browser-driven test suites. The advantages of Selenium over the functional tests in Symfony are cross browser testing and JavaScript testing.

We have this product called “SSC” (this is the hosting control panel for the customers) and we test it thoroughly and often. For this product we have a DTAP environment to do testing and acceptance testing. Within the tests of this product the testers encountered a challenge: How do the testers know what revision of our application they are testing and what branch they are testing? This is especially a problem with acceptance testing where interactions between various systems are tested. Because the testers do not have command line access on the Linux machines that run the acceptance environment they cannot simply issue the “svn info” command like developers can.

To solve this problem we wrote a Symfony plugin called “lwTestingInformationPlugin”. It shows the “svn info” command output in a popup that is accessible from the Symfony debug toolbar. Even though it was extremely easy to write, it could be very useful. We made it available for download, so if you think it might be useful for your project as well, please try it and send us your feedback!

Click here to download the plugin (lwTestingInformationPlugin.tar)

TestLink = http://www.teamst.org/
Symfony = http://www.symfony-project.org
Selenium = http://seleniumhq.org/
Subversion = http://subversion.apache.org

Choosing the right tool for automated GUI testing

Within LeaseWeb we did our first steps in automated GUI testing and we use a tool called “Selenium” for our automated tests. Before choosing Selenium as an automated testing execution tool, we also checked:

– Fitnesse (http://fitnesse.org/)
– Rational Robot (http://www-01.ibm.com/software/rational/)
– Selenium IDE and webdriver (http://seleniumhq.org/)
– AutoIT3 (http://www.autoitscript.com/site/autoit/)

We preselected these tools for research, because they all pretend to have a low learning curve and could be (more or less) used out of the box. Also we wanted open source tooling. Rational is not open source, but was an alternative, because I used to work with the Rational tooling with good results in the past.

Fitnesse
Works with input from Wiki like pages, and very usable if you want to test a lot of data that comes from the business. So very usable for testing data that is going through the software from a business point of view. Fitnesse has no Record and Playback options (yet), so the learning curve is not that high for testers just starting with automating their tests. Maybe something to use in combination in the future.

AutoIT
This is more of a macro recorder. It has record and playback, but that is very limited and it clicks around on applications based on pixels on the screen. If you play a script from AutoIT on another PC with a different resolution or when a button is placed somewhere else, the script will fail. It is usable for quick recording, editing and playback when you need “throw away automation”. Nevertheless it has full programming capabilities, so can be used if you’re writing tests that go beyond the User interface.

Selenium
Selenium is the most well known automated test tool for web applications and I immediately could play around with the Record & Playback tool of this product. It was reasonably easy to export this to a programming language as “Python” and then the options for using it as a test tool becomes almost limitless.

IBM Rational Robot
I used Rational Robot a few years ago and tried to install the latest version. But I couldn’t get it to start within my time box for this research. I checked this after checking Selenium, so I quickly went back to Selenium as an automated GUI testing platform.

Conclusion
This was a post of the selection of an automated tool. We have chosen the Selenium platform because it is a web based automated GUI tool that can be used almost immediately with little knowledge of programming. It doesn’t stop there, because you can use a programming environment to adapt it to your needs. Also it has lots of plug-ins for the record and playback software (Selenium IDE) that make more sophisticated test scripts possible in Selenium IDE. I’ve heard of combination s of Selenium with the Fitnesse tool, so that could be a good combination in the future for data driven testing too. Open source, open platform. What’s not to like?

Just try it for yourself and see the possibilities.