Automated testing with Selenium part II

We are starting with test automation and after a selection of tools; we decided to go with Selenium. An open source tool for user interface testing for web applications.

Features that are important in the Selenium tool

  •  Record & playback interface for quick scripting and low learning curve.
  •  The record & playback interface is working as a plug-in in Firefox.
  •  There is a plug-in for Record and Playback to do loops in IDE scripts, but is limited
  •  There is a plug-in for data driven testing
  •  Selenium is usable with open source programming platforms (such as Ruby or Python)

Combination with Python

For now within LeaseWeb we experimented with the Python development platform. Python is also open Source software, with a low learning curve. When the scripts are programmed (or imported from Record and Playback IDE interface) it can be used for more sophisticated automated testing. For example read data from a database and check this against web software.

When this is setup with Selenium webdriver you can do cross browser platform automated checking.

  • Cross browser and cross-platform
  • Performance testing by using basic test scenarios

Install Selenium IDE (the record and playback tool)
Go to http://seleniumhq.org/ and follow the instructions on that page for installing Selenium IDE
Install the Selenium webdriver

Get Python

  • Download Python from www.python.org (please install Python 2.x, not Python 3.x, this new version has little support yet)

Add Python to your PATH

  • Right click on “My Computer”.
  • Select “Properties” from the context menu.
  • In the “System Properties” dialog box, click on the “Advanced” tab.
  • Click on the “Environment Variables” button.
  • Highlight the “Path” Variable in “System variables” section.
  • Click the “Edit” button.
  • Append the following lines to the text inside the “Variable value” text box semi-colon delimited.
  • C:\Python25\;C:\Python25\Scripts\ (where 25 is the version number of your downloaded Python version)
  • Click “OK” on the “Edit System Variable” dialog box then “OK” on the “Environment Variables” dialog box to commit the changes.

Install setuptools
Download setuptools via http://pypi.python.org/pypi/setuptools#downloads
The file “setuptools-x.x.win32-pyx.x.exe (md5)” is an executable that will self extract and setup the setuptools.

Install Python
For the next step you’ll need Python 2 installed, which you can get from http://www.python.org/getit/. You’ll also need to install setuptools from http://pypi.python.org/pypi/setuptools.
Once you have these, you can run the following to install the Selenium python client library:
– easy_install pip
– pip install selenium

Support for browsers
At the time of writing this only the Firefox and Chrome webdrivers work (1st of November 2011). This is because of the webdriver is a new platform for Selenium. The old Selenium platform (Selenium RC and Selenium Grid) are used now in lots of companies, but within Leaseweb we decided to use the new one. This is something for the near future to experiment with within Leaseweb.
Install Google Chrome driver and Internet Explorer
If you want to play around with the Selenium Webdriver and some other browsers than Firefox, you will have to install the drivers for these web browsers.

  • Google Chrome: you should download the chrome driver and place in a system path (you could for example put the driver in the c:\pythonxx folder)
  • For Internet Explorer you will have to turn on ALL security zones: Turn ON protected mode in ALL Internet Explorer Zones (Security Tab in IE settings)

Conclusion
The new version of Selenium is better and more user friendly than the old version, but the compatibility with web browsers is not good enough yet. But for learning and starting with this tool it is good enough. Reasonably sophisticated scripts can be made in combination with Python as a programming environment, what makes it very flexible. Selenium is updated a lot, so in near future I guess all the internet browsers will work.

Risk-based testing

Within LeaseWeb there is a lot of testing done on the software the customers use. Because it is impossible to test everything, we use a risk based strategy to focus on the most important parts.

Professionals in the IT know it: You can’t test everything. Even a simple web page with one edit box can be tested and looked at from many perspectives. Capitals, normal characters, strange characters, numbers, negative numbers, leave it empty, big arrays of texts, using other fonts, …. Just think about it and you will have a big checklist when you’re through.

What should happen then, if you push the [Submit] button? Is the data from the edit box placed in a database or a text file? How does the database look? Should the data be saved secure or is a text file enough? If it is saved in a text file, how to deal with strange characters. Or is that of no importance to the customer? Is it maybe more important that a textbox pops up with the text in it? Or should it be send to another device via mail or another communication protocol. And if it is sent to another device, what about security of the communication channel. Encryption? Is that important? Should we get feedback if something is sent?

If you don’t know what’s important for the user, on what part of the software should you focus? Security of saving or sending data, or should you focus on functionality and usability of the software?

Mostly we would only have a limited time for testing. So we will have to have a approach to focus testing on our software.

Change of failure and damage

To get focus on only the parts that probably have a big risk in failing or causing a lot of damage when it gets in production. This means talking to the right persons to discuss the risks and the priority of those risks.

No damage or low damage when something goes wrong? Little or no testing. Things could go wrong, but only once a year? But has big consequences and thus big damage. Lots of testing there!

So doing a risk analysis is getting focus on the parts that matter for the people that will use the software or otherwise are involved in this software (the stakeholders).

Risks have a chance of failure and height of damage

The technical people (developers, designers, testers) often can estimate the chance of failure. The damage for the business is better estimated by the business people (sales, support, and management etc).

Classification of risks

  • Risk = Failure Probability X Damage / R = F X D
    • Risk: The chance that the product will fail in relation to the expected damage
    • Failure probability: The chance that a system or component will fail within specified circumstances or within a specific time period
    • Damage: Prejudice, loss, damage, injury, money lost

Format of risks

About the format: I would expect a risk to have a cause and something happening because of that.

Examples:

–       Because the user doesn’t know how to select the right product, he will order something he doesn’t want

–       If more than 1000 users will try to order something, the website will be too slow to use.

Examples of aspects of failure

  • Frequency of use
  • Complicated or unknown product
  • New tooling
  • Development of product is taken over from others
  • Time pressure
  • Functions in subsystems are complex
  • Functions that have been changed a lot

Examples of damage

  • Loss of image of company or organization
  • Costs of correction actions, also extra documentation
  • Employees cannot do their work
  • Clients can’t be supported
  • The business processes are less efficient (or not efficient)
  • There is arising a negative image of the ICT organization

Conclusion

With this approach the most risky parts of the software are identified and a developer, functional designer and the tester can use this as a checklist for doing the quality part of the job.

In a next blog some tips for doing product risks sessions yourself if you like this idea.

Subversion revision information in the Symfony debug toolbar

At LeaseWeb we are very serious about software development and that is why we hire software testers. These people do risk analysis, think of new test cases, execute them and organize them in test suites using TestLink. We also use PHP and are great fans of the Symfony MVC framework. Being serious about Agile software development means that we do TDD using unit tests and functional test. For this we use the Symfony build-in unit and functional tests. We also use Selenium for creating browser-driven test suites. The advantages of Selenium over the functional tests in Symfony are cross browser testing and JavaScript testing.

We have this product called “SSC” (this is the hosting control panel for the customers) and we test it thoroughly and often. For this product we have a DTAP environment to do testing and acceptance testing. Within the tests of this product the testers encountered a challenge: How do the testers know what revision of our application they are testing and what branch they are testing? This is especially a problem with acceptance testing where interactions between various systems are tested. Because the testers do not have command line access on the Linux machines that run the acceptance environment they cannot simply issue the “svn info” command like developers can.

To solve this problem we wrote a Symfony plugin called “lwTestingInformationPlugin”. It shows the “svn info” command output in a popup that is accessible from the Symfony debug toolbar. Even though it was extremely easy to write, it could be very useful. We made it available for download, so if you think it might be useful for your project as well, please try it and send us your feedback!

Click here to download the plugin (lwTestingInformationPlugin.tar)

TestLink = http://www.teamst.org/
Symfony = http://www.symfony-project.org
Selenium = http://seleniumhq.org/
Subversion = http://subversion.apache.org

Choosing the right tool for automated GUI testing

Within LeaseWeb we did our first steps in automated GUI testing and we use a tool called “Selenium” for our automated tests. Before choosing Selenium as an automated testing execution tool, we also checked:

– Fitnesse (http://fitnesse.org/)
– Rational Robot (http://www-01.ibm.com/software/rational/)
– Selenium IDE and webdriver (http://seleniumhq.org/)
– AutoIT3 (http://www.autoitscript.com/site/autoit/)

We preselected these tools for research, because they all pretend to have a low learning curve and could be (more or less) used out of the box. Also we wanted open source tooling. Rational is not open source, but was an alternative, because I used to work with the Rational tooling with good results in the past.

Fitnesse
Works with input from Wiki like pages, and very usable if you want to test a lot of data that comes from the business. So very usable for testing data that is going through the software from a business point of view. Fitnesse has no Record and Playback options (yet), so the learning curve is not that high for testers just starting with automating their tests. Maybe something to use in combination in the future.

AutoIT
This is more of a macro recorder. It has record and playback, but that is very limited and it clicks around on applications based on pixels on the screen. If you play a script from AutoIT on another PC with a different resolution or when a button is placed somewhere else, the script will fail. It is usable for quick recording, editing and playback when you need “throw away automation”. Nevertheless it has full programming capabilities, so can be used if you’re writing tests that go beyond the User interface.

Selenium
Selenium is the most well known automated test tool for web applications and I immediately could play around with the Record & Playback tool of this product. It was reasonably easy to export this to a programming language as “Python” and then the options for using it as a test tool becomes almost limitless.

IBM Rational Robot
I used Rational Robot a few years ago and tried to install the latest version. But I couldn’t get it to start within my time box for this research. I checked this after checking Selenium, so I quickly went back to Selenium as an automated GUI testing platform.

Conclusion
This was a post of the selection of an automated tool. We have chosen the Selenium platform because it is a web based automated GUI tool that can be used almost immediately with little knowledge of programming. It doesn’t stop there, because you can use a programming environment to adapt it to your needs. Also it has lots of plug-ins for the record and playback software (Selenium IDE) that make more sophisticated test scripts possible in Selenium IDE. I’ve heard of combination s of Selenium with the Fitnesse tool, so that could be a good combination in the future for data driven testing too. Open source, open platform. What’s not to like?

Just try it for yourself and see the possibilities.