Archiv der Kategorie: Testautomatisierung

To automate or not to automate

The question of „which test cases are to automate“ is a core question in a QA-project. To answer this question properly means saving a lot of money.
In my experience there are four core criteria to help you answer this question properly. I call them the STEW criteria – on the one hand it’s an acronym for the criterias and on the other hand it’s paying attribute to Simon STEWart, the creator of Selenium Webdriver. It’s really simple and nothing more than:
Stable: Don’t waste time on automating regression tests for features, that are planned yet for redesign in the next sprint.
Troubling: Choose to automate test cases for features that are often failing in regression tests or that frequently trouble the build process.
Easy: Choose test cases, that are easy to automate, compared to manual testing. E.g. don’t automate visual comparision of bleeding edge grafics.
Worth: Don’t automate test-cases that aren’t worth it in respect to business goals. E.g. a test-case for a special feature of an enterprises local website in Cyprus isn’t worth it – sorry to all Cypriot readers.
That’s it!

Definition of Done (DoD) for Testautomation

Managers want new shiny features on their webapp, and they want them tomorrow. Yes, you can produce them over night. But doing that too often will produce unmaintainable code, it will draw you day-by-day deeper in the regression hell. Scrum-people invented the Defintion of Done to prevent such a shortsighted coding. This is their weapon in the daily struggle against the management pressure: „hey, managers, if you want us to be able to add new features also in half a year, you shouldn’t be too happy, if you see your lovely new feature working well today on the webapp. You should be aware, that a new feature is only done, if it meets defined quality criteria.“ And yes, it takes time to write a unit test.
What is true for WebDevs (web developers) is also true for TestAuts (functional test automater). One DoD-criteria for WebDevs is writing unit tests that covers the code to a certain degree. Do we, as TestAuts, need some tests to check our functional tests, too? – what a mind game! No, because actually the website is our „test“. If we take the latest stable release of the website and all our functional tests are green, the „test“ of our functional tests passed.
[Excursion: To be more precise, the above statement is true for regression testing. The aim of regression testing is to prevent regression of the web-app. (Another word for regression is step backwards.) So regression testing is testing of yet created and released features with the goal to keep these old features stable. You have the features yet and they are stable – use them to „test“ your feature tests! Obviously the above statement is false, if you’re brave and try to parallely develop the feature and its feature tests, or you try to write the feature tests before its feature.]
Timing Issues
A big difference between web development and functional test automation is the criteria for „it works“. In web development code can be considered as „it works“ if it ran once successfully (simplification!). In browser automation you may have race-conditions between the webapp and your testprogram. You can run the functional tests against a webapp once and it passes. But e.g. on the fifth run the test failed without any code-change neither in your webapp nor in your testprogram. The reason may be, that you put a sleep of 5000 ms in your testprogram (compare explicit waits). As soon as the webapp is getting slower to some reason, your testprogram gets the control back too early and the test fails.
1.) A functional test is done, only if it passes X times against the latest stable release of the webapp.
The WebDevs should produce a webapp, that is running on more than one browser-OS-combo. (Have a look on your web analytics to find the mostly used browser-OS-combos of your visitors.) Ok, but how to ensure that? Have a look at Sauce Labs, there you find over 100 browser-OS-combos. Obviously with manual testing it isn’t possible to cope with such a number – you need automated tests.
2.) A functional test is done, only if it passes with a well-defined list of browser-OS-combos.
Most of companies that can afford a test automation engineer use their platform in many countries with many local specifics.
3.) A functional test is done, only if it passes on every national website.
Coding Conventions
Every team member should be able to read and work on your code easily. Therefore you need to follow your team’s Coding Conventions. To ensure that, you can use Code Review, Pair Programming and/or tools for static code analysis
4.) A functional test is done, only if it meets the Coding Conventions.
An automated test, that only runs on your local machine has little worth for others. Use a (CI-)Server to show the results to whom it may concern.
5.) A functional test is done, only if it passes on the (CI-)Server.

Dreamteam: Selenium WebDriver and FitNesse

This year’s fantastic Hackathon at in Berlin, offered me a productive environment (Club-Mate, pizza and really nice talks to really smart people) to dive deeper in the wiring of Selenium and FitNesse. And here we go:
Read this article to find an overview of testing-frameworks and their use for acceptance testing.
Read Specification by Example, as FitNesse is a cool tool for that approach.
Read my guide to setup Selenium.
Read my guide to setup FitNesse.
My primary goal, that I want to accomplish with any acceptance-test-framework: I want the non-programmers (managers, product owner, …) to understand which tests are implemented for their web-application yet – and which tests are still missing. I want to be sure, that the tests I (as a test-automater) have in my mind, really meet the requirements. The responsibles for the business-logic of a web-application should be able to check that at a glance. Additionally I like to give them a tool to easily change/enhance the test-data and simply push a test-button to get immediate feedback from their web-application (or its staging page). And yes, I’m really talking about automated functional tests with Selenium WebDriver.
I’m presenting a really simple example to just show one possible wiring between Selenium and Fitnesse (without using Page Object-Pattern and just running Selenium locally).
In the specification we have everything lovely included: descriptive text, nice pictures/drafts of your feature and you have a button that executes the specifiation of that feature = the feature-test:
This pretty specification was created with that piece of code:

!*< [configurations]
!define TEST_SYSTEM {slim}
!path C:\Users\IT Kosmopolit\Dropbox\IT Kosmopolit\Blogging\Fitnesse
!path C:\Users\IT Kosmopolit\Dropbox\IT Kosmopolit\Blogging\Fitnesse\selenium-server-2.28.0\selenium-2.28.0\selenium-server-standalone-2.28.0.jar
Test if the page title is displayed correct (case-sensitive), after search for "Cheese!"
|Test Google page title|

With !*< [configurations] I start an invisible block.
With !define TEST_SYSTEM I select the engine.
With !path I define a classpath.
|import| works like an import statement
Simple text doesn’t need any mark-up – just write it down.
!img brings us an image to the wiki-page.
And with |something| you start a table.
To dive deeper into the subject: In the TwoMinuteExample you find general information about the decision table, a main concept in FitNesse. In the MarkupLanguagReference you find more information about the markup I’ve used in the wiki-page:
If you like to have this nice specification on your computer, too:
1.) execute „localhost“ in the address-bar (be sure to have set up FitNesse on your computer)
2.) push the Add-button
3.) select Test page
4.) write GoogleBasicTest for the Page name
5.) delete the default in the code box (!contents -R2 -g -p -f -h)
6.) copy&paste the above code in the code box
7.) click save
You can find your page under: http://localhost/GoogleBasicTest now.
Note: Be sure to name your future wiki-pages as WikiWords.
Now you have a cool specification, but it’s not doing anything valuable, if you push the Test-button. FitNesse uses so-called fixtures to wire a wiki-page with the Application Under Test. I appreciate the decision table as a plain, clear, noise-less way of defining all the scenarios of a feature (*waving at Gherkin*) – so find the appropriate description here. Note: Every row in the decision table represents one scenario.
Go to the folder in which your fitnesse-standalone.jar resides. Create there the folder-structure: de\itkosmopolit\fixtures, open an editor, copy&paste the following code in it and save it with the name (compare this name with the title of our decision table!)

package de.itkosmopolit.fixtures;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import java.util.List;
public class TestGooglePageTitle {
  private String text;
  private String pageTitle;
  public void table(List
&amp;lt;list&amp;lt;string&amp;gt;&amp;amp;gt; table) throws InterruptedException{
		// Create a new instance of the html unit driver
        // Notice that the remainder of the code relies on the interface,
        // not the implementation.
        WebDriver driver = new FirefoxDriver();
        // And now use this to visit Google
        // Find the text input element by its name
        WebElement element = driver.findElement("q"));
        // Enter something to search for
        // Now submit the form. WebDriver will find the form for us from the element
        // Check the title of the page
        pageTitle = driver.getTitle();
  public void setTextInTitle(String text) {this.text = text;}
  public String isTitleCorrect() {
		if (pageTitle.contains(text)) return "yes";
		else return "no";

1.) Putting the Selenium code in the table-method feels a bit … hack-y, but … it works.
2.) Because of black magic, you need a Thread.sleep() – otherwise while executing you’d get the wrong(!) error-message: „Testing was interupted and results are incomplete.“
Then compile it, e.g. with your Command Prompt: (being in the folder where fitnesse-standalone.jar resides) javac -classpath "pathToSeleniumJAR\selenium-server-standalone-x.y.z.jar" de\itkosmopolit\fixtures\
And after pushing the test button you find this result:
The results are beautiful and I’m quite confident, that Selenium WebDriver & FitNesse are becoming a new dreamteam in the future. Anyhow consider, that I haven’t seen this aproach running on large projects yet – there may be some challenges …