Definition of Done (DoD) for Testautomation

Managers want new shiny features on their webapp, and they want them tomorrow. Yes, you can produce them over night. But doing that too often will produce unmaintainable code, it will draw you day-by-day deeper in the regression hell. Scrum-people invented the Defintion of Done to prevent such a shortsighted coding. This is their weapon in the daily struggle against the management pressure: „hey, managers, if you want us to be able to add new features also in half a year, you shouldn’t be too happy, if you see your lovely new feature working well today on the webapp. You should be aware, that a new feature is only done, if it meets defined quality criteria.“ And yes, it takes time to write a unit test.
What is true for WebDevs (web developers) is also true for TestAuts (functional test automater). One DoD-criteria for WebDevs is writing unit tests that covers the code to a certain degree. Do we, as TestAuts, need some tests to check our functional tests, too? – what a mind game! No, because actually the website is our „test“. If we take the latest stable release of the website and all our functional tests are green, the „test“ of our functional tests passed.
[Excursion: To be more precise, the above statement is true for regression testing. The aim of regression testing is to prevent regression of the web-app. (Another word for regression is step backwards.) So regression testing is testing of yet created and released features with the goal to keep these old features stable. You have the features yet and they are stable – use them to „test“ your feature tests! Obviously the above statement is false, if you’re brave and try to parallely develop the feature and its feature tests, or you try to write the feature tests before its feature.]
Timing Issues
A big difference between web development and functional test automation is the criteria for „it works“. In web development code can be considered as „it works“ if it ran once successfully (simplification!). In browser automation you may have race-conditions between the webapp and your testprogram. You can run the functional tests against a webapp once and it passes. But e.g. on the fifth run the test failed without any code-change neither in your webapp nor in your testprogram. The reason may be, that you put a sleep of 5000 ms in your testprogram (compare explicit waits). As soon as the webapp is getting slower to some reason, your testprogram gets the control back too early and the test fails.
1.) A functional test is done, only if it passes X times against the latest stable release of the webapp.
X-Browser-Testing
The WebDevs should produce a webapp, that is running on more than one browser-OS-combo. (Have a look on your web analytics to find the mostly used browser-OS-combos of your visitors.) Ok, but how to ensure that? Have a look at Sauce Labs, there you find over 100 browser-OS-combos. Obviously with manual testing it isn’t possible to cope with such a number – you need automated tests.
2.) A functional test is done, only if it passes with a well-defined list of browser-OS-combos.
I18n
Most of companies that can afford a test automation engineer use their platform in many countries with many local specifics.
3.) A functional test is done, only if it passes on every national website.
Coding Conventions
Every team member should be able to read and work on your code easily. Therefore you need to follow your team’s Coding Conventions. To ensure that, you can use Code Review, Pair Programming and/or tools for static code analysis
4.) A functional test is done, only if it meets the Coding Conventions.
Reporting
An automated test, that only runs on your local machine has little worth for others. Use a (CI-)Server to show the results to whom it may concern.
5.) A functional test is done, only if it passes on the (CI-)Server.
tbc