The myth of the efficiency of colocation in IT was created by software engineers from the First World to protect their wage level against programmers from India, China, Brasil, …
To protect the myth against critical reviews, they injected it into the sacrosanct frameworks of Scrum, XP …
Ironically the evangelists of colocation love Selenium, *the* framework for browser automation. But the Selenium project has no location – committers, contributers and supporters are spread all over the world and they colaborate in a chatroom based on antique technologies.
Let’s hope that the wage level in the developing countries rises soon …
… it’s just polemics, don’t ask me for evidence.
The question of “which test cases are to automate” is a core question in a QA-project. To answer this question properly means saving a lot of money.
In my experience there are four core criteria to help you answer this question properly. I call them the STEW criteria – on the one hand it’s an acronym for the criterias and on the other hand it’s paying attribute to Simon STEWart, the creator of Selenium Webdriver. It’s really simple and nothing more than:
Stable: Don’t waste time on automating regression tests for features, that are planned yet for redesign in the next sprint.
Troubling: Choose to automate test cases for features that are often failing in regression tests or that frequently trouble the build process.
Easy: Choose test cases, that are easy to automate, compared to manual testing. E.g. don’t automate visual comparision of bleeding edge grafics.
Worth: Don’t automate test-cases that aren’t worth it in respect to business goals. E.g. a test-case for a special feature of an enterprises local website in Cyprus isn’t worth it – sorry to all Cypriot readers.
Managers want new shiny features on their webapp, and they want them tomorrow. Yes, you can produce them over night. But doing that too often will produce unmaintainable code, it will draw you day-by-day deeper in the regression hell. Scrum-people invented the Defintion of Done to prevent such a shortsighted coding. This is their weapon in the daily struggle against the management pressure: “hey, managers, if you want us to be able to add new features also in half a year, you shouldn’t be too happy, if you see your lovely new feature working well today on the webapp. You should be aware, that a new feature is only done, if it meets defined quality criteria.” And yes, it takes time to write a unit test.
What is true for WebDevs (web developers) is also true for TestAuts (functional test automater). One DoD-criteria for WebDevs is writing unit tests that covers the code to a certain degree. Do we, as TestAuts, need some tests to check our functional tests, too? – what a mind game! No, because actually the website is our “test”. If we take the latest stable release of the website and all our functional tests are green, the “test” of our functional tests passed.
[Excursion: To be more precise, the above statement is true for regression testing. The aim of regression testing is to prevent regression of the web-app. (Another word for regression is step backwards.) So regression testing is testing of yet created and released features with the goal to keep these old features stable. You have the features yet and they are stable – use them to “test” your feature tests! Obviously the above statement is false, if you’re brave and try to parallely develop the feature and its feature tests, or you try to write the feature tests before its feature.]
A big difference between web development and functional test automation is the criteria for “it works”. In web development code can be considered as “it works” if it ran once successfully (simplification!). In browser automation you may have race-conditions between the webapp and your testprogram. You can run the functional tests against a webapp once and it passes. But e.g. on the fifth run the test failed without any code-change neither in your webapp nor in your testprogram. The reason may be, that you put a sleep of 5000 ms in your testprogram (compare explicit waits). As soon as the webapp is getting slower to some reason, your testprogram gets the control back too early and the test fails.
1.) A functional test is done, only if it passes X times against the latest stable release of the webapp.
The WebDevs should produce a webapp, that is running on more than one browser-OS-combo. (Have a look on your web analytics to find the mostly used browser-OS-combos of your visitors.) Ok, but how to ensure that? Have a look at Sauce Labs, there you find over 100 browser-OS-combos. Obviously with manual testing it isn’t possible to cope with such a number – you need automated tests.
2.) A functional test is done, only if it passes with a well-defined list of browser-OS-combos.
Most of companies that can afford a test automation engineer use their platform in many countries with many local specifics.
3.) A functional test is done, only if it passes on every national website.
Every team member should be able to read and work on your code easily. Therefore you need to follow your team’s Coding Conventions. To ensure that, you can use Code Review, Pair Programming and/or tools for static code analysis
4.) A functional test is done, only if it meets the Coding Conventions.
An automated test, that only runs on your local machine has little worth for others. Use a (CI-)Server to show the results to whom it may concern.
5.) A functional test is done, only if it passes on the (CI-)Server.