|
| Software Test Automation - Facts (Realities) and Fictions (Myths) !!! |
| Tuesday, December 12, 2006 |
Test automation is a gimmick, most QA/Test managers use, to divert attention from serious issues of a project. This is what I have realised about test automation in all these years. Believe me, I am saying this from my past experience. If you are also in software testing/QA field then you might have noticed this too. Whenever a projects fails to meet its deadline or it fails to meet the expected quality, then only test automation is considered as an option. As if by using this magic bullet, we could attain quality over night ! As the requirements keep changing (mostly new requirements are getting introduced on daily basis) constantly and the testing window is getting smaller and smaller everyday, the managers are realizing a greater need for test automation. This is good news for us (people who do test automation). But, I am afraid this is the only good news.
Myths & Facts
A number of articles and books are written on different aspects of Software Test Automation. "Test Automation Snake Oil" by, James Bach is an excellent article on some of the myths of automation. I like to discuss some of these myths and will try to point out the facts about these myths. I also like to discuss some of my observations and hopefully point out possible solutions. These are based on my experience with a number of automation projects I was involved with.
Find more bugs: Some QA/Test managers think that by doing automation they should be able to find more bugs. It's a myth. Let's think about it for a minute. The process of automation involves a set of written test cases. In most places the test cases are written by test engineers who are familiar with the application they are testing. The test cases are then given to the automation engineers. In most cases the automation engineers are not very familiar with the test cases they are automating. From test cases to test scripts, automation does not add anything in the process to find more bugs. The test scripts will work only as good as the test cases when comes to finding bugs. So, it's the test cases that find bugs (or don't find bugs), not the test scripts.
Eliminate or reduce manual testers: In order to justify automation, some point out that they should be able to eliminate or reduce the number of manual testers in the long run and thus save money in the process. Absolutely not true. Elimination or reduction of manual testers is not any of the objectives of test automation. Here is why - as I have pointed out earlier that the test scripts are only as good as the test cases and the test cases are written primarily by manual testers. They are the ones who know the application inside out. If the word gets out (it usually does) that the number of manual testers will be reduced by introducing automation then, most if not all manual testers will walk out the door and quality will go with them as well.
Observations
I have met a number of QA/Test managers who are frustrated with their automation. According to them the tool is not doing what it is supposed to do. A manager of one of the major third party testing companies that I had a recent interview with told me that after four years and more than a million dollar he is still struggling with automation. This is pretty sad and I get the feeling that he is not alone.
Solutions/Suggestions
Let's discuss some of the reasons for this frustration and some of the solutions to this problem.
Unrealistic expectations: Most managers have their first encounter with any automation tool when they look at the demo and everything looks nice and simple. But everything is not so nice and simple when you try to use the tool with your application. The vendors will only tell you the things you want to hear (how easy to use, how simple to set up, how it will save time and money, how it will help you find more bugs etc.). This builds a false set of hopes and expectations. Anyway we can not blame the vendor for this, as he is only trying to sell his tool. So he is supposed to say some good words abt his product !
Lack of planning: A great deal of planning is required from selection to implementation of the tool. The best way is to involve the "Tool Audience" (the testers who are going to use the tool ultimately). It may not happen in every place because of the everyday workload of the people involved. But the participation of the users in the process is very important, because they are the ones who will use the tool day in and day out. I am almost certain that what happened to one of my clients (the tool they have bought did not support the application they were testing) would not have happened if the users were involved in the selection process.
So we should do a market study and some kind of research before purchasing any tool. We should purchase that tool which best suits our requirement and best fits our environment. If we are not able to find such a tool then it always better to deveop our own in-house test automation tool (many top companies are doing that exactly). By this we can also save money in terms of license renewal fee.
Lack of a process: Lack of a process may also contribute to failure of automation. Most places do have some kind of process in place. In most cases (although it differs from place to place) developers write code against a set of requirements. If the requirement does not call for a change in GUI then, there should not be any change in GUI. But if the GUI keep changing constantly from one release to another without any requirement for that change then, there is a problem in the process. You may have the best tool and the best (for your environment) architecture is in place and you will still have problems with your automation because of a faulty process.
Conclusion
I think there is a need to educate QA/Test managers about the benefits and limitations of automation. There is a need to separate the facts from the fictions. But here is the problem, in most cases consultants are brought in to fix problems of prior attempts instead of initial setup. At this point the managers have already learned (painfully) the pitfalls of automation. In order to avoid this painful experience I would recommend (most automation engineers will agree with me) to spend more time up front doing research about the styles and techniques of automation and find an architecture that fits the environment. There is no doubt that automation adds a great value to overall QA process but, short of knowledge and understanding about automation and lack of planning can also cause a nightmare. So wake up guys (I mean QA/Test managers) and try to realise the difference between facts and fictions. Then only test automation can become a SUCCESS STORY!
|
Posted @ Tuesday, December 12, 2006   |
|
|
Back to Top |
| Bookmark This Site |
|
|
|
|
|
