You can’t buy right tool if you are not sure why you need it. The article begins with introduction to various levels of maturity in test automation, pros and cons of each, the grey areas – what is the status quo, what capabilities do you really need, what is the true cost of owning and using a tool etc. The article will then take the users through some factors – technical, operational as well as behavioral to help them get them the clarity they need to make a choice.
1. Overview :
Getting test automation right is becoming increasingly critical for success of businesses. Moving from manual testing to automated testing is a big change in itself. What makes this change even more difficult is the confusion around test automation. If an organization decides to move towards test automation, the first step itself is critical as well as challenging – selecting a right test automation tool.
The test automation market is crowded with a large number of players – of different sizes, capabilities and technologies, each of them claiming to be the ‘best test automation tool’. If one is looking to move to test automation, how should one identify which tool is the right one for them?
Test automation can have a large number of associated overheads – salary, training costs, licenses etc. To select the ‘right’ test automation tool for one’s needs, cutting through all the noise on the internet is very critical. In this article, we try to clear the sky for you so that you can find the best test automation tool for your requirement.
2. Different types of automation technics/tools/methodologies
When we talk about test automation, we look at it as a singular object. In reality, test automation is actually a spectrum. Depending upon how much and which part of the testing process is automated, there can be multiple levels of maturity among the test automation itself. There are different tools which are designed for different levels of maturity.
For instance, at its very beginning, test automation could be limited to automating the UAT test cases through scripting methodologies. Test automation at this stage has its benefits as well as
disadvantages. The capital investment is low but managing changes in the application or automating complex scenarios is impossible. At a slightly advanced stage, where the coverage for automated tests ranges from 30%-40%, one gets higher reusability and maintainability but at this stage, the test automation will be heavily dependent on skilled resources which will increase the operational cost. The next step in the maturity of test automation is achieved when the mature framework scales across the applications. At this stage, reusability is even higher but manual test engineers cannot contribute to the testing process.
The most advanced level of test automation is scriptless test automation where highly intuitive GUI acts like an interface between the framework and the users. Scriptless test automation can scale across complete QA teamed can handle any complex testing scenario. Scriptless test automation provides highest test coverage and maximum ROI.
While evaluating different test automation tools, it is important to see where do they fit in this spectrum of test automation maturity.
3. What makes the selection difficult?
What makes selecting a test automation tool a confusing and tiring process? The reasons are many and understanding them can simplify the process considerably.
Lack of awareness of status quo : Knowing what going right and what’s not with the existing testing setup is very critical when it comes to selecting a new tool. What are the present challenges? What is the quality of present releases? What is the quantum of efforts required for reporting and maintenance of scripts? Is finding the right talent a constraint?
Getting clarity on some of these questions will be helpful as it will give you a concrete list of reasons for which you want to automate. The clear ‘why’ will significantly help in selecting the right test automation tool.
Multiple solutions talking almost same language : As we mentioned earlier, it is really confusing to to select a right tool because every tool talks the same language. Every tool claims to reduce your time to automate by X% and automation cost by Y%. Does that mean it will do so for you as well? It is very important to understand what was the size of organizations that used these tools and what were their costs and time earlier.
Getting a tool purely based on the copy on the website is a big mistake. Insist for a full product demo. Ask very pointed questions about the claims that are being made. That will clear the smoke immediately.
Claimed capabilities vs actual capabilities : Almost every tool claims to be powered by AI and ML. Do these features translate into tangible benefits like reduction of human effort, improved quality?
Existence of a feature vs its usability : More options do not necessarily mean more freedom and capabilities. For safe maneuvering, a car only needs four wheels. Less or more would be detrimental to the performance. When it comes to test automation, what all and how
much do you actually need? The ‘feature loaded’ tools can cost more than you need and can induce learning curve and fatigue in the users.
Arriving at true cost of ownership is not straight forward : Total cost of automation can vary significantly from the cost of your automation tool. Tool is just one component. It important to look at other cost considerations like salaries of engineers, training costs specific to the tool, other tools/licenses required, maintenance costs etc. Before making a decision, getting full clarity about the one time and recurring costs is important. Many tools will have a complex pricing which will make getting the clear picture very difficult.
4. Factors to consider while choosing the tool :
To select the right test automation tool, one should think about following factors carefully :
Roadmap : What are your automation priorities? Out of 50,000 test cases, which 5,000 cases you want to automate first? Does the tool fulfill the requirements and provide the scalability that you would eventually need?
Change matrix : What all will change when you move from your existing method to the tool you are going to select? Do you need people with new skillset? How long will it take to train the existing team? What will happen to ongoing projects? Can they seamlessly onboard the new tool? Getting a clear idea of efforts required
User experience of the tool : A tool benefits only when people use it. No one wants another automation effort that everyone abandons halfway like a sinking ship. How easy to use the tool you are considering is? Talk to people who are already using the tool you are considering.
All encompassing pricing : Not every solution provider will give you all encompassing prices. Ask pointed questions to arrive at the actual cost of owning, using and maintaining the tool.
Support : No matter how perfect any tool is, there will always be that moment when you need someone to get you out of a jinx quickly. Almost every provider would claim that they have exceptional support. The only way to find out is actually talking to their customers; not just one or two, but at least five to six.
Knowing the people : If you like the people who have built a product, there are far more chances that you will end up liking the tool. Go beyond the tool, understand the team that built it and why did they build the tool. Try to gauge how committed they are to keep the end user satisfied with the product.
4. Summary :
Selecting a test automation tool is difficult and confusing. Begin with understanding the existing challenges and the priority in which you want to overcome them. Don’t get sold by just reading the website. Talk to people, talk to customers and ask them specific questions. Finally, look beyond the tool. Try to know the people behind it as that will ensure that you are not just buying a tool but an experience