Conquering Common Test Automation Challenges
With over twenty-five years in IT I’ve had the good fortune to have provided consulting, training, and mentoring for over 240 different organizations. This has exposed me to numerous environments, experiences, and, of course, challenges. I started my career as a DB2, IBM, COBOL programmer. (How many of you remember the term “spaghetti code”?) I quickly transitioned over to designing and developing client/server applications and before I knew it the wonderful world of web applications and the internet was upon us. And now we have smart phones and tablets. We still “click, drag, and drop” using a mouse but now we also “pinch and flick” with our fingers. So many changes in such a relatively short period of time. But, I digress.
In the early nineties the science, and art, of software testing started to evolve. The need for a structured process to perform verification and validation became evident. Why? It could have been for a number of reasons: the understanding that both robust quality control and quality assurance processes play a critical role in ensuring quality, the common realization that resolving an issue while in development is less expensive than resolving the issue once in production, the fact that end-users are more knowledgeable in the use of technology (and becoming more so every day), the need to test with a greater number of conditions thus increasing the amount of test cases and test data, and the list goes on. These contributed to an increased popularity of test automation solutions – both in-house and COTS. The benefits of test automation became evident…as did the challenges. When I discuss these challenges people are often surprised that they’re not more “technically advanced” in nature. They truly are more strategic in nature; more focused on the implementation and maintenance approach. I’d like to share some of the most common challenges I see organizations facing in today’s ever-changing IT environment.
- Organizations fail to define a selection process when evaluating test automation solutions. Many have a tendency to start defining the selection process as its occurring. That’s like writing your requirements as you’re testing. You’re almost sure to pass. Some steps to follow:
- Beforeyou start selecting solutions define the needs and objectives you’re looking to address.
- Are you looking for ways to increase productivity and realize a significant, pre-determined return-on-investment?
- Are you focused on improving your organization’s overall testing processes?
- Is your organization faced with technology or a process that can only be tested using an automation solution?
- Define the selection criteria and requirements that will enable you to meet your needs and objectives.
- What are essential (required) features?
- What are the optional (nice-to-have) features?
- What types of testing? Application security? Functional? Performance?
- What technologies must the solution be able to support? Is it required that add-ins be purchased separately to support certain technologies?
- Do any requirements need to be defined based on the skill set of your existing team? Perhaps you have a number of experts with VBScript. Should that be a requirement as to what scripting language the solution uses?
- How important is the availability of active user groups for any solution you choose?
- Have you researched the availability of resources in the job market, in your local area, with expertise in a solution?
- Evaluate tools and perform a proof-of-concept
- Before you start your evaluation get your vendor to provide some mentoring so that you can perform a better, more educated evaluation. Understand that most solutions are robust and the full training may not be practical (for you or the vendor) prior to the evaluation. Some assistance with the installation and a few hours of remote training should be enough to get you up and going.
- Define length of evaluation period upfront. Commit and be focused and ensure your evaluation team understands the requirements.
- Be sure to use the tool in your environment and on your terms – within reason. For the majority of solutions a 2 – 4 week evaluation should be more than adequate. If the vendor provides a solution expert to work with your resource(s) during the evaluation than often 1 – 3 days is more than enough (if those days are dedicated to the proof-of-concept).
- Identify the level of vendor support during evaluation period and communicate this to the vendor. Depending on the number of licenses you’re looking to purchase this should not be an issue.
- Evaluation Criteria
- Capabilities: Does it include all required features?
- Reliability: Does it consistently work as expected with minimal configuration changes?
- Ease of use: Is training required? How available is training? What’s the cost?
- Compatibility: Is the test automation solution compatible with your existing environment, platforms, and application?
- Robustness and scalability: Can it be used with a variety of technologies with no, or minimal, customizations and configuration changes?
- Non-intrusiveness: How accurately does it simulate real-world, actual use?
- Product and vendor track record: What’s your research tell you? What’s the “word on the street”?
- There are often unrealistic expectations. These unrealistic expectations are typically in regard to:
- Benefits – Management’s ROI (return-on-investment) objectives do not get calculated correctly. Immediate and long-term improvements to the testing process are frequently overlooked. Too much weight is applied to savings in man-hours. I’ve never known an organization to reduce their work force upon implementing test automation. Rather they correctly use test automation to maintain the size of their team and start validating a greater number of conditions.
- What it takes to get started – If your team has never used the specific test automation tool you’re implementing then don’t think for a minute that they won’t need training or some form of education. And keep in mind that there’s education on how to use the test automation tool AND there’s education on test automation methodologies (strategies, best practice, approaches, etc.). Remember, the old adage holds true – “you don’t know what you don’t know”.
- What it takes to maintain a successful test automation implementation – Yes, there’s maintenance. There’s maintenance in regard to the people (the users) and the technology. As new versions of the test automation tool are released, as new ideas around methodologies evolve, and as technology itself changes – people must be educated on an on-going basis. As your applications change the automated tests typically will involve some level of maintenance as well –especially if those changes involve the interface (the addition or removal of objects, the way objects are defined, changes to the navigation throughout the application, etc.). When implementing test automation and tracking your ROI it’s important to allocate time for maintenance.
- Organization’s often lack an automation process. This approach then can lead to the either extreme of “automating everything” or “automating nothing”. It can lead to wasted time (by automating processes that shouldn’t be automated, and then having to maintain those automated processes), frustration (in that team members don’t see the success they expect), unrealized benefits (by notautomating processes that can and should be automated), and so on. A strategic process includes, at a minimum, the following:The development of an overly complex automation framework is a challenge I’ve seen a great deal. I know of one project where, over the course of 6 months during the creation of the automation framework, not a single automated test had been executed! Not one! Don’t misunderstand. I’m a firm believer in frameworks. A modular, robust framework speeds up automation development time and significantly decreases maintenance. It minimizes the learning curve for your team. It provides incredible benefits. But there’s no reason that an organization cannot reap the benefits of test automation while the framework is being developed. I emphasize to my clients to develop a strategy that allows the framework to evolve! Start off by identifying a specific number of tests and processes that, once automated, can be executed immediately. This allows quick realization of the benefits. As you start automating (and executing) you will quickly start identifying common routines and processes that can be extracted and easily redefined as common, modular routines. Before you know it your framework is in place.
- A detailed description of the framework with timelines for the development and evolution of the framework
- Objectives and goals ranked by risk and criticality.
- Metrics to be tracked. These should be used to provide an indication as to when your automation effort is “on track” or perhaps has “gone astray”. Identify and gather metrics prior to the implementation of automation to provide a baseline for comparison.
- Have predefined criteria for what should be automated and what should remain manual.
- A process for gathering and managing test data
- Guidelines, procedures, and naming conventions
- Identify the roles within your team and each member’s responsibilities as it pertains to the automation effort
- Develop an enablement process to specify how the automation knowledge of your team will improve. How will they be trained and educated?
- This, by no means, is all there is to defining an automation process but it will get you started.
These are just a few of the common challenges I’ve seen relative to test automation. If you have any questions or thoughts and experiences you’d like to share I’d love to hear from you. Please contact me at firstname.lastname@example.org.
Author – Bob Crews is Checkpoint Technologies’ Founder and President. Checkpoint Technologies is an HP Gold Partner who specializes in IT Quality Assurance and software testing, including mobile and application security.