Before the First Line of Code: 7 Key Questions for Starting a Test Automation Project
Last year, I had an interesting client call. Our Sales Manager informed me about a new client who wanted to implement test automation in their upcoming project. Naturally, I was excited because it's always great when a client is already convinced about the benefits of automating tests. The next day, I had my first discussion with the client, and it quickly became apparent that they wanted to start immediately. As we began talking, several crucial questions emerged. It wasn't clear who would be responsible for maintaining the tests, whether the system was ready for test automation, and what the overall goal of the test automation project was. These uncertainties highlighted the need for a more detailed conversation to ensure we could move forward effectively.
Unfortunately, this situation is more common than some might think. Many companies are enthusiastic about implementing test automation but haven't considered all the essential questions beforehand. To help you avoid potential issues down the line, I've compiled seven important questions to ask before writing a single line of code in your test automation project. These questions will help ensure a smoother and more successful implementation, preventing problems and discussions later on.
What is the Goal of Your Test Automation Project?
The goal of a test automation project can vary significantly depending on the specific needs and priorities of the project. One primary objective might be to reduce the time spent on regression testing. Automated tests can run quickly and frequently, ensuring that any changes or new features do not break existing functionality. Additionally, automation can help reduce technical debt by catching issues early, thereby maintaining the health of the codebase and preventing the accumulation of unresolved bugs and inefficiencies.
Another aim could be to enable continuous testing. By integrating automated tests into the continuous integration and continuous deployment (CI/CD) pipeline, immediate feedback on the impact of code changes can be provided. This not only accelerates development but also improves overall code quality. Developers and testers get an automated feedback for every change they push to the repository.
Sometimes companies believe that a major goal of test automation is to replace manual efforts. In my opinion, the aim of every test automation project should never be to replace a "manual" tester but rather to support them. Automation can handle repetitive and mundane tasks, allowing testers to focus on more complex testing. However, there will always be a need for “manual” testers. Just remember… you can’t automate everything!
It's also crucial to set clear expectations and hoped-for results. Defining what success looks like for the automation project—be it faster release cycles, fewer bugs in production, or improved code quality—helps in evaluating the project's success and aligning it with broader business goals. As I mentioned, people often believe that once automation is implemented, it requires no effort and that there will be no bugs. However, this is far from reality. It's important to clarify these expectations from the beginning to ensure everyone is on the same page.
Reduce the time of regression testing?
Reduce technical debt?
Enable continuous testing?
Replace manual efforts?
What are the expectations and hoped-for results?
How Ready is Your Project for Automation?
Evaluating the current state of a project in terms of automation involves several critical questions. First, it’s essential to determine whether the existing processes and requirements can be automated. Some specifications might be too complex or require human judgment, making them less suitable for automation. Ensuring that the requirements for automation are clear and well-defined is essential for effective test creation.
Additionally, when people haven't worked with automation before, they tend to overestimate its capabilities and believe everything can be effortlessly transferred into automated scripts. This is rarely the case. It's essential to thoroughly examine the tests or the parts of the software that should be automated and decide if it is even possible or not.
The stability of the system is another crucial factor. A stable system is essential for reliable test automation since frequent changes or unstable environments can lead to flaky tests and unreliable results. If the system isn't stable, how can the tests be stable?
Finally, assessing the available resources is important. Enough resources are necessary to support the development and maintenance of automated tests, ensuring the project's long-term success. Be clear about the process, and how much time and effort are required to achieve these goals. If the client isn't aware of these details, you might find yourself midway through the project without a sufficient budget.
Are the requirements for the automation well-defined?
Can the project be automated?
Is the system stable and running?
What resources are available (budget, etc.)?
Who Will Be Involved in the Test Project?
The success of a test automation project depends significantly on the involvement and collaboration of various team members. In smaller teams, it might be the case that one tester handles everything—from writing and monitoring tests to maintaining them. While this ensures consistency, it can be quite challenging, especially if the project gets bigger and bigger. Larger projects might have one or more dedicated automation engineers focusing solely on automation efforts. Also, clear the roles at the beginning of the project and clarify how many testers will work on this project. Including developers in the process is beneficial as they can write or contribute to tests, ensuring better test coverage and more robust code. Involving Product Owners or Business Analysts can provide valuable insights into critical business functionalities that need testing, ensuring that the automation efforts align with business goals. Cultivating a culture that values quality and automation, providing training, and clearly demonstrating the benefits can help get everyone on board and committed to the success of the project.
One tester does everything (Write/Monitor/Maintain)?
Dedicated automation engineer or on the side?
Will the dev be involved / write tests themselves?
Involve Po / Business Analyst for their perspective?
How to get people on board?
Which Tests Should Be Automated?
Determining which tests to automate involves several factors. First, it’s necessary to evaluate whether existing manual tests can be automated or if new automated tests need to be written from scratch. While automation can significantly reduce manual efforts, some areas might still require manual testing, such as exploratory testing or usability testing. It’s also important to identify which areas of the application are already covered by tests and which should be prioritized for automation. Recurring tests, such as those in a regression test set, are obviously good candidates for automation, just like sanity checks. Are there any parts of the software that could be tested fully with automation?
Additionally, using established design patterns like the Page Object Model (POM) or Behavior Driven Development (BDD) can help structure the tests efficiently, making them easier to maintain and scale. Which design patterns are familiar to the engineers and align with the structure of the software? Choosing the right design pattern will be a crucial foundation for your entire test automation project and should be chosen wisely.
Are there already existing tests or need to write new ones?
Which areas of the application should be covered by automation?
Still manual effort?
Which design patterns will be used?
What Will Be Your Execution and Reporting Strategy?
A solid execution and reporting strategy is key to effective test automation. Defining who has the authority to trigger test runs ensures accountability and clarity. Scheduling tests to run at specific times, such as nightly or after each code commit, can help identify issues promptly. Integrating tests into the CI pipeline ensures that every code push is tested, providing immediate feedback to developers. Automated test reports can be used for quickly identifying and addressing failures. It’s also important to have a clear strategy for handling test failures. Deciding whether failures should block development or be handled later can significantly impact the development process and the overall quality of the product.
Who is responsible for deciding when to start the tests?
Should the tests be executed at certain times?
Should the tests be integrated into the development CI job after every new code push?
Will there be automated test reports?
What is the strategy when tests fail? Should development be blocked?
What Technologies Will Be Used?
Choosing the right technologies is also a crucial step for the success of a test automation project. Selecting a programming language that aligns with your team's expertise and the application's tech stack ensures smoother implementation. The choice of the testing framework is also critical. Frameworks like Selenium, Cypress, or JUnit are popular choices, but it’s essential to ensure that the selected framework meets the specific testing needs of the project. Additional tools for test management, reporting, or CI/CD integration might also be necessary to support the automation efforts. Assessing the team’s familiarity with the chosen tech stack and providing the necessary training can help in overcoming potential challenges and ensuring the successful adoption of the automation tools.
Which programming language is being used?
Which testing framework will be used?
Does this framework meet the testing requirements?
Are any additional tools necessary?
What is the knowledge and experience level with the tech stack among the team members?
What is the Long-Term Plan for Your Test Automation Project?
Planning for the future is essential to ensure the sustainability and scalability of the test automation project. As the project grows, it’s important to plan for scaling the automation effort, which might involve adding more tests, increasing test coverage, or improving the test infrastructure. Considering future enhancements, such as adding performance tests or integrating new tools, can help keep the automation efforts relevant and effective. Evaluating the need for additional resources, such as hiring more testers or investing in better tools, is also important for the long-term success of the project. Finally, establishing a robust documentation strategy ensures that knowledge is preserved and easily accessible to new team members, facilitating smooth transitions and continued success.
How will the project be scaled?
How will the project evolve in the future?
Are there plans for more test automation? What resources are needed?
How will information be handed over and preserved?
What is the documentation strategy?
In the end, the project I mentioned in the beginning never started. My first question was, "What is the goal of the test automation project?" He replied that he wanted to replace some testers and possibly outsource the automation to developers over time. I knew immediately that his expectations were unrealistic and that he had a misguided impression of automation.
If I had simply agreed, saying we could automate a lot of tests and everything would be fine, we would have faced a longer argument down the line about why he still needed to keep his other testers and why the project required a significant budget. This experience taught me the importance of setting clear and realistic expectations from the start to avoid misunderstandings and ensure the success of any test automation project.
In summary, a successful test automation project requires careful planning, clear communication, and realistic expectations from the start. By addressing key aspects such as the goals of the project, the current state of the system, available resources, team roles, and long-term plans, you can lay a solid foundation for your test automation efforts. With a clear strategy and the right preparations, you'll be better equipped to handle challenges and achieve your automation goals successfully.