Benefits of Keyword Driven Testing for Test Automation

Most software companies have considered automated testing and many have fully automated their regression test cases in an effort to reduce manual effort needed to test new builds of their software. Many companies that have been successful with automation attribute it to keyword driven testing techniques that reduce the time spent creating test cases.

This newsletter addresses why companies consider automation and best practices for ensuring that time spent automating test cases provide a return on investment.

Why Automate Your Test Cases?
Many companies run their regression test cases manually, so when does it make sense to begin automating your regression test cases? It makes sense to automate your test cases when you can no longer run the regression test cases on each build created. For example, if you are doing daily or weekly builds of your code to quality assurance and you cannot quickly run your regression test cases with each build, it is time to consider automating them. Automating your test cases provide these benefits:

• Quicker Releases – By having your regression test cases run automatically, your software quality team can concentrate on testing new features of your software and less time regressing existing features.
• Higher quality releases – Your software releases will have fewer bugs and require less customer support because they will be of higher quality.
• Happier Customers – Your customers will be happier and more willing to serve as testimonials for future prospects.

Why Does Automation Fail?
Many companies are thinking about test case automation or have experimented with it in the past. Some companies that experimented with it eventually abandoned it because of these reasons:

• Poor Understanding of Test Automation – Many companies see automation as a silver bullet that will allow them to quickly automate every test scenario quickly and allow them to abandon manual testing and reduce staff. In reality, automated testing is designed to quicken the running of regression test cases but it is not a substitute for manual testing. A quality-oriented software team will see the value of utilizing both automated and manual test cases to ensure great test coverage and higher quality releases.
• Improper Tester Education – Automation requires a tester to learn the testing tool. Companies that purchase the tool but not any training will eventually abandon the tool because the testers are not equipped to use the tool.
• Lure of Record / Playback – Many companies think that they can quickly get up and running with automation by simply recording their screen actions and playing them back. While record and playback will create scripts that they can use as a starting point for your automation, it does take time to update the scripts to be more re-usable and it takes scripting language knowledge to do this.

What is Keyword Driven Testing?
Most automated tools require the test engineer to understand a scripting language (VB Script, Java Script, etc.) to write their automated test cases. Most tools have the ability to create the scripts using record and playback, but this does not always write the most efficient scripting code and is not as re-usable and maintainable.

Since many testers do not have deep scripting skills, it is imperative that your automated testing tool has a way to create Keyword Driven Tests. Keyword Driven Testing is a way to define automated test cases without the need for scripting skills. It allows a tester (or even a subject matter expert) to create automated tests by describing each step of the automation.

For example, if you are automating the login process of your application, your user will access your application, type in their user-id and password and press a button to login. Traditionally, testers would do this by writing VB Script that will navigate to your application, identify each object on the screen (user-id, password and login button), then write script to enter in the user-id, password and to press the login button.

With keyword driven testing, the tester does not need to understand the scripting language to make this happen, they can simply describe the event (navigate to your application, enter in "abc" for the user-id, enter in "xxx" for the password, press the Login button when done). As you can imagine, this is a much simpler approach to automated testing than scripting.

How have Successful Companies Implemented Automation?
Successful companies understand the enormous benefits of automated testing and have implemented strategies to ensure that they receive the maximum return on investment with their test efforts. Below are the secrets to becoming successful with test automation:

• Start Smart – Automation efforts are similar to software development efforts -- it takes upfront thought and a good test design architecture to ensure that your automated test cases will be re-usable and easy to maintain. Before jumping into automation, ensure that your automated tool has the ability to maximize your efforts with keyword driven testing -- as this will ensure that you can easily re-use your automated test cases.
• Start Small – Start your automation efforts on an established project that already has a good set of manual regression test cases written. Take those manual regression test cases and develop your automated test cases that will replace them. Once this is done, you will find that you can automatically run those automated test cases each day of builds and with minimal efforts, freeing your team up to do more manual exploratory test cases. Once you have this established for a single project, move on to other projects and expand your effort.
• Blend Automated and Manual Tests – Not all test cases should be automated. Some test cases require a human eye to ensure that screen cosmetics are appropriate, that data is reasonable, etc. Spend time automating test cases that do not require this type of human contact and continue to use manual test cases when appropriate. A good application lifecycle tool should allow you to track both automated and manual test cases.
• Keep track of Metrics – Keep track of how many automated and manual test cases are run for each build, how many pass, how many fail, etc. Track how many additional automated test cases your team can write with each release so that you can determine average time needed to develop automated test cases in the future. Track how many defects are found post-production to determine if your automation efforts are paying dividends. A good application lifecycle tool should allow you to track metrics for both automated and manual test cases.
• Build on your Successes – Once you have successfully implemented automated testing on a single project, roll it out to more projects. Help other teams in your organization learn the benefits of automation and help them do it right.
• Schedule Your Automation Efforts - A good application lifecycle tool will allow you to organize your automated test cases into "test sets" that allow you to run test cases in a specific order. It should also allow you to schedule those test sets to run at specific intervals (nightly, weekly, monthly, etc.).

How can I learn more about Test Automation?
There are a number of automated and manual test management solutions on the market, as long as your selected solution allows you to follow the best practices of this newsletter, you can easily begin making the transition to automation and begin receiving a return on investment from your efforts.

If you don't have a solution for both automated and manual testing, Software Planner (http://www.SoftwarePlanner.com) and TestComplete (http://www.testcomplete.com) offer an integrated solution that offers both automated and manual test cases and has a keyword driven testing engine that ensures re-usability and maintainability.

Helpful Resources
Below are some helpful resources and templates to aid you in developing software solutions:

• Software Planner - http://www.SoftwarePlanner.com
• AutomatedQA TestComplete (Automated Testing Tool) - http://www.TestComplete.com
• Software Development /QA Templates -http://www.pragmaticsw.com/Templates.asp
• Test Case Training - http://www.PragmaticSW.com/Services.asp
• Pragmatic Agile Development -http://www.pragmaticsw.com/PADOverview.pdf

Automating Your Regression Test Cases

Companies that develop and maintain software can dramatically improve the quality of their software releases by creating regression test cases that ensure that existing features are not broken with new releases. This newsletter discusses:

• How to create regression test cases
• When to automate regression test cases
• Best practices for automation analysis

Creating Regression Test Cases
Once a software product has been released to production, each new release of
the software could cause existing features to fail. To prevent this, it is wise to create a set of regression test cases that are run with each new release. Below are some best practices when developing a regression suite:

• Categorize by Functional Area – Your software product most likely has different sets of functional areas (e.g. Invoicing, Billing, etc). When creating regression test cases, categorize them by functional area so that you can ensure you have good test coverage for each functional area of your software.
• Regression Test Case Design – Regression test cases do not normally need to test bounds, invalid data entry, etc – normally they will be designed to test the software the way it is designed to work. The reason for this is that when the feature was originally designed, it should have been thoroughly tested for bounds, invalid data, etc. An exception to this is if you find that new releases tend to break existing features from a validation perspective. If this is the case, keep some specialized regression test cases to ensure that the validations are still in place.
• Revisit the Regression Set with each New Release – Upon implementing a new release of your software, it is wise to recognize new features shipped with the new release and to create a new set of regression test cases that test the new features. If you do not revisit your regression test cases with each new release of your software, the regression test cases will become stale and out dated.

When to Automate Regression Test Cases
Many companies run their regression test cases manually, so when does it make sense to begin automating your regression test cases? It makes sense to automate your test cases when you can no longer run the regression test cases on each build created. For example, if you are doing daily or weekly builds of your code to quality assurance and you cannot quickly run your regression test cases with each build, it is time to consider automating them.

To automate test cases, you must purchase an automated testing tool. There are many great tools on the market, including Automated QA Test Complete (http://www.TestComplete.com), HP Quick Test Pro, HP Win Runner, Rational Robot and Rational Functional Test, just to name a few. We normally recommend Automated QA Test Complete, as it is competitively priced and has similar features as the others. Once you have purchased an automated tool, you can use the tool to create your regression test cases.

How does Automation Work?
Each test case becomes a script. Many tools have record and playback features where you can turn the recorder on, open your software and perform the actions for a test scenario, then save the recording. This is a great way to learn the scripting engine, but it is not usually adequate to create well designed automated test scripts. Normally, you will want to have a technically minded software quality engineer in your organization that creates and maintains the automated scripts, as using these tools require knowledge of the tool, programming skills and great trouble shooting skills.

If you are initially creating your automation strategy, it is wise to consult with an automation expert to ensure best practices for your automation design. There are many companies that specialize in this; we have worked extensively with Star‐QA (http://www.star-qa.com) with great results.

Contracting with an automation expert can save effort and costs in the long term, as they will normally work with you to provide an automation framework that will be reusable and can provide training to your software quality engineer(s), allowing them to make great strides with their automation skill set in very little time. Another advantage of working with an automation expert is that they can implement "keyword driven automation". This simply means that they can create a set of re-usable automation scripts that can be invoked by name (Login, AddOrder, AddtoCart, etc), allowing less technical team members to create new sets of automated scripts.

Best Practices for Automation Analysis
Once your regression test cases are automated, they should be automatically run upon new builds of your software. If you can do daily builds of your software into your quality assurance environment, this is ideal. Once the automation is running daily, you will need a way to quickly determine how many automation test cases were run, how many passed and how many failed.

For failed tests, you will want to drill into the detailed logs to determine what caused the failure. Software Planner (http://www.SoftwarePlanner.com) is an ALM tool that can manage this process. Software Planner integrates with all the major automated testing tools including Automated QA Test Complete, HP Quick Test Pro, HP Win Runner, Rational Robot, and Rational Functional Test. By integrating automated testing into Software Planner, you can launch the tests from within Software Planner, create test sets, analyze the results (which tests passed or failed), and automatically send emails upon test completion. You can also trend these results using graphical dashboards. Below is an example of a dashboard that shows trending of your automation runs:

As you can see from the graph above, the past 2 days (Oct 10/11) has introduced a problem in the code because 19 automated test cases failed while 17 passed. Looking at the graph, you can see that the the issue was introduced on Oct 9, as all test cases passed from Oct 2 - Oct 6. This type of information is invaluable for quality assurance teams.

Summary
As we have seen, automating your regression test cases can be valuable. You should see a return on investment within one release of your software after implementing an automation test strategy. This will be achieved by:

1. Quicker Releases – By having your regression test cases run automatically, your software quality team can concentrate on testing new features of your software and less time regressing existing features.
2. Higher quality releases – Your software releases will have fewer bugs and require less customer support because they will be of higher quality.
3. Happier Customers – Your customers will be happier and more willing to serve as testimonials for future prospects.

Helpful Templates
Below are some helpful templates to aid you in developing software solutions on-time and on-budget:

• Pragmatic Agile Development -http://www.pragmaticsw.com/PADOverview.pdf
• Software Development /QA Templates -http://www.pragmaticsw.com/Templates.asp
• Software Planner - http://www.SoftwarePlanner.com
• Agile Training - http://www.PragmaticSW.com/Services.asp