All About Automation Testing

Glasdon Falcao
Aug 19, 2016
14 min read

Automated software testing is becoming more and more important for many software projects in order to automatically verify key functionality, test for regressions and help teams run a large number of tests in a short period of time.

What is Automation Testing?

The process of using special automation software to control the execution of software tests to compare the outcome with the expected results is called Automation Testing.

Why automate testing?

In today´s fast moving world, it is a challenge for any company to continuously maintain and improve the quality and efficiency of software systems development.

In many software projects, testing is neglected because of time or cost constraints.

This leads to a lack of product quality, followed by customer dissatisfaction and ultimately to increased overall quality costs.

The main reasons for these added costs are primarily:

Poor test strategy
Underestimated effort of test case generation
Delay in testing
Subsequent test maintenance

Automated Testing can add value by providing:

Speed
Repeatability
Coverage
Reliability
Reusability

Major Objective:

The objective of automated testing is to simplify as much of the testing effort as possible with a minimum set of scripts. If unit testing consumes a large percentage of a quality assurance (QA) team's resources, for example, then this process might be a good candidate for automation.

Test Automation Process

#1 Test Automation - Plan

This is the first step in the test automation process. The major action item here is to create a plan that specifies Purpose, Scope, Strategies, Major Requirements, Schedule, Budget.

#2 Test Automation - Design and Development The major action item here is to create a detailed automation solution. This will adress the major objectives and meet all the automation requirements. This is more of a detailed breakup adress majority of the automation plan items. In the development phase the various test automation framework and scripts are developed.

#3 Test Automation Tool - Preparation The major action item here is evaluate the various automation tools and decide a on a tool to be used for the project. This is more of a feasibility study. In this stage an inhouse tool can also be developed (if feasibile). Once the tool is decided upon, the tool is deployed with the various required configuration required for the project.

#4 Test Automation solution - Deployment Once the tool and the scripts are ready, they are integrated together and deployed on the test environment.

#5 Test Automation - Review The working of the automation solution is reviewed and to identify issues and limitations and provide feedback. This will help to further enhance the solution.

Approaches to Test Automation

There are several innovative automation testing tools but before we discuss that, we need to understand the different kinds of approaches to automation, as mentioned below:

Code-driven testing – this approach uses testing frameworks like the xUnit framework etc. that focus on test case execution to find out if the various sections of code are performing as per expectations under different conditions. Code-driven testing approach is a popular method used in agile software development wherein unit test cases are written to determine the requirements of the software being built and test for its functional capability even before the code is actually written. For more insights on code-driven testing approach, visit our course on automation with JUnit framework.

Graphical user interface (GUI) testing – applications that have GUIs may be tested using this approach as it allows the testers to record user actions and analyse them any number of times. For example, if you want to test a website you can use automated testing tools like Selenium that provides a record and playback tool for authoring tests without any knowledge on a test scripting language. Test cases can be written in a number of programming languages like C#, Java, Perl, Python etc.

Framework approach – the framework approach to automation is nothing but an integrated system of rules to test a specific product. The framework brings together function libraries, test data sources, object details and other reusable modules. It lays down the general rules of automation and simplifies the efforts required while bringing in a low maintenance cost. For example, if there is any change in a test case, then only that particular test case file needs to be updated without making any change to the driver or start up script. Frameworks could be linear, structured, data driven, keyword driven or a hybrid of any of these. Find out more on testing tools in our course on automation with Selenium.

Common Misconceptions About Automated Testing

Automating certain programmable aspects of your test suite thus can help software delivery massively.

Now that we’ve provided an introduction to test automation, how it works, and who should be involved, let’s take a closer look at some common misconceptions about test automation:

1. Automation will provide you with more free time

The misconception that automated testing will give you more free time is both true and false. In manual testing, most of the time is devoted to exploratory and functional testing where you would manually search for errors. Once that process is complete, the manual tester must repeatedly go through the same steps over again.

With automated testing, that time is cut drastically. The work for automated testers is instead spent coding the tests and making improvements to these tests repeatedly as adjustments are needed. Once the test is complete, however, automated testing allows for the recycled use of tests so that they do not have to go through this whole process again. In essence, the time spent on the mundane tasks and repetition a manual tester would go through is instead spent focusing on larger, more important issues involving the software you’re developing.

2. The Cost of Automated Testing is Too High At first, the investment in automated testing might seem cost prohibitive, especially if you’re a smaller company. But analysis has shown that, over time, automated testing pays for itself.

As mentioned previously, automated testing frees you up to focus on larger issues such as customer needs, functionality and improvements.

Automated testing also reduces the cost and need for multiple code revisions, so over the course of time, the investment pays out. In addition, each time the source code is modified, the software tests can be repeated. Manually repeating these tests is costly and time-consuming, but automated tests can be run over and over again at no additional cost.

3. Automated Testing is Better Than Manual Testing The reality is, there is no “better” or “worse” in the automated vs. manual debate, there’s just “different.” Each approach has its own advantages and disadvantages. Manual testing is performed by a human sitting in front of a computer carefully going through application via SQL and log analysis, trying various usage and input combinations, comparing the results to the expected behavior and recording the results.

Automated testing is often used after the initial software has been developed. Lengthy tests that are often avoided during manual testing can be run unattended. They can even be run on multiple computers with different configurations.

In the end, both manual and automated testing have their roles, especially if the software you’re developing is too large and too complex to rely just on the manual approach.

4. Automated Testing Inhibits Human Interaction Another common misconception about automated testing is that it undermines human interaction. In all honesty, automated testing is more clear-cut and faster than what humans could do without suffering extensive human errors, so this misconception is understandable. That said, products like TestComplete are designed to facilitate a collaborative approach by including features that allow co-workers to go through a piece of test coding and comment on the script.

This doesn’t replace the face-to-face communication that’s a necessary part of software development. Instead, it enhances that aspect by providing another channel through which to communicate. Think of it this way – email didn’t replace the telephone; it was just an additional tool that could be used to communicate.

Automated Testing Best Practices and Tips

Use these top tips to ensure that your software testing is successful and you get the maximum return on investment (ROI):

Decide what Test Cases to Automate
Test Early and Test Often
Select the Right Automated Testing Tool
Divide your Automated Testing Efforts
Create Good, Quality Test Data
Create Automated Tests that are Resistant to Changes in the UI

Decide What Test Cases to Automate

It is impossible to automate all testing, so it is important to determine what test cases should be automated first.

The benefit of automated testing is linked to how many times a given test can be repeated. Tests that are only performed a few times are better left for manual testing. Good test cases for automation are ones that are run frequently and require large amounts of data to perform the same action.

You can get the most benefit out of your automated testing efforts by automating:

Repetitive tests that run for multiple builds.
Tests that tend to cause human error.
Tests that require multiple data sets.
Frequently used functionality that introduces high risk conditions.
Tests that are impossible to perform manually.
Tests that run on several different hardware or software platforms and configurations.
Tests that take a lot of effort and time when manual testing.
Success in test automation requires careful planning and design work. Start out by creating an automation plan. This allows you to identify the initial set of tests to automate, and serve as a guide for future tests. First, you should define your goal for automated testing and determine which types of tests to automate. There are a few different types of testing, and each has its place in the testing process. For instance, unit testing is used to test a small part of the intended application. To test a certain piece of the application’s UI, you would use functional or GUI testing.
After determining your goal and which types of tests to automate, you should decide what actions your automated tests will perform. Don’t just create test steps that test various aspects of the application’s behavior at one time. Large, complex automated tests are difficult to edit and debug. It is best to divide your tests into several logical, smaller tests. It makes your test environment more coherent and manageable and allows you to share test code, test data and processes. You will get more opportunities to update your automated tests just by adding small tests that address new functionality. Test the functionality of your application as you add it, rather than waiting until the whole feature is implemented.
When creating tests, try to keep them small and focused on one objective. For example, separate tests for read-only versus read/write tests. This allows you to use these individual tests repeatedly without including them in every automated test.
Once you create several simple automated tests, you can group your tests into one, larger automated test. You can organize automated tests by the application’s functional area, major/minor division in the application, common functions or a base set of test data. If an automated test refers to other tests, you may need to create a test tree, where you can run tests in a specific order.

Test Early and Test Often

To get the most out of your automated testing, testing should be started as early as possible and ran as often as needed. The earlier testers get involved in the life cycle of the project the better, and the more you test, the more bugs you find. Automated unit testing can be implemented on day one and then you can gradually build your automated test suite. Bugs detected early are a lot cheaper to fix than those discovered later in production or deployment.

Select the Right Automated Testing Tool

Selecting an automated testing tool is essential for test automation. There are a lot of automated testing tools on the market, and it is important to choose the automated testing tool that best suits your overall requirements.

Divide Your Automated Testing Efforts

Usually, the creation of different tests is based on the QA engineers’ skill levels. It is important to identify the level of experience and skills for each of your team members and divide your automated testing efforts accordingly. For instance, writing automated test scripts requires expert knowledge of scripting languages. Thus, in order to perform these tasks, you should have QA engineers that know the script language provided by the automated testing tool.

Some team members may not be versed in writing automated test scripts. These QA engineers may be better at writing test cases. It is better when an automated testing tool has a way to create automated tests that do not require an in-depth knowledge of scripting languages.

A keyword test (also known as keyword-driven testing) is a simple series of keywords with a specified action. With keyword tests, you can simulate keystrokes, click buttons, select menu items, call object methods and properties, and do a lot more. Keyword tests are often seen as an alternative to automated test scripts. Unlike scripts, they can be easily used by technical and non-technical users and allow users of all levels to create robust and powerful automated tests.

You should also collaborate on your automated testing project with other QA engineers in your department. Testing performed by a team is more effective for finding defects and the right automated testing tool allows you to share your projects with several testers.

Create Good, Quality Test Data

Good test data is extremely useful for data-driven testing. The data that should be entered into input fields during an automated test is usually stored in an external file. This data might be read from a database or any other data source like text or XML files, Excel sheets, and database tables. A good automated testing tool actually understands the contents of the data files and iterates over the contents in the automated test. Using external data makes your automated tests reusable and easier to maintain. To add different testing scenarios, the data files can be easily extended with new data without needing to edit the actual automated test.

Creating test data for your automated tests is boring, but you should invest time and effort into creating data that is well structured. With good test data available, writing automated tests becomes a lot easier. The earlier you create good-quality data, the easier it is to extend existing automated tests along with the application's development.

Create Automated Tests That Are Resistant to Changes in the UI

Automated tests created with scripts or keyword tests are dependent on the application under test. The user interface of the application may change between builds, especially in the early stages. These changes may affect the test results, or your automated tests may no longer work with future versions of the application. The problem is automated testing tools use a series of properties to identify and locate an object. Sometimes a testing tool relies on location coordinates to find the object.

For instance, if the control caption or its location has changed, the automated test will no longer be able to find the object when it runs and will fail. To run the automated test successfully, you may need to replace old names with new ones in the entire project, before running the test against the new version of the application. However, if you provide unique names for your controls, it makes your automated tests resistant to these UI changes and ensures that your automated tests work without having to make changes to the test itself. This also eliminates the automated testing tool from relying on location coordinates to find the control, which is less stable and breaks easily.

Various Automation Tools:

Automated testing tools are capable of executing tests, reporting outcomes and comparing results with earlier test runs.

Tests carried out with these tools can be run repeatedly, at any time of day.

Consider these key points when selecting an automated testing tool:

Support for your platforms and technology. Are you testing .Net, C# or WPF applications and on what operating systems? Are you going to test web applications? Do you need support for mobile application testing? Do you work with Android or iOS, or do you work with both operating sistems?

Flexibility for testers of all skill levels. Can your QA department write automated test scripts or is there a need for keyword testing?

Feature rich but also easy to create automated tests. Does the automated testing tool support record-and-playback test creation as well as manual creation of automated tests; does it include features for implementing checkpoints to verify values, databases, or key functionality of your application?

Create automated tests that are reusable, maintainable and resistant to changes in the applications UI. Will my automated tests break if my UI changes?

For detailed information about selecting automated testing tools for automated testing, see Selecting Automated Testing Tools.

The Automation testing tools can be classified as below. Some of the tools are also provided.

a) Functional & Regression Test Tools:

Ex: HP - WinRunner, QTP (Quick Test Professional). IBM - Rational Robot, RFT (Rational Functional Tester). Micro Focus - SilkTest. Open source - Selenium.

b) Performance/Load/Stress test tools

Ex: HP - LoadRunner. IBM - RPT (Rational Performance Tester). Microfocus - Silk Performer. Open source - Jmeter, WebLOAD.

Test Automation Methodologies

The following is a description of two methods that have proven effective in implementing an Automated Testing Solution:

"Functional Decomposition" Method:

The main concept behind the "Functional Decomposition" script development methodology is to reduce all test cases to their most fundamental tasks, and write User-Defined Functions, Business Function Scripts, and "Sub-routine" or "Utility" Scripts which perform these tasks independently of one another.

In general, these fundamental areas include:

Navigation (e.g. "Access Payment Screen from Main Menu")
Specific Business Function (e.g. "Post a Payment")
Data Verification (e.g. "Verify Payment Updates Current Balance")
Return (e.g. "Return to Main Menu")

In order to accomplish this, it is necessary to separate Data from Function. This allows an automated test script to be written for a Business Function, using data-files to provide the both the input and the expected-results verification. A hierarchical architecture is employed, using a structured or modular design.

The highest level is the Driver script, which is the engine of the test. The Driver begins a chain of calls to the lower level components of the test. Drivers may perform one or more test case scenarios by calling one or more Main scripts. The Main scripts contain the test case logic, calling the Business Function scripts necessary to do the application testing. All utility scripts and functions are called as needed by Drivers, Main, and Business Function scripts.

Driver Scripts: Perform initialization (if required), then call the Main Scripts in the desired order.
Main Scripts: Perform the application test case logic using Business Function Scripts.
Business Function Scripts: Perform specific Business Functions within the application.
Subroutine Scripts: Perform application specific tasks required by two or more Business Function scripts.
User-Defined Functions: General, Application-Specific, and Screen-Access Functions.

Advantages:

Utilizing a modular design, and using files or records to both input and verify data, reduces redundancy and duplication of effort in creating automated test scripts.
Scripts may be developed while application development is still in progress. If functionality changes, only the specific "Business Function" script needs to be updated.
Since scripts are written to perform and test individual Business Functions, they can easily be combined in a "higher level" test script in order to accommodate complex test scenarios.
Data input/output and expected results is stored as easily maintainable text records. The user's expected results are used for verification, which is a requirement for System Testing.
Functions return "TRUE" or "FALSE" values to the calling script, rather than aborting, allowing for more effective error handling, and increasing the robustness of the test scripts. This, along with a well-designed "recovery" routine, enables "unattended" execution of test scripts.

Disadvantages:

Requires proficiency in the tool's Scripting Language;
Multiple data-files are required for each Test Case. There may be any number of data-inputs and verifications required, depending on how many different screens are accessed. This requires data-files to be kept in separate directories by Test Case.
Tester must not only maintain the Detail Test Plan with specific data, but must also re-enter this data in the various data-files.
If a simple "text editor" such as Notepad is used to create and maintain the data-files, careful attention must be paid to the format required by the scripts/functions that process the files, or processing-errors will occur.

Automation Framework

The method or process being used to implement automation is called a test automation framework. A test automation framework consists of a set of assumptions, concepts and tools that provide support for automated software testing. The main advantage of such a framework is the low cost for maintenance. If there is change to any test case then only the test case file needs to be updated. However the Driver Script and Startup script will remain the same.

Choosing the right framework/scripting technique helps in maintaining lower costs. The costs associated with test scripting are due to development and maintenance efforts. The approach of scripting used during test automation has effect on costs.

Various Framework/scripting techniques are generally used:

Linear (procedural code, possibly generated by tools like those that use record and playback)
Structured (uses control structures - typically ‘if-else’, ‘switch’, ‘for’, ‘while’ conditions/ statements)
Data-driven (data is persisted outside of tests in a database, spreadsheet, or other mechanism)
Keyword-driven
Hybrid (two or more of the above patterns are used)

The Testing framework is responsible for:

Defining the format in which to express expectations.
Creating a mechanism to hook into or drive the application under test.
Executing the tests.
Reporting results.

Some of the drawbacks of Automation Testing are,

If the application / product do not have many release cycles, it is not worth the time and money automating for one/two cycles.
The automation process is expensive.
All areas of the application/product cannot be automated. There would be certain limitations in certain areas.

When should the application be automated?