360Logica.com

Agile Projects present their own challenges to the Automation team; Unclear project scope, Multiple iterations, Minimal documentation, early and frequent Automation needs and active stakeholder involvement all demand lot of challenges from the Automation Team. Some of these challenges are:

Challenge 1: Requirement Phase

Test Automation developer captures requirements in the form of “user stories”, which are brief descriptions of customer-relevant functionality.

Each requirement has to be prioritized as follows:

• High: These are mission critical requirements that absolutely have to be done in the first release
• Medium: These are requirements that are important but can be worked around until implemented.
• Low: These are requirements that are nice-to-have but not critical to the operation of the software.

Once priories are established, the release “iterations” are planned. Normally, each Agile release iteration takes between 1 to 3 months to deliver. Customers/software folks take liberty to make too many changes to the requirements. Sometimes, these changes are so volatile that the iterations are bumped off. These changes are greater challenges in implementing Agile Automation testing process.

Challenge 2: Selecting the Right Tools

Traditional, test-last tools with record-and-playback-features force teams to wait until after the software is done. More over, traditional test automation tools don’t work for an Agile context because they solve traditional problems, and those are different from the challenges facing Agile Automation teams. Automation in the early stages of an agile project is usually very tough, but as the system grows and evolves, some aspects settle and it becomes appropriate to deploy automation. So the choice of testing tools becomes critical for reaping the efficiency and quality benefits of agile.

Challenge 3: Script Development Phase

The Automation testers, developers, business analysts and project stakeholders all contribute to kick-off meetings where “user-stories” are selected to next sprint. Once the “user-stories” are selected for the sprint, they are used as the basis for a set of tests.

As functionality grows with each iteration, regression testing must be performed to ensure that existing functionality has not been impacted by the introduction of new functionality in each iteration cycle. The scale of the regression testing grows with each sprint and ensures that this remains a manageable task the test team use the test automation for the regression suite.

Challenge 4: Resource Management

The Agile approach requires a mixture of testing skills, that is, test resource will be required to define unclear scenarios and test cases, conduct manual testing alongside developers, write automated regression tests and execute the automated regression packages. As the project progresses, specialist skills will also be required to cover further test areas that might include integration and performance testing. There should be an appropriate mix of domain specialist who plan and gather requirements. The challenging part in the Resource management is to find out test resources with multiple skills and
allocate them.

Challenge 5: Communication

Good communication must exist among Automation testing team, developers, business analysts and stake holders. There must be highly collaborative interaction between client and the delivery teams. More client involvement implies more suggestions or changes from the client. It implies more bandwidth for communication. The key challenge is that the process should be able to capture and effectively implement all the changes and data integrity needs to be retained. In traditional testing, developers and testers are like oil and water, but in agile environment, the challenging task is that they both must work together to achieve the target.

Challenge 6: Daily Scrum Meeting

Daily Scrum Meeting is one of the key activities in Agile Process. Teams do meet for 15 minutes stand up sessions. What is the effectiveness of these meetings? How far these meetings help Automation practice Developers?

Challenge 7: Release Phase

The aim of Agile project is to deliver a basic working product as quickly as possible and then to go through a process of continual improvement. This means that there is no single release phase for a product. The challenging part lies in integration testing and acceptance testing of the product.

If we can meet these challenges in a well optimized manner, then Automated Regression Testing in Agile environment is an excellent opportunity for QA to take leadership of the agile processes. It is better placed to bridge the gap between users and developers, understand both what is required, how it can be achieved and how it can be assured prior to deployment. Automation practice should have a vested interest in both the how and the result, as well as continuing to assure that the whole evolving system meets business objectives and is fit for purpose.

Source: http://www.softwaretestinghelp.com

What is Regression Software Testing?

Regression means retesting the unchanged parts of the application. Test cases are re-executed in order to check whether previous functionality of application is working fine and new changes have not introduced any new bugs.

This is the method of verification. Verifying that the bugs are fixed and the newly added feature have not created in problem in previous working version of software.

Why regression Testing?

Regression testing is initiated when programmer fix any bug or add new code for new functionality to the system. It is a quality measure to check that new code complies with old code and unmodified code is not getting affected.
Most of the time testing team has task to check the last minute changes in the system. In such situation testing only affected application area in necessary to complete the testing process in time with covering all major system aspects.

How much regression testing?

This depends on the scope of new added feature. If the scope of the fix or feature is large then the application area getting affected is quite large and testing should be thoroughly including all the application test cases. But this can be effectively decided when tester gets input from developer about the scope, nature and amount of change.

What we do in regression testing?

• Rerunning the previously conducted tests
• Comparing current results with previously executed test results.

Regression Testing Tools:

Automated Regression testing is the testing area where we can automate most of the testing efforts. We run all the previously executed test cases this means we have test case set available and running these test cases manually is time consuming. We know the expected results so automating these test cases is time saving and efficient regression testing method. Extent of automation depends on the number of test cases that are going to remain applicable over the time. If test cases are varying time to time as application scope goes on increasing then automation of regression procedure will be the waste of time.

Most of the regression testing tools are record and playback type. Means you will record the test cases by navigating through the AUT and verify whether expected results are coming or not.

Example regression testing tools are:

• Winrunner
• QTP
• AdventNet QEngine
• Regression Tester
• vTest
• Watir
• Selenium
• actiWate
• Rational Functional Tester
• SilkTest

Most of the tools are both Functional as well as regression testing tools.

Regression Testing Of GUI application:

It is difficult to perform GUI(Graphical User Interface) regression testing when GUI structure is modified. The test cases written on old GUI either becomes obsolete or need to reuse. Reusing the regression testing test cases means GUI test cases are modified according to new GUI. But this task becomes cumbersome if you have large set of GUI test cases.

The Client

Customer is leading player in online marketplace which directly connects buyers and providers through a comprehensive and collaborative management platform.

The Requirements

• Human Effort: Reduce the human effort in the functional test of the application.
• Sanity Test: Perform Sanity test of the application on the daily build.
• Regression: Regression testing of the application on any changes in the code.
• Scope of Automation – Perform functional, UI, client validation, Database validation and Exception handling.

The Solution

• Designed the framework which is the combination of data driven, library and keyword driven framework.
• Framework makes the connection to the database, maintains library and uses user defined function through out the script.
• Scripts are written using Selenium IDE and in C# on Visual Studio 2008.
• Exceptions are handled throughout the scripts and capturing all objects on the screen.
• Scripts are designed for both http & https protocol.

The Technology

• Microsoft .NET
• AJAX
• SQL Server
• IIS Server
• Selenium IDE
• Selenium RC 0.9.0
• Microsoft Visual C# 2008 Express Edition
• NUnit 2.4.7

Contribution

• Successful competition of robust scripts which performs sanity test of daily build application and reduces the manual effort to 10%
• Exhaustive coverage of test cases and scenarios help to achieve the consistent testing cycle in minimal time on demand.
• Breakage in existing feature can be caught in very early stage.

You can also download the case study here.

JMeter tests on the protocol layer and not the client layer i.e. JavaScripts, applets, etc. It does not give 100% browser like rendering and will not be able to render HTML pages which restrict it to test the GUI of an application. However, to compensate for these shortcomings, JMeter allows creating assertions based on the tags and text of the page as the HTML file is received by the client. With some knowledge of HTML tags, QA guy can test and verify any elements as you would expect them in the browser.

Test Plan to contain functional validations (or assertions) by incorporating various essential JMeter components, particularly the 'Response Assertion' element and 'Assertion Result' Listener. By using the User Defined Variable, Configuration element, we have also parameterized several values in order to give our Test Plan better flexibility. In addition, we have observed the result of these assertions as we performed a 'live' run of the application under test. An HTTP Request sampler may require to be modified, if there are any changes to the parameter(s) that the sampler sends with each request. Once created, a JMeter Test Plan that contains assertions can then be used and modified in subsequent Regression tests for the application.

High level Approach:

1. Creation of Test Plan which comprises of different requests need to be tested with the details of the response.
2. Setting the Configuration elements, Listeners and Assertions.
3. Setting up of HTTP header manager to capture page header details.
4. User defined variables should be used depending on the scenario and test case design.

Execution:

Once the assertions are properly completed, we are expecting that running our Test Plan would pass all the assertions. Passed assertions will not show any error in Assertion Results | Listener installed within the same scope. As for all Listeners, results as captured by the Listeners can be saved and reproduced at a later time. On the other hand, a failed Assertion would show an error message in the same Listener.