360Logica.com

Software Testing has lot of challenges both in manual as well as in automation. Generally in manual testing scenario, developers pass through the build to test team assuming the responsible test team or tester will pick the build and will come to ask what the build is about? This is the case in organizations not following so-called ‘processes’. Tester is the middleman between developing team and the customers, handling the pressure from both the sides.

So here we go with the top challenges in software testing and assurance

1. Testing the complete application
   Is it possible? There are millions of test combinations. It’s not possible to test each and every combination both in manual as well as in automation testing. If you try all these combinations you will never ship the product.
2. Misunderstanding of company processes
   Some times you just don’t pay proper attention what the company-defined processes are and these are for what purposes. There are some myths in testers that they should only go with company processes even these processes are not applicable for their current testing scenario. This results in incomplete and inappropriate application testing.
3. Relationship with developers
   Big challenge. Requires very skilled tester to handle this relation positively and even by completing the work in testers way. There are simply hundreds of excuses developers or testers can make when they are not agreeing with some points. For this tester also requires good communication, troubleshooting and analyzing skill.
4. Regression testing
   When project goes on expanding the regression testing work simply becomes uncontrolled. Pressure to handle the current functionality changes, previous working functionality checks and bug tracking.
5. Lack of skilled testers
   This can be called this as ‘wrong management decision’ while selecting or training testers for their project task in hand. These unskilled fellows may add more chaos than simplifying the testing work. This results into incomplete, insufficient and ad-hoc testing throughout the testing life cycle.
6. Testing always under time constraint
   Hey tester, we want to ship this product by this weekend, are you ready for completion? When this order comes from boss, tester simply focuses on task completion and not on the test coverage and quality of work. There is huge list of tasks that you need to complete within specified time. This includes writing, executing, automating and reviewing the test cases.
7. Which tests to execute first?
   If you are facing the challenge stated in point no 6, then how will you take decision which test cases should be executed and with what priority? Which tests are important over others? This requires good experience to work under pressure.
8. Understanding the requirements
   Some times testers are responsible for communicating with customers for understanding the requirements. What if tester fails to understand the requirements? Will he be able to test the application properly? Definitely Not! Testers require good listening and understanding capabilities.
9. Automation testing
   Many sub challenges – Should automate the testing work? Till what level automation should be done? Do you have sufficient and skilled resources for automation? Is time permissible for automating the test cases? Decision of automation or manual testing will need to address the pros and cons of each process.
10. Decision to stop the testing
    When to stop testing? Very difficult decision. Requires core judgment of testing processes and importance of each process. Also requires ‘on the fly’ decision ability.
11. One test team under multiple projects
    Challenging to keep track of each task. Communication challenges. Many times results in failure of one or both the projects.
12. Reuse of Test scripts
    Application development methods are changing rapidly, making it difficult to manage the test tools and test scripts. Test script migration or reuse is very essential but difficult task.
13. Testers focusing on finding easy bugs
    If organization is rewarding testers based on number of bugs (very bad approach to judge testers performance) then some testers only concentrate on finding easy bugs those don’t require deep understanding and testing. A hard or subtle bug remains unnoticed in such testing approach.
14. To cope with attrition
    Increasing salaries and benefits making many employees leave the company at very short career intervals. Managements are facing hard problems to cope with attrition rate. Challenges – New testers require project training from the beginning, complex projects are difficult to understand, delay in shipping date!

Source: http://www.softwaretestinghelp.com

The Client

Customer is leading player in online marketplace which directly connects buyers and providers through a comprehensive and collaborative management platform.

The Requirements

• Human Effort: Reduce the human effort in the functional test of the application.
• Sanity Test: Perform Sanity test of the application on the daily build.
• Regression: Regression testing of the application on any changes in the code.
• Scope of Automation – Perform functional, UI, client validation, Database validation and Exception handling.

The Solution

• Designed the framework which is the combination of data driven, library and keyword driven framework.
• Framework makes the connection to the database, maintains library and uses user defined function through out the script.
• Scripts are written using Selenium IDE and in C# on Visual Studio 2008.
• Exceptions are handled throughout the scripts and capturing all objects on the screen.
• Scripts are designed for both http & https protocol.

The Technology

• Microsoft .NET
• AJAX
• SQL Server
• IIS Server
• Selenium IDE
• Selenium RC 0.9.0
• Microsoft Visual C# 2008 Express Edition
• NUnit 2.4.7

Contribution

• Successful competition of robust scripts which performs sanity test of daily build application and reduces the manual effort to 10%
• Exhaustive coverage of test cases and scenarios help to achieve the consistent testing cycle in minimal time on demand.
• Breakage in existing feature can be caught in very early stage.

You can also download the case study here.

Development frameworks have been around for years, as a means for standardization and acceleration of the development process. The same reasoning applies to testing frameworks. Yet, in the past, many enterprises questioned its cost effectiveness as they questioned the need for test automation at all.

While test automation of functional testing can be substituted by manual testing but load testing is always done with automated tools or not done at all.

3PG have defined the load testing architecture and developed framework which can make the load testing much easier and cost effective.

Load testing architecture divides the system into three parts:

1. Client GUI application, usually implemented as a browser application using static HTML or a rich internet application such as Ajax applications.
2. Internet protocol available to access the system under test, usually HTTP/HTTPS but may also include multimedia protocols such as RTP/RTSP or “Web 2.0” protocols such as SOAP/XML or RSS over HTTP.
3. System under test (SUT) which is the subject of our load testing effort.

Performance testing Framework enables repeatedly building new load test agendas without recording a new script, eliminating the additional work that may have been caused by changes to the UI. The Load Testing Framework avoids dependence on the UI by directly calling the pre-defined system APIs, like any other callable object within your JavaScript code. The agenda accesses an external resource to get the parameters for each of the calls.

3PG (360logica Performance Group) works dedicatedly on the Performance testing of versatile application on various industry standard tools.

Commercial tools: LoadRunner, WebLoad, Rational

Open Source tools: OpenSTA, JMeter, Grinder

OpenSTA: It has the capability of performing scripted HTTP and HTTPS heavy load tests with performance measurements from Win32 platforms and much more other features.

JMeter: This is used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Databases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types.

Grinder: It is a JavaTM load testing framework that makes it easy to run a distributed test using many load injector machines.

3PG have successfully designed framework on the open source tools, are customized as per the client’s requirement and fast analysis is done.

These help to achieve the test execution in more reduced time and in very cost effective manner.

Possibly. For small projects, the time needed to learn and implement them may not be worth it unless personnel are already familiar with the tools. For larger projects, or on-going long-term projects they can be valuable.

A common type of automated tool is the 'record/playback' type. For example, a tester could click through all combinations of menu choices, dialog box choices, buttons, etc. in an application GUI and have them 'recorded' and the results logged by a tool. The 'recording' is typically in the form of text based on a scripting language that is interpretable by the testing tool. Usually the recorded script is manually modified and enhanced. If new buttons are added, or some underlying code in the application is changed, etc. the application might then be retested by just 'playing back' the 'recorded' actions, and comparing the logging results to check effects of the changes. The problem with such tools is that if there are continual changes to the system being tested, the 'recordings' may have to be changed so much that it becomes very time-consuming to continuously update the scripts. Additionally, interpretation and analysis of results (screens, data, logs, etc.) can be a difficult task. Note that there are record/playback tools for text-based interfaces also, and for all types of platforms.

Another common type of approach for automation of functional testing is 'data-driven' or 'keyword-driven' automated testing, in which the test drivers are separated from the data and/or actions utilized in testing (an 'action' would be something like 'enter a value in a text box'). Test drivers can be in the form of automated test tools or custom-written testing software. The data and actions can be more easily maintained - such as via a spreadsheet - since they are separate from the test drivers. The test drivers 'read' the data/action information to perform specified tests. This approach can enable more efficient control, development, documentation, and maintenance of automated tests/test cases.

Other automated tools can include:

Code Analyzers - monitor code complexity, adherence to standards, etc.

Coverage Analyzers - these tools check which parts of the code have been exercised by a test, and may be oriented to code statement coverage, condition coverage, path coverage, etc.

Memory Analyzers - such as bounds-checkers and leak detectors.

Load/Performance Test Tools - for testing client/server and web applications under various load levels.

Web Test Tools - to check that links are valid, HTML code usage is correct, client-side and server-side programs work, a web site's interactions are secure.

Other Tools - for test case management, documentation management, bug reporting, and configuration management, file and database comparisons, screen captures, security testing, macro recorders, etc.