360Logica.com

Software Testing has lot of challenges both in manual as well as in automation. Generally in manual testing scenario, developers pass through the build to test team assuming the responsible test team or tester will pick the build and will come to ask what the build is about? This is the case in organizations not following so-called ‘processes’. Tester is the middleman between developing team and the customers, handling the pressure from both the sides.

So here we go with the top challenges in software testing and assurance

1. Testing the complete application
   Is it possible? There are millions of test combinations. It’s not possible to test each and every combination both in manual as well as in automation testing. If you try all these combinations you will never ship the product.
2. Misunderstanding of company processes
   Some times you just don’t pay proper attention what the company-defined processes are and these are for what purposes. There are some myths in testers that they should only go with company processes even these processes are not applicable for their current testing scenario. This results in incomplete and inappropriate application testing.
3. Relationship with developers
   Big challenge. Requires very skilled tester to handle this relation positively and even by completing the work in testers way. There are simply hundreds of excuses developers or testers can make when they are not agreeing with some points. For this tester also requires good communication, troubleshooting and analyzing skill.
4. Regression testing
   When project goes on expanding the regression testing work simply becomes uncontrolled. Pressure to handle the current functionality changes, previous working functionality checks and bug tracking.
5. Lack of skilled testers
   This can be called this as ‘wrong management decision’ while selecting or training testers for their project task in hand. These unskilled fellows may add more chaos than simplifying the testing work. This results into incomplete, insufficient and ad-hoc testing throughout the testing life cycle.
6. Testing always under time constraint
   Hey tester, we want to ship this product by this weekend, are you ready for completion? When this order comes from boss, tester simply focuses on task completion and not on the test coverage and quality of work. There is huge list of tasks that you need to complete within specified time. This includes writing, executing, automating and reviewing the test cases.
7. Which tests to execute first?
   If you are facing the challenge stated in point no 6, then how will you take decision which test cases should be executed and with what priority? Which tests are important over others? This requires good experience to work under pressure.
8. Understanding the requirements
   Some times testers are responsible for communicating with customers for understanding the requirements. What if tester fails to understand the requirements? Will he be able to test the application properly? Definitely Not! Testers require good listening and understanding capabilities.
9. Automation testing
   Many sub challenges – Should automate the testing work? Till what level automation should be done? Do you have sufficient and skilled resources for automation? Is time permissible for automating the test cases? Decision of automation or manual testing will need to address the pros and cons of each process.
10. Decision to stop the testing
    When to stop testing? Very difficult decision. Requires core judgment of testing processes and importance of each process. Also requires ‘on the fly’ decision ability.
11. One test team under multiple projects
    Challenging to keep track of each task. Communication challenges. Many times results in failure of one or both the projects.
12. Reuse of Test scripts
    Application development methods are changing rapidly, making it difficult to manage the test tools and test scripts. Test script migration or reuse is very essential but difficult task.
13. Testers focusing on finding easy bugs
    If organization is rewarding testers based on number of bugs (very bad approach to judge testers performance) then some testers only concentrate on finding easy bugs those don’t require deep understanding and testing. A hard or subtle bug remains unnoticed in such testing approach.
14. To cope with attrition
    Increasing salaries and benefits making many employees leave the company at very short career intervals. Managements are facing hard problems to cope with attrition rate. Challenges – New testers require project training from the beginning, complex projects are difficult to understand, delay in shipping date!

Source: http://www.softwaretestinghelp.com

Have you performed software installation testing? How was the experience? Well, Installation testing (Implementation Testing) is quite interesting part of software testing life cycle.

If your installation is successful on the new system then customer will be definitely happy but what if things are completely opposite. If installation fails then our program will not work on that system; not only this but can also leave user’s system badly damaged. User might require to reinstall the full operating system.

In above case will you make any impression on user? Definitely not! Your first impression to make a loyal customer is ruined due to incomplete installation testing.

What you need to do for a good first impression? Test the installer appropriately with combination of both manual and automated processes on different machines with different configuration. Major concern of installation testing is Time! It requires a lot of time to even execute a single test case. If you are going to test a big application installer then think about time required to perform so many test cases on different configurations.

We will see different methods to perform manual installer testing and some basic guideline for automating the installation process.

To start installation testing first decide on how many different system configurations you want to test the installation. Prepare one basic hard disk drive. Format this HDD with most common or default file system, install most common operating system (Windows) on this HDD. Install some basic required components on this HDD. Each time create images of this base HDD and you can create other configurations on this base drive. Make one set of each configuration like Operating system and file format to be used for further testing.

How we can use automation in this process? We will make some systems dedicated for creating basic images (use software’s like Norton Ghost for creating exact images of operating system quickly) of base configuration. This will save you tremendous time in each test case. For example if time to install one OS with basic configuration is say 1 hour then for each test case on fresh OS you will require 1+ hour. But creating image of OS will hardly require 5 to 10 minutes and you will save approximately 40 to 50 minutes!

You can use one operating system with multiple attempts of installation of installer. Each time uninstalling the application and preparing the base state for next test case. Be careful here that your uninstallation program should be tested before and should be working fine.

Installation testing tips with some broad test cases:

• Use flow diagrams to perform installation testing. Flow diagrams simplify our task.
• If you have previously installed compact basic version of application then in next test case install the full application version on the same path as used for compact version.
• If you are using flow diagram to test different files to be written on disk while installation then use the same flow diagram in reverse order to test uninstallation of all the installed files on disk.
• Use flow diagrams to automate the testing efforts. It will be very easy to convert diagrams into automated scripts.
• Test the installer scripts used for checking the required disk space. If installer is prompting required disk space 1MB, then make sure exactly 1MB is used or whether more disk space is utilized during installation. If yes flag this as error.
• Test disk space requirement on different file system format. Like FAT16 will require more space than efficient NTFS or FAT32 file systems.
• If possible set a dedicated system for only creating disk images. As mentioned above this will save your testing time.
• Use distributed testing environment in order to carry out installation testing. Distributed environment simply saves your time and you can effectively manage all the different test cases from a single machine. The good approach for this is to create a master machine, which will drive different slave machines on network. You can start installation simultaneously on different machines from the master system.
• Try to automate the routine to test the number of files to be written on disk. You can maintain this file list to be written on disk in an excel sheet and can give this list as an input to automated script that will check each and every path to verify the correct installation.
• Use software’s available freely in market to verify registry changes on successful installation. Verify the registry changes with your expected change list after installation.
• Forcefully break the installation process in between. See the behavior of system and whether system recovers to its original state without any issues. You can test this “break of installation” on every installation step.
• Disk space checking: This is crucial in the installation-testing scenario. You can choose different manual and automated methods to do this checking. In manual methods you can check free disk space available on drive before installation and disk space reported by installer script to check whether installer is calculating and reporting disk space accurately. Check the disk space after the installation to verify accurate usage of installation disk space. Run various combination of disk space availability by using some tools to automatically make disk space full while installation. Check system behavior on low disk space conditions while installation.
• As you check installation you can test for uninstallation also. Before each new iteration of installation make sure that all the files written to disk are removed after uninstallation. Some times uninstallation routine removes files from only last upgraded installation keeping the old version files untouched. Also check for rebooting option after uninstallation manually and forcefully not to reboot.

The Client

Customer is leading player in online marketplace which directly connects buyers and providers through a comprehensive and collaborative management platform.

The Requirements

• Human Effort: Reduce the human effort in the functional test of the application.
• Sanity Test: Perform Sanity test of the application on the daily build.
• Regression: Regression testing of the application on any changes in the code.
• Scope of Automation – Perform functional, UI, client validation, Database validation and Exception handling.

The Solution

• Designed the framework which is the combination of data driven, library and keyword driven framework.
• Framework makes the connection to the database, maintains library and uses user defined function through out the script.
• Scripts are written using Selenium IDE and in C# on Visual Studio 2008.
• Exceptions are handled throughout the scripts and capturing all objects on the screen.
• Scripts are designed for both http & https protocol.

The Technology

• Microsoft .NET
• AJAX
• SQL Server
• IIS Server
• Selenium IDE
• Selenium RC 0.9.0
• Microsoft Visual C# 2008 Express Edition
• NUnit 2.4.7

Contribution

• Successful competition of robust scripts which performs sanity test of daily build application and reduces the manual effort to 10%
• Exhaustive coverage of test cases and scenarios help to achieve the consistent testing cycle in minimal time on demand.
• Breakage in existing feature can be caught in very early stage.

You can also download the case study here.

How to monitor Unix resources from controller?

There is no need for any LoadRunner installation to be on a Unix/Linux Machine to be monitored. However, the machine must have the default RSTAT daemon installed and running. The controller establishes a UDP socket connection with the UNIX machine. It first communicates with port 111 on the Unix machine and this port is then mapped to the dynamic port on which the rstatd is working. The controller then queries rstatd and all communication takes place through this established UDP socket connection.

Starting the rstatd process in Unix

To monitor UNIX resources, you must configure the rstatd daemon. Note that the rstatd daemon might already be configured, because when a machine receives an rstatd request, the inetd on that machine activates the rstatd automatically.

To verify whether the rstatd daemon is already configured:

The rup command reports various machine statistics, including rstatd configuration. Run the following command to view the machine statistics:

>rup host

A remote host will only respond if it is running the rstatd daemon. If you do not receive a response, or if you receive an error message, the rstatd daemon is not configured.

To configure the rstatd daemon:

1. Run the command: su root
2. Go to /etc/inetd.conf and look for the rstatd row (it begins with the word rstatd). If it is commented out (with a #), remove the comment directive, and save the file.
3. From the command line, run:
   > kill -1 inet_pid
   where inet_pid is the pid of the inetd process. This instructs the inetd to rescan the /etc/inetd.conf file and register all daemons which are uncommented, including the rstatd daemon.
4. Run rup again.
   If the command still does not indicate that the rstatd daemon is configured, contact your system administrator.

Which port is rstatd daemon running on?

You can run a UNIX utility called rpcinfo and identify the rstatd's port number. By running > rpcinfo -p you will receive a list of all RPC servers registered in the host's portmapper, along with the port number.

This video explains how data driven testing can be done using TestComplete to verify different test cases.

Data driven testing means using a single test to verify many different test cases by driving the test with input and expected values from an external data source instead of using the same hard-coded values each time the test runs. This way, it can be tested how the application handles various input without having a lot of similar tests that only have different data sets.

The data-driven testing approach can be used with unit and functional testing. It is especially useful in scenarios that are too cumbersome to execute with manual testing, for example, when an application needs to be quickly supplied a large amount of input values.