Testing all possible scenarios (all permutation and combination of functionalities) is called exhaustive testing. Exhaustive testing is usually done when the programs and the scope of project is small. For bigger projects exhaustive testing is impractical and is not used. Exhaustive testing is time consuming and costly, thus it has only theoretical significance.
Exhaustive testing is usually impossible, there are a few possible options that can help when applied either individually or in combination with each other:
- Support Less – Sometimes we can’t support every platform or combination. This option reduces the number of potential customers for product, but it makes developing and testing easier. This is a good choice for products with narrower markets.
- Test More – We can’t test everything, but maybe we can test more than before. This option raises costs, but it’s a good choice when stability is crucial.
- Make it Simpler – Leave out complicated features or simplify the design. Leaving out features may mean that customers are unsatisfied, but it can also mean a product is ready for shipping sooner.
- Ship with Bugs – Maybe the bugs doesn’t matter. In this case, one accepts that product is buggy but they ship it anyway. Customers may not be happy when things crash, but it may not matter. This is a good choice if they know the bugs can be managed, or if their customers don’t really care about stability.
When testing is used as a technique to establish a property, the property of interest determines in large part the number of tests required. This is the case no matter whether the goal is to establish properties in a statistical sense or in the sense of a proof using exhaustive testing. By careful definition of the property and by the application of a technique called specification limitation, we have been able to prove a number of significant properties of a large software system by exhaustive testing.
Exhaustive testing is infeasible. However, this infeasibility is a direct result of the goal of testing for overall functional correctness. In considering the general issue of how safety-critical systems might be tested, we have concluded that a different view of testing is required.