A sneak peek into ‘test framework’, ‘test pyramid’ & ‘testing pyramid’
What is a test framework?
A test framework is simply a formalisation or statement of the method and processes followed when testing within any given organisation, based on the complete test approach. Expressed as a collection of assumptions, concepts and tools this can be implemented by automated software testing.
Test frameworks are popular because they are low maintenance, and because changes can be made to test case files without having to update things like Driver and Startup scripts. This keeps things clean and simple and reduces the risk of mistakes. Choosing the right framework technique also helps in maintaining lower costs, as the costs associated with test frameworks are due to development and maintenance efforts.
What is test pyramid?
The test pyramid is a concept developed by Mike Cohn. Its essential point is that you should have many more low-level unit tests than high level end-to-end tests running through a GUI. A test pyramid delivers a graphical representation of a best-case test scenario where you have a large number of low-level unit tests (around 70%) and comparatively few high level end-to-end system tests (about 10%), with an intermediate layer of integration tests sandwiched in between which adds up to around 20%.
Why opt for testing pyramid?
The testing pyramid is a concept that can help you better balance your tests, speeding up your test suite and reducing the cost of changing the functionality of your applications. It centers on the composition of different types of tests in your suite. It’s an incredibly efficient and logical way to cover a multitude of alternative code paths, application, and combinations of tests. Sometimes it is best to test a small part of the system in isolation; other times a tester will need to look at a group of objects and how they talk to one another; and on occasion will need to test the whole system as one. From a test customer perspective, pyramid testing is great because it’s faster and more efficient, and cuts costs.
The necessary ‘ad-ons’ to the pyramid
The pyramid might be an ideal framework for testing, but it’s not necessarily a complete guide. There are a couple of other tests that should be executed along the way, and they’re not included in the pyramid as they are difficult too. These include:
Acceptance Tests – closely related to the stakeholders of the application and defined in the language of the business, these tests attempt to mimic the user completely.
Contract Tests – more specialised and less commonly used, these check that the application works with all of the third party components the business depends on.
The Upside Down Testing Pyramid
Most software organizations today suffer from Upside Down Testing Pyramid – ‘Inverted Testing Pyramid’ problem. They spend maximum time and effort manually checking software. Some invest in automation, but mostly building slow, complex, fragile end-to-end GUI test. Very little effort is spent on building a solid foundation of unit & acceptance tests.
This over-investment in end-to-end tests is a slippery slope. Once you start on this path, you end up investing even more time & effort on testing which gives you diminishing returns. They end up with majority (80-90%) of their tests being end-to-end GUI tests. Some effort is spent on writing Integration test (typically 5-15%). Resulting in a shocking 1-5% of their tests being unit/micro tests.
A properly tested application feature requires both unit tests and acceptance tests. Start by making sure you have good unit test coverage. Your unit tests should catch edge cases and confirm correct object behavior. Carefully supplement your unit tests with acceptance tests that exercise the application like an end-user. These will give you confidence that all of the objects are playing their role well. Teams often end up with way too many of these tests, slowing development cycles to a crawl. If you have 20 Capybara-based tests for user registration to confirm that all validation errors are handled correctly, you’re testing at the wrong level. This is known as the Inverted Testing Pyramid anti-pattern.
Also read: Test Automation Frameworks- Then & Now!