The modern Testing Pyramid for Continuous Integration
When you start creating a CI pipeline, you have to recall your knowledge about the different kinds of tests you know. You do some research, you realize you will need some functional and performance tests, some unit, component and acceptance tests, you also need end-to-end tests. Then of course these are all automated (as opposed to manual tests), then you go further down in the methodology, you encounter black box and white box testing and so on.
Whenever I read an article about testing, I have always felt that terminology is unclear, and one term can have different meanings depending on the aspect I examine.
Always create a nomenclature first and agree on it with your team or company
This nomenclature or glossary should define the level of granularity, the purpose of test, the technical setup of the test, whether considered functional or non-functional.
Here is my a testing pyramid
This testing pyramid is created regarding a “generic” enterprise-grade web application with mostly functional testing, and perhaps using a microservices architecture.
Let me explain
The main idea of the testing pyramid represents testing levels and amount of tests on these levels i.e. you need a lot of small simple unit tests and fewer system tests.
But these testing levels have some more implications:
- granularity of SUT (system under test)
- dependencies
- complexity of test setup
- initial investment of test implementation
- timespan of test executions
- testing environment and dependencies
I use the levels unit, integration, component (or module) and system tests, and explain the main purpose of these levels (see figure above).
Functional vs Non-Functional tests
Orthogonal to these levels are whether we are testing functional or non-functional aspects of the software.
Functional tests are every test that verify that the SUT can fulfill it’s main business function e.g. the user can register, purchase an item in the webshop and so on. Or a method can sum up the price of the basket in a webshop in a unit test.
Non-Functional tests are usually performance tests, security tests and so on. I consider load testing, throughput etc. as subtype of performance tests because performance can be measured by different KPIs.
It’s easy to see that both functional and non-functional tests can appear in every levels. You may want to monitor and verify the speed of a particular algorithm, and also a complex business transaction that uses multiple components. One is unit test level, the other one is system test level.
Technical setup
The implementation of tests largely depend on the technology and software development tools you are using. There are some generic approaches like in-process and out-of-process testing, the former is rather a gray box, the latter is a black box kind of test. But how you achieve the correct test setup depends largely on the programming language, the build tool, the frameworks and application server you are using, the test runner, the CI/CD tool and so forth. For system tests you need to consider the automation of packaging and automated deployment of your application (microservices).
The implementation of your tests and CI/CD pipeline is a continuous effort. My advice is try to use established conventions and don’t reinvent the wheel.
Acceptance or End-to-End tests?
Sure they are on the system test level, but what is exactly the difference?
Well, acceptance test is more of a term for custom software project development, where at the end of the release cycle the customer has to accept sign off the software handover, so they do a thorough acceptance testing, and if it fulfills the acceptance criteria, the software is accepted by the customer and the project is delivered successfully (on time, without bugs of course)
In a CI/CD environment there is no separate acceptance test as the new version will be rolled out if all automated tests were green throughout the whole testing pyramid.
In my opinion, the term end-to-end test is more appropriate because it better represents that the full software system as a whole is tested.
A thorough example of end-to-end test is user registration including web ui, receiving verification e-mail, setting up 2nd factor authentication in the process.
Component integration test
Whether the term component integration test is the best, is easily debatable. That is the reason that every team or company should create their own nomenclature to avoid confusion.
In my case I wanted to separate system tests that use the web UI (for example via Selenium) from those that merely test the back channels such as REST APIs, Websockets or message queue events.
The reason for this separation was organizational i.e. frontend and backend teams were separated, and we needed to know to be able to test the backend and frontend separately. Yet another aspect of how you organize your tests: organizational structure.
Conclusion
Before creating your CI pipeline you have to plan and design your testing methodology. The first step should always be the establishment of a common understanding. It does not matter if you call your “integration tests” “component tests” or vice versa as long as you are consistent about your naming, and consequent about your testing pipeline.
Recommended: https://martinfowler.com/articles/microservice-testing/
