Written by 
Among diverse approaches, the teams can find a unique way on their test automation journey and select the strategy that matches their needs. So let’s dive deeper into pyramids, diamonds, hourglasses and other interesting objects.
In the massive hype around automation in the QA field, it is easy to get lost in various approaches. The multiple perspectives on automating tests coexist. There is no recipe for success but there are some common-sense patterns to follow. Responding to the feedback of the readers of Test smart: how to apply automation and stay sane?, I’ve decided to leave my two cents on automation strategies that might be handy for agile teams.
Sometimes having no strategy is also a strategy. As someone who lives in a Mediterranean country, I have noticed an interesting thing about harvesting olives. Some bigger olive oil makers have used electric rakes to collect the olives from the trees. At the same time, others like us, who possess only five not-so-high olive trees, have picked the yield manually.
The same refers to the reality of testing. If your product is small-scale (a few quite low olive trees — e.g. a landing page with registration and login flows), it may be too early and pricy to automate UI tests. Instead, the manual tests could be done promptly.
This works well when you are developing application prototypes or working on Proof of Concept, where the focus is more on rapid feature development than maintaining automated tests.
Also, if the user flows are raw, it will be a waste of time and budget if you start automating tests. Once the rework is needed in the UI/UX, some intervention will be required for the automated tests, too.
Yet, if your product matures and more olive trees — flows — are added, get ready to start using electric rakes: automate the tests for the most critical flows where applicable.
If you intend to make a lot out of automating your testing efforts, think strategically and involve the team members in a discussion. From my observations, the discussions might be very hot.
Do you remember the old test automation pyramid?
The last time I talked about it to developers, some guys had yawning faces. Yet what if we turn the pyramid upside down to understand it better? Magically, here the pyramid turned into the bug filter.
Look how multiple bugs fly into it:
➡ Unit tests could filter bugs faster (that’s why more unit tests are needed!);
➡ Integration tests will be your safety net in the middle (you should not neglect this part);
➡ End-to-end (UI) tests will catch the bugs at the latest stage (and more rework will be required).
As Janet Gregory and Lisa Crispin note:
“Teams that practice test-driven development (TDD) build up a solid base of unit- and component-level tests that help guide code design. In most contexts, teams want to have the biggest proportion of their automated tests at the lowest level of the pyramid.”
Furthermore, it is smart not to wait until the bugs fly into the product’s UI layer. Think whether your team can create more tests for the prior layers of the pyramid (unit, integration) and catch the defects early.
Anyhow, let’s be realistic: the pyramid (bug filter) is the goal that should be targeted. However, in the real world, you and your teammates might agree that your current automation approach reminds you more of other objects.
For instance, your current automation strategy might be reflected in a diamond, where integration tests are in favour. It makes sense for some types of projects. Once your product is based on microservices, integration tests could bring more value than unit tests or end-to-end (UI) tests.
In short, integration tests check how an application behaves once it interacts with other services. Usually, the mocks represent external services so that integration tests run in isolation. It is important to note that the main advantage of integration tests is that they are not as flaky as end-to-end tests in terms of maintenance.
Your approach can also look like an hourglass: there is a large suite of unit tests and there is a certain part of end-to-end (UI) tests, yet there is low number of integration tests or they are missing at all. This could work in certain cases but due to the lack of integration tests, the teams risk that more bugs may fly into the UI layer.
As a result, there could be too many end-to-end test failures. You can avoid this case by adding a suite of medium-scope tests — integration tests.
Besides, there might be a pattern of an inverted pyramid, or ice cream cone: many end-to-end (UI) tests are automated, and there are quite a few integration tests but very few unit tests.
Overall, this pattern should be avoided when planning a testing strategy. The chances of detecting bugs in later stages of development are getting higher, thus it may cost a lot to make additional adjustments. Nevertheless, according to Kristijan Kralj, there are a few exceptions where the inverted pyramid approach could be useful.
“The cases where this makes sense are:
- Putting legacy code into test automation — you are trying to add some automated tests to your application, but UI and business logic are tightly coupled. This means that it’s not possible to write unit tests, but you need to start with UI and integration tests. After you decouple UI from business logic, then you might add unit tests.
- Your app mainly calls other API-s — if your application’s logic consists mainly of calling other libraries and/or performing lots of network requests, then it doesn’t make too much sense to write unit tests for it.
- Interaction with the real device — if you are working on an application that will be deployed on a physical device, then you might benefit more from having UI tests that will check the application works fine.”
Traditionally, there is a debate about whether test automation should be the sole responsibility of QA Engineers or Developers. To my mind, the truth is in the middle. Janet Gregory and Lisa Crispin suggest:
“Because automating tests through the UI tends to be more time-consuming, there’s a temptation to hand that off to a separate automation team or have the testers on the team take full responsibility for it. We recommend that the developers, who are good at writing efficient, maintainable code, work together with the testers, who are good at specifying test cases, to automate tests through the UI as well as all other layers above the base level of the pyramid. Having testers and programmers pair or work in ensembles to automate tests saves time. Your team will enjoy better-designed tests whose results you can trust.”
From my experience, the unit tests are more in the domain of Developers, whereas creating integration (e.g. API) and end-to-end tests could be done collaboratively. Surely, the QA Engineer could give a hint about tests that should be picked for automation, and together with the Developer, they could speed up the test automation endeavour. After all, let’s keep in mind that quality is a team effort.