Every testing strategy deck has a pyramid with percentages on it. Ours said 70 percent unit tests, 20 percent interface tests, 10 percent screenshot tests. It looked authoritative. When we checked it against the platform's own guidance, there was nothing behind the numbers.
What the Guidance Says
Android's testing guidance gives a shape and stops there: "Most apps should have many small tests and relatively few big tests" (Android). No ratios. The reasoning is a trade. Small tests are quick and dependable. Large ones are closer to reality and expensive to keep working.
A ratio is attractive because it can be put on a dashboard. It also invites teams to write tests to hit a number. The useful question is whether each layer catches the failures only it can catch.
The Layer Most Teams Skip
Functional tests confirm that a button works. They cannot see that it is the wrong colour or that its label is cut off. The platform now names the answer plainly: "Screenshot testing is the recommended way to verify visual attributes in Compose UIs" (Android). A product whose brand matters and whose suite has no visual checks is testing everything except what customers look at.
A Deadline Nobody Scheduled
There is also a migration waiting in most Android test suites. The entry points nearly every interface test uses have been replaced, and the documentation does not soften it: "The v1 APIs are deprecated, and it's strongly recommended to migrate to the new APIs." (Android). Most tests move with an import change. The ones that break were relying on timing they should not have relied on, which is worth knowing regardless.
What to Ask Your Team
- Where did the ratios in our test strategy come from?
- What would catch a visual regression before release?
- Are we on deprecated test APIs, and is the move planned?
- Which of our tests have been proved able to fail?
-Rocky
#QAStrategy #TestAutomation #SoftwareQuality #EngineeringDreams #StrategiaX
Originally published on ANDROID-ARCHITECT, the Android engineering blog of Strategia-X, with the test code.