Get certified - Transform your world of work today


Agile Testing and the Cost of Assumption

21 September 2017

"Quality is never an accident. It is always the result of intelligent effort."
—John Ruskin

The Cambridge dictionary defines an assumption as something that you accept as true without question or proof. This is a departure from what testing is meant to accomplish in Agile, and even in traditional software development methods.

Software testing is an investigation conducted to provide information about the quality of a product or service under test so that stakeholders can make informed decisions. But there is a strong tendency for testers, developers, and stakeholders to make decisions with the assumption that a product or feature will work as expected, without any form of proof. Most of these assumptions could be based on extrapolation, experience, or even historical data.

Let us look at some quality assumptions, the reasons for them, and their costs.

Time and speed

I have seen stakeholders skipping some testing as they rush to meet milestones because they want to hit the market fast — their reason usually is “we are constrained by time.” The conclusion most times is, “We have decided not to run the planned regression testing because we have no time. The system/functional testing that was conducted will suffice.” Sometimes the planned user acceptance testing (UAT) is moved out of scope for the same reason.

Even though it is not absolutely and always unwise to skip UAT because of the confidence in system/functional testing, what we should note is that the elements of assumption will lead to some uncertainties that may become magnified to the extent of jeopardizing the project at some point. This is because the uncertainties will harbor unseen risks that will continue to grow bigger and bigger. It is not uncommon for a project team to descope UAT, or at the very best conduct it at a very high level. The danger is that those who make this decision are usually oblivious to the huge risks associated with their decisions. It should be very clear that no amount of confidence from system testing can ever be a justification for skipping the UAT.

The potential cost of skipping any testing, or conducting lightweight testing, is huge. Aside from technical debt, there could be hidden costs of fixing issues, retesting, reputational damage, customer dissatisfaction, etc. These costs may be enormous — enough that it could take a while to reestablish a business.

It not a best practice to skip any testing planned. Microsoft, referring to one of Jeff Sutherland’s reviews, wrote about a CMMI Level 5 company that has one of the lowest defect rates in the world and how they were able to systematically double the speed of production and reduce defects by 40% by the following actions:
  • Define acceptance tests when defining the feature.
  • Implement features serially and in priority order.
  • Run acceptance tests on each feature as soon as they are implemented.
  • Fix bugs that are identified as the highest priority as soon as possible.
  • With the above, Jeff Sutherland gives credence to how quality can be achieved in Agile without a compromise.

Continuous integration

Continuous integration is an Agile technical practice for teams so they can deliver a product version suitable for release at any moment. In order to achieve this, it must be an integration procedure that is reproducible at the very least, and largely automated. However, continuous integration is in contrast to "scheduled" integration.

The weak link here in terms of assumption here is that Agile teams sometimes could see continuous integration to be centered on tools instead of an attitude and overarching continuous improvement process that includes testing, automated build processes, and tools for version control.

The other assumption is that Agile teams see continuous integration as continuous deployment rather than a process of eliminating defects while continuously coupling bits of deployment modules.

The cost of not adopting continuous integration, or not practicing it effectively to eliminate defects, could be huge on a project, especially because it’s not cheap to acquire tools, resources, skills, training, etc. It is simply capital intensive and could be an enormous waste if predicated on assumptions and not done properly.

Collective ownership

Product owners and other stakeholders in an Agile project may assume or even believe that the tester embedded in the team is solely responsible for the quality assurance of the product being developed.

For me to explain the benefit of collectively owning quality in Agile projects, it will be easier to use Scrum roles. This is because in more than 10 years of VersionOne Agile surveys, they have shown that companies practicing Agile are using Scrum, or at least the Scrum terminology, more than any other.

In order to answer the popular question of who owns quality in Agile development, it is better to refer to the Scrum Guide, which states that “During each Sprint Retrospective, the Scrum Team plans ways to increase product quality by adapting the definition of ‘Done’ as appropriate.” If the Agile team plans ways to increase product quality, it will be easy to conclude that what to test, how to test, and when to test are all invariably the consensus of the team, not the plans of one tester who is embedded in the team. The Scrum Guide is clear enough on this and does not need embellishing.

Another quote from the Guide states, “As Scrum Teams mature, it is expected that their definitions of ‘Done’ will expand to include more stringent criteria for higher quality. Any one product or system should have a definition of ‘Done’ that is a standard for any work done on it.”

It is clear that the entire Scrum Team owns the product quality. To further buttress that quality is not the exclusive preserve of an Agile tester, The Scrum Guide also states that the product owner is responsible for “optimizing the value of the work the Development Team performs.”

If any project or Agile Team assumes or believes that the testers exclusively own quality, the chances of defects being found late or not found at all is higher. The associated costs (fixing, retesting, and deploying) of defects late in a project will also be higher than when found early — the later a defect is found, the more expensive it becomes.

No Agile team, stakeholders, or anyone should ever make decisions based on assumptions.

If you’re not sure, get an expert opinion; if you’re constrained by time, know the cost of the risk or reprioritize; and if it’s difficult to estimate, get more information. Again, "Quality is never an accident. It is always the result of intelligent effort."


Opinions represent those of the author and not of Scrum Alliance. The sharing of member-contributed content on this site does not imply endorsement of specific Scrum methods or practices beyond those taught by Scrum Alliance Certified Trainers and Coaches.

Article Rating

Current rating: 0 (0 ratings)


Be the first to add a comment...

You must Login or Signup to comment.

The community welcomes feedback that is constructive and supportive, in the spirit of better understanding and implementation of Scrum.


Newsletter Sign-Up