From Quality Assurance to Quality Assistance
[authors note: this is a re-publication of a blog post originally published in April 2018]
One of the key goals of the Engineering group at amaysim is to deliver a continuous stream of features to our customers. A fundamental part of this process is ensuring that the software and features we ship to our customers is always to the highest quality levels we can reasonably achieve. There’s no point in continuously delivering broken software.
As part of achieving this, we recently implemented a change in our quality process, moving away from the traditional ‘Quality Assurance’ model and instead adopting a ‘Quality Assistance’ approach.
Traditional Quality Assurance
Let’s start by discussing the ‘traditional’ approach to Quality Assurance. Shown below is a representation of what our previous Story Lifecycle used to look like, from when a story is picked up for development to when it goes into production. There are various Quality activities baked in into this lifecycle, which focus on early detection of defects, preferably avoiding them in the first place, and a collective responsibility of quality:
The steps in the process explained:
Kickoff: This is a short catchup between the BA, Developer and Tester, (aka 3 Amigos) to kick off the implementation of the story. The aim of this is to ensure everyone involved is on the same page regarding the expectations from this story. The BA explains the context of the story, what it is expected to accomplish, how end users would use this feature etc. Implementation details are discussed and several ‘what if’ scenarios are clarified. The acceptance criteria are reviewed and modified/added to if needed. By making sure everyone knows what to do, this step helps prevent defects from happening.
Coding: As the name suggests, this is where the developer(s) code the implementation of the story. Along with the implementation, automated tests are added too. This includes Unit, Integration and End to End tests (if deemed necessary).
Walkthru: Once the developer is confident his/her code works, they call over the Tester and BA and do a ‘walkthru’. This involves demonstrating the story in action. The 3 Amigos also may try out the acceptance criteria and some ad hoc scenarios, exploring how the software behaves. This step has proved immensely helpful in that it often helps find low hanging fruit, and the developers are able to make quick changes to fix these, as the code is still fresh in their minds.
Testing: Following a successful walkthru, the software is merged into the master branch and deployed into an integrated QA environment. Here, the Tester tests the story, mostly using exploratory testing techniques. If any important bugs are found, the story gets sent back to the Developer for fixing. Once the story is tested successfully, the story is now ready to release. The actual release to production is a business decision, and most projects release stories into production multiple times a week.
This process has served us very well but over time however, a couple of problems became apparent:
Many stories, while in the Testing stage, go through a cycle of bug fixes and retest. Whilst it’s great that bugs get found out in testing rather than getting out into the wild, however, this still introduces inefficiency in the process. Firstly, there’s the rework involved, in fixing the software and retesting it. And secondly, it often happens that due to testers being busy with other stories, the testing may only start a day or two later, by which time the developer has moved on to another story. In this case, when a bug is identified, they’ll have the pause their work, context switch back into the story headspace, and fix the bug. This is disruptive to a continuous flow of work.
Most of our teams are structured in a way that the ratio of testers to developers is quite low, usually 1 tester in a team of 4–6 developers. Assuming each developer works on a separate task/story, this can translate to 4–6 stories in progress at any time. This leads to a pattern shown below, where stories can get queued up waiting to be tested. In this scenario, the Tester becomes a bottleneck.
Sometimes, to mitigate this bottleneck effect, other developers get drafted in to help test. However this is still inefficient, as these developers wouldn’t have been part of the kickoffs or walkthru’s and therefore are lacking context on the story they now have to test.
Introducing Quality Assistance
Quality Assistance is a model that is seeing increasing adoption in the agile space. Under this model, the developers are responsible for testing their own work, with advice and guidance from the Testers. The main function of the Testers is now to help the developers get better at testing through coaching and mentoring as well as helping to build and champion test tooling and establishing testing patterns.
This model resonates well with our Engineering team, especially as we have similar goals for our Devops team (i.e. they play an enabling role, helping teams self-serve their Devops needs rather than being reliant on the Devops team).
So, with consultation among the QA team and the tech stakeholders, we made changes to our process, adopting elements of Quality Assistance. We drew upon the experience of others who’ve traveled this road before. As is commonly said, context is everything, and a process that worked well in one organisation isn’t necessarily a good fit for another. So, we went about customising this to suit us, but keeping the essence intact. We experimented with these changes within a couple of teams and tweaked as necessary and eventually ended up with something that works well for us. This isn’t a complete radical revamp but has some key differences compared to our old way of working, highlighted in purple below:
The main changes are:
Following the Kickoff, we now do a informal Test Analysis session, discussing how we’d test this story. Things like negative cases, edge cases, test data & test environment requirements, automated tests etc. get discussed here (and usually documented along with the story). Coming out of this session, the Developer has a clear idea on how to test this.
As part of the Coding stage, the Developer is now expected to perform the testing too, using information from the Test Analysis session as guidelines. The bulk of the testing would be done at this stage.
Once the Walkthru is done successfully, the Tester and the Developer together review the testing that was performed so far, with a view to identify what further testing needs to be done. Sometimes, the two may do a paired exploratory testing session for the story. If any further testing gaps are identified, the developer goes off and performs those tests too.
Note that the Tester is still very much a part of the story life cycle. They provide guidance regarding tests both before and after coding is done. Also, for more involved/trickier stories, the Tester sits together with the Developer to test various end to end scenarios. However, the expectation is that the Developer still owns testing of the story, and the Tester plays a consulting role, and does not take the work off and go do it by themselves. It has to be a paired activity so that over time, the developers become more and more self sufficient.
Some of the advantages this gives us:
Improve testing skill set of developers leading to them looking at their own software more critically, thinking upfront of scenarios under which it can break. By doing this well, we prevent most bugs from happening in the first place.
Less bug fixing means less rework, hence a more efficient process.
Improve domain knowledge within team over time, as integration testing of stories force you to learn other parts of the software, thus breaking down knowledge silos.
The Tester can take now on more bigger-picture QA tasks, that facilitate and enable efficient testing.
So in this model, what does a Tester actually do?
At a Story level:
As explained earlier, the Tester works closely with the Developer helping identify test scenarios before coding commences (Test Analysis stage), and reviews the testing done, discuss test ideas, and identify gaps following Walkthru. In addition, where needed, the Tester pairs with the Developer to test stories, thus coaching valuable exploratory testing skills.
At a Feature level:
For larger features (composed of many stories) that cut across multiple teams & applications, a more holistic view of the solution is needed to test effectively. The developers still own testing of the individual stories, but the planning and coordinating of end to end testing for such big sized features is owned by the QA Engineers. Such feature releases happen frequently enough for us to take this factor into consideration. For example, within the last one year, as the business expanded to more verticals, we released funnels and self service portals for Broadband, Energy (Electricity & Gas) & Devices. With various teams involved in developing different parts of these solution, you can see why there is need for End to End testing to ensure all these play well together, and the QA Engineers are best placed to coordinate and run these.
At a Team level:
There are various activities that improve the ease of testing within teams. With less time being spent on hands-on testing of stories, the QA Engineers now have time to spend on working on these kind of tasks. This could include creating automated test frameworks that other developers later add tests to, improving the run time of automated tests (using parallelization or other techniques), looking into reducing flakiness of tests, improving test data creation tools, exploring ways of improving how we do things etc.
Some FAQs:
Wouldn’t this increase a Developer’s workload now that they have to do the hands-on testing of their stories too?
It has always been an expectation that the Developers test their own code before turning it over. What this process does is help improve the quality of testing done at this stage. Better testing early on leads to less bugs, leading to less rework.
While introducing this, I expected to have a bit of apprehension from Developers who may be concerned about more work coming their way. However, I’m pleasantly surprised by how enthusiastic the Developers are to take on this responsibility.
If the same Developer that developed the code is now responsible for testing it, wouldn't they have a biased view of it? Wouldn't they test it with the same (possibly incorrect) assumptions etc they had when coding it? Where is the safety net?
Some stages in the process are there specifically to address this. The Kickoff, Test Analysis and Walkthru are opportunities for other sets of eyes to look at it with their own perspectives.
In addition to this, as mentioned before, Integration testing of larger features is still owned and run by QA Engineers.
Wouldn’t I need quite a bit of domain knowledge to test effectively?
Yes, indeed. Testing some stories will require knowing how other applications/feature interact with the story. Use this as an opportunity for developers to build up domain knowledge. The most effective way of resolving this is to work together with a Tester to test this as a pair (but not just outsourcing the work to them). So, over time knowledge is spread and there’s less of these silos.
The tests discussed during kickoff — is that ALL I need to test?
Likely not. Use those test ideas as a guide rather than checklist.
Summing Up
The move from Quality Assurance to Quality Assistance is one that we’ve already seen benefits from. We’ll continue to tweak and modify our process and it will evolve over time. But the central theme of empowering our engineers to test their own code not only helps improve the quality levels of the software we ship to our customers, but also keeps moving us along the path to true Continuous Delivery.
…
Has any of the above piqued your interest? Does amaysim sound like the sort of place where you think you could make an impact? Do you thrive in organisations where you are empowered to bring change and constant improvement? Why not take a few minutes to learn more about our open roles and opportunities and if you like what you see then say hi, we’d love to hear from you..
The views expressed on this blog post are mine alone and do not necessarily reflect the views of my employer, Optus Administration Pty Ltd.
