In the context of software development, quality assurance (QA) is the process of manually checking whether submitted work meets its specified requirements free of any gaps, errors or usability issues.QA testing is absolutely crucial for productive code reviews. Without it, we simply cannot be sure that someone’s work will function as it’s expected to. Enforcing it as a requirement on all pull requests will improve the effectiveness of your development team and save the wider organisation time in the long run.
Above all else, we need to ensure that code actually does what it’s intended to do before we review, merge then deploy it to production. But this doesn’t quite do it justice, there are more specific reasons QA testing is such an important part of the pull request process.
Commits, diffs and changed files are on their own rather abstract. A reviewer might capably be able to identify faulty or functioning code just by looking at it, however they might just as easily overlook how it actually performs if they don’t test it.For instance, it’s all well and good scanning over some markup responsible for creating part of an interface. However without any testing, the reviewer doesn’t really know what it looks like on screen. As such it’s hard to know what it’s like to interact with, meaning we can’t be sure we’re delivering something useful to the end user.Likewise on the back-end, a reviewer might feel comfortable a new service functions correctly just by reading the code in front of them. However until they actually run it themselves, they can’t be completely sure what will happen. Is it slow? Does it introduce any n+1 queries? Is there any unintended secondary behaviour?This is not an exhaustive list, but the point is that QA testing provides us with substance to the code we are reviewing. It allows the reviewer to see with their own eyes what the submitted changes do.
There is rarely going to be one path a user might take to reach their goal when using our software. As such, the QA review is an opportunity to experiment with new functionality in an effort to anticipate atypical user journeys and unearth any unexpected issues ahead of time.This means thinking about discrete actions and putting oneself in the shoes of somebody using our software. If you’ve added new filters to a table, what happens if you apply all of them then enter something into the search bar? Similarly, perhaps you’re using url query parameters; is there any unexpected behaviour if you supply something you know isn’t expected?None of this is to say we assume users won’t use our software correctly. Developers are generally quite far removed from the end user, and therefore aren’t typically privy to their preferences, habits and expectations.We tend to make assumptions on how we think people will use our software. The QA review exists to try and close this gap by simulating as many different scenarios as possible.
Sometimes we need to rely on QA reviews when automated testing isn’t possible.An example would be trying to test something that relies on an external service. The author knows it’s bad practice to make http requests in a test suite, and has opted to use a stub to mock this action.For the purposes of testing code that interacts with this service, this is perfectly reasonable. However, it does mean that the person reviewing this code can’t be absolutely sure the functionality works without manually testing it.
In addition to helping ensure your organisation releases quality software, we shouldn’t underestimate how valuable QA reviews are for building better development teams.
In most cases, it’s only going to be a good thing to know as much as you can about the software you’re developing. Performing QA reviews can broaden your knowledge of a system by exposing you to functionality you’ve not yet come across.Future code reviews will become easier as a result. Although I might be unfamiliar with certain functionality today, after I have tested it once, I’ll be in a much better position going forward should I have to revisit it in another review.Similarly, identifying any problems when QA testing will help generally increase your awareness for certain types of issues. This might be something as straightforward as page responsiveness — a developer won’t forget being told their work looks great, but is completely broken on mobile.
Consistently performing thorough QA reviews is one way to show your team that you can be relied upon to action pull requests.This is great news for your own career progression. It’s demonstrative that you can be entrusted with more responsibility. Your word alone will start to count for more, meaning you will become more accountable for new functionality.In theory, this is mutually reinforcing, motivating you to be as comprehensive as possible. You wouldn’t want to undermine your hard-earned reputation with subsequent slapdash QA testing.Your team will also reap the benefits. Having another person that can reliably help review pull requests reduces the burden on those that are usually depended upon.Removing bottlenecks like this is a massive win for engineering teams, because it lowers the likelihood that certain people are spread too thin. It’s only a good thing when there are five people you can count on to review and test code at the end of the sprint instead of two.
In an ideal world, anybody taking time to review someone else’s work ought to include QA testing as part of their process.Of course this won’t always be feasible or practical. Code reviews on their own are still valuable as long as someone has performed QA testing.The nature of the work in question is also a consideration. For more straightforward changes, a single QA review might be sufficient.On the contrary, if submitted work includes a major version upgrade to the web framework your platform is running off, for example, you’ll almost certainly want the support of multiple developers helping with testing, to ensure nothing has been missed.
Beyond just constituting a final sanity check, testing your own work allows you to spend time actively trying to find edge cases, unexpected bugs, or any silly mistakes.If you’ve identified and fixed a problem during this process, the reviewer won’t come across it. This could be the difference between your work being approved and it being sent back to you.Although that might sound trivial, it’s not. Code reviews are time consuming, and it’s frustrating to request changes that could have easily been avoided had the author performed one final run through.
There are primarily two parts to the pull request process: the QA review and the code review.By starting with the QA review, you’re able to determine if the changes work as you expect without spending any time reading code.This is more efficient than the alternative. If you begin with the code review, you risk spending time analysing changes that might prove to be erroneous following subsequent QA testing.
Any time the author commits new changes, it’s worth recalling an earlier point: code reviews alone aren’t enough.Obviously there is a degree of reasonability, and you should use good judgement to gauge whether newly submitted code requires testing. You perhaps don’t need to worry if the latest commit changes the name of a few variables.If you have requested changes as part of your initial review though, it makes sense that you should test that the author has addressed those issues. The potential implications of not doing this probably don’t need spelling out.
Although everybody will have their own preferences for code reviews, there are a few guiding principles that can serve as a common basis for QA testing, in light of what has been discussed.
Generally speaking, most pull requests can be traced back to a set of requirements for work that needs to be done.This is your initial point of reference when undertaking a QA review. If the feature spec requests functionality that allows users to send certain data from your platform to a third party via their API, this is what you need to test.Requirements are either met or they’re not; it’s perhaps the most objective measure of success as far as QA testing goes.
Meeting all of a spec’s stipulations doesn’t necessarily mean a feature is ready for your users.Using the example from the last section, during the QA review you uncover that the form used to send data hasn’t been pre-filled with information you know exists within your own platform.Although initial requirements have been met, the implementation is needlessly onerous, and can be improved upon to deliver a more convenient user experience (the solution, of course, is to pre-fill the form fields).It is hard to overstate the importance of usability. We should be aiming to provide our users with polished software that is easy to use, which is difficult to ascertain if we decide to skip QA testing.
“Happy-path” testing involves using known inputs to produce an expected output. Given users will interact with your software in unexpected ways, you should proactively test as many alternative use cases as you can think of.Sticking with the same example, our typical user journey involves opening a form, editing some fields and pressing submit. However, what happens if a user re-submits the form multiple times with the same data, is the endpoint idempotent or are duplicate records endlessly created?We’ll only find out if we simulate this scenario in our QA review. The aim should be to test as many unknown outcomes as possible.
Go a step further and do things that simply don’t make any sense.Add thousands of characters to text fields, put emojis in inappropriate places, put negative numbers into inputs, upload PDFs instead of images, double click buttons. You get the picture.These actions might seem absurd, but we need to be sure we’ve accounted for exceptional circumstances.At some point someone is going to reach out to customer support when faced with an obscure server error. The development team will then spend time investigating, addressing, reviewing and deploying a fix, all the while wishing the original reviewer had just bothered to test what would happen if lorem ipsum in its entirety had been copy pasted into a text input.QA testing seemingly ridiculous steps could save your team hours of time later on.
Although there is no one way to standardise QA testing in code reviews, one approach that I find works well for me is easily repeatable:
When submitting code for review, I include evidence of QA testing in the pull request description. Leaning heavily on screenshots and annotations, my aim is to prove to both the reviewer and myself that the code works.Additionally, I structure this information in an ordered fashion such that it effectively constitutes a guide for testing. This just makes the reviewer’s life easier, especially if it’s an extensive piece of work.
My process for reviewing work aims to be just as demonstrative and communicative.Again, I prioritise screenshots, server logs and annotations to show, step-by-step, that the work has been thoroughly reviewed.It is important to present this information in a readable format, so that you’re easily able to communicate any questions or issues to the author on the conversation thread.My preference is to make use of markdown collapsible sections, allowing you to conceal everything within expandable headed blocks:
<details>
<summary>QA Testing</summary>
## Testing new functionality
### Login to platform and invite a user
<img width="1783" alt="image" src="https://url/link/to/screenshot/1">
### Ensure invite email is sent to user
<img width="1783" alt="image" src="https://url/link/to/screenshot/2">
...
</details>The QA review is ultimately an exercise to ensure we’re delivering value to the people using our software.End users pretty much constitute our system’s raison d’être. As such we ought to be doing everything in our power as an organisation to put them at the forefront of the decisions we make.Embedding QA testing as a core part of your pull request process will help engineering teams deliver better quality software by fashioning more conscientious and attentive developers that ship fewer bugs and more suitable features that consumers can then use to achieve their goals more effectively.