In Agile, we believe in the shift left testing approach to software and system testing. This means testing is performed earlier in the lifecycle (i.e. moved left on the delivery timeline). It is the first half of the maxim "Test early and often." The continuous delivery method provides many new challenges for test engineers. With the iterative development approach, it’s important to maintain the quality of the product in every step of the delivery cycle. This approach is often called continuous testing - this means testing from the very first sprint, reducing bottlenecks and handover times by combining various tools and practices. It is also ones associated so far with different roles in the engineering process. Testing in continuous delivery means building and maintaining a quality assurance strategy in the whole development and delivery cycle.

Test-driven development (TDD) is an advanced technique of using automated unit tests (referenced in the above Pyramid) to drive the design of software and force decoupling of dependencies. This first step, writing the test, requires the developer to determine the outcome of a small segment of code. The test, in a sense, then becomes the documentation of the requirement.

 

Before reading more, let’s define what is a Unit Test and then why automate?

 

Unit testing is the testing of a small "unit" of code, typically performed by the developer of the code that's being tested. Ideally, the unit tests themselves are coded so that they can be run over and over again in an automated way.

 

For example, let's say you have a piece of code that is supposed to accept a date representing a birthday. Your application wants to find out the age (in days) of the person who has this birthday. The pseudo-code for this routine might look something like this:

 

GetAge(birthday):

 

If birthday NOT valid format

 

Return error message: Invalid date format

 

today = system date

 

Convert today and birthday to integers that represent days

 

If birthday > today

 

Return error message: Birthday cannot be in future

 

If birthday == today

 

Return message: Today is your Birthday! Happy Birthday!

 

today – birthday = age_in_days

 

Return message: You are age_in_days days old

 

Now, in order to unit test this code thoroughly, the developer would want to make sure each line of code is executed and validate that it performs as expected. In order to do that, he'd need tests in which the birthday was entered with an invalid format, tests where the birthday was in the future, a test where the birthday was the current date, and tests where the birthday was in the past. For example, calling GetAge with various parameters such as GetAge(2/29/60), GetAge(10204020), GetAge(20/52/2011) and checking that the outputs are correct.

 

Now automate

 

However, rather than manually initiating tests and checking for validity, it would be more efficient to automate those Unit Tests as part of the Continuous Integration process.

Once these tests are written, they can easily be executed as part of regression tests. Later, if the GetAge function changes internally, GetAgeTest will ensure that all the test conditions are still executing correctly.

 

The result of using this practice is a comprehensive suite of unit tests that can be run at any time to provide feedback that the software is still working. This technique is heavily emphasized by those using Agile development methodologies. The concept is to “get something working now and perfect it later.” After each test, refactoring is done and then the same or a similar test is performed again. The process is iterated as many times as necessary until each unit is functioning according to the desired specifications.

Characteristics of a Good Unit Test

A good unit test has the following characteristics.

  • Runs fast, runs fast, runs fast. If the tests are slow, they will not be run often.
  • Separates or simulates environmental dependencies such as databases, file systems, networks, queues, and so on. Tests that exercise these will not run fast, and a failure does not give meaningful feedback about what the problem actually is.
  • Is very limited in scope. If the test fails, it's obvious where to look for the problem. Use few Assert calls so that the offending code is obvious. It's important to only test one thing in a single test.
  • Runs and passes in isolation. If the tests require special environmental setup or fail unexpectedly, then they are not good unit tests. Change them for simplicity and reliability. Tests should run and pass on any machine. The "works on my box" excuse doesn't work.
  • Often uses stubs and mock objects. If the code being tested typically calls out to a database or file system, these dependencies must be simulated, or mocked. These dependencies will ordinarily be abstracted away by using interfaces.
  • Clearly reveals its intention. Another developer can look at the test and understand what is expected of the production code.

Benefits of TDD

  • The suite of unit tests provides constant feedback that each component is still working.
  • The unit tests act as documentation that cannot go out-of-date, unlike separate documentation, which can and frequently does.
  • When the test passes and the production code is refactored to remove duplication, it is clear that the code is finished, and the developer can move on to a new test.
  • Test-driven development forces critical analysis and design because the developer cannot create the production code without truly understanding what the desired result should be and how to test it.
  • The software tends to be better designed, that is, loosely coupled and easily maintainable, because the developer is free to make design decisions and refactor at any time with confidence that the software is still working. This confidence is gained by running the tests. The need for a design pattern may emerge, and the code can be changed at that time.
  • The test suite acts as a regression safety net on bugs: If a bug is found, the developer should create a test to reveal the bug and then modify the production code so that the bug goes away and all other tests still pass. On each successive test run, all previous bug fixes are verified.
  • Reduced debugging time!

 

TDD is slightly different than Behavioral Driven Development (BDD) and Acceptance Test Driven Development (ATDD). Below is a table that summarizes the differences, but also states the value of BDD and ATDD.

 

User Story Functional Testing

 

Example User Story & Acceptance Criteria

 

As a recurring customer, I want to reorder items from my previous orders so I don’t have to search for them each time.

  • AC1. Order history option is displayed on accounts page.
  • AC2. Previously purchased items are displayed when clicking on order history.
  • AC3. User may add previously ordered items to the cart.

 

Visualize the Workflow

 

Whether you have been testing this website (or app) for years or this is your very first time testing it, start with visualizing the workflow. What do you see when the user clicks “Order history”? What happens when they click “Add to cart”? If the user clicks the “Back” button, what is displayed? If you know the application well, you can probably quickly visualize this and get started with test case writing. If you are not as familiar with the website, you will want to observe other workflows in the application. For example, observe the behavior when clicking “Add to cart” from general search results.

 

Create the Happy Path

 

Start with the happy path, a route which should encounter no errors or exceptions. It can cover a large portion of the workflow, and if it doesn’t work smoothly, the rest of your testing may be blocked. In this example, the happy path would be something like this:

  • Purchase items from general search
  • Click order history from accounts page
  • Verify that previously ordered items are displayed
  • Add items to cart from the previously ordered list
  • Complete purchase of previously ordered items

 

When the code is delivered to me for testing, this would be the first test case to execute.

 

Standard Items

 

Another quick and easy test case in this example, would be to verify that standard objects are displayed on the new pages. Items such as headers, footers and standard menu options can easily be missed if you don’t document them in a test case. I’ve seen links to things such as terms and conditions that are broken when a new page is added. Create test cases for verifying these standard pieces. You may have tested them so many times that you just expect them to work, and don’t consider them. A test case will save you from missing this.

 

Negative Testing

 

What are all the possible error cases that can happen? Here are some examples that come to mind with this user story:

  • Item is out of stock. They ordered it previously, but now it’s out of stock. Does this information display on the order history page?
  • Price has changed. Is this reflected on the order history page.
  • System issue occurs. What errors are displayed throughout the website when a system issue occurs? Create a test case to verify every error is displayed when it should be.

 

Boundaries

  • How many items from order history should be displayed? If it’s not in the acceptance criteria, this needs to be addressed with the BA or product owner. The developer may code this to display all items from a customer’s history but what happens when a user has an order history of 5,000 items?
  • On the other hand, what if the user has never placed an order? Should the Order history option be displayed in the accounts page? If so, what text is displayed when the user navigates to order history?

 

These are examples of testing for the maximum and minimum boundaries. After asking questions like these, the product owner or BA may decide they need to create more user stories to add sorting, filtering and pagination.

 

Unit and User Story Testing is part of the Definition of Done at the Story level. This effort may be represented as sub-tasks of a User Story in JIRA.For the testing that cannot be automated or included in the unit test, the business analyst and/or tester should create a manual test case that can be executed within the Sprint. As time permits in the future, those manual User Story test cases can be automated as functional scripts.

 

Feature Functional Testing

 

To meet the Definition of Done at the Feature level, we have to ensure that all the Stories collectively are passing for the Feature from End-to-End (e.g. can include performance, security, etc.). So, we would need Feature level test cases that tests across all of the Stories and non-functional requirements that makes up the Feature. This can a roll up of previous User Story test cases, automated functional scripts, unit tests, with additional Feature level test cases to be written by the BA or tester. Again, as time permits in the future, those manual feature test cases can be automated as functional scripts.

 

A part from the Dev Team testing the Feature from end-to-end, it is common that there is an additional level of testing by the Business or Product Owner. This is known as User Acceptance Testing (UAT). UAT involves validating software in a real setting by the intended audience. The aim is not so much to check the defined requirements but to ensure that the software satisfies the customer’s needs.

 

RegressionTesting also needs to be done to ensure that other related Features and code was not broken nor the behaviour was modified. Regression testing can be executed by re-using the already developed automated unit tests and scripts, and then identifying what new scripts and test cases need to be written.

 

For Feature testing executed by the Dev Team, a Technical Story would be created and planned for in a Sprint.