Usability Testing with Belly Merchants

At Belly, we have a few different ways of getting feedback from our Belly Merchants. These methods include surveys, interviews, and quantitative A/B tests. We also have the luxury of being able to physically visit some of our Merchants in-person at their business location and see how they actually interact with Belly’s digital products.

What is a Usability Field Test

A usability field test is observing how people in their natural environment use a product. By watching, asking questions, and having users perform tasks based on real-life scenarios, usability field tests help us determine what parts of our designs are working well and what aspects need improvement. In addition, by conducting the tests where our Merchants physically interact with Belly allows us to see how their natural surroundings (their business location) impacts their experience.

Usability testing helps us confirm our design assumptions and allows us to uncover hurdles getting in the way of the users, it also lets us ask, and interpret, “why.” Why did a user click there? Why did they think that would happen? Why didn’t they use the interface as we thought they might? Typically, these types of insight are difficult or impossible to synthesis from other methods of testing. To see someone struggle with your product is hard but very effective in motivating you to improve. Seeing is definitely believing.

Who We Test With

Belly Merchants are the individuals that own or work at a business that offers Belly as their loyalty program.

The Belly office is located in Chicago, so that’s where we do our field testing. We typically select nearby Merchants that would be a good fit for a particular test we want to conduct. For example, if we want to test a new email campaign feature we might want to target Merchants who have already sent a campaign with Belly. Or perhaps we might find it more useful to test with Merchants who have never sent an email campaign. It all depends on the predetermined goal of the test and how we plan to measure the success.

After filtering down our local Belly Merchant list by location and user behavior we either call, email, or stop in their business and ask if they would like to participate in a usability study that will help us improve our product offering. Most Merchants are happy to help, especially since we incentivize our participants with Amazon gift cards for their time and feedback. It’s helpful to schedule multiple tests (at the very least 3 Merchants) in a single day or week. That way the collecting, organizing and analyzing of the tests can happen quickly while everything is still fresh in our minds.

What We Test

Usability field studies are great for testing and refining general ease of use, user flows and tasks, as well as uncovering confusion or user created work-arounds.

A great place to start with usability testing is critical paths and features, like sign-up funnels or check-out paths.

It’s crucial to get user feedback on product decisions early on in the design stages. We don’t want to put too many hours and resources towards an idea that isn’t effectively solving a problem. For a new feature, we want to ensure we are headed in the right direction before we dive into code or platform changes. Typically, we test what we are currently designing. This could be sketches, mock-ups, prototypes, or anything we might need user input on. We want to know we’re actually solving a problem that exists for our users.

How We Test in the Field

Our testing method is largely inspired by Steve Krug’s excellent guide book, Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems. Following this book’s outline helps us effectively uncovering the biggest usability problems with our products quickly and easily.

Our basic process is as follows:

1. Introductions & Set Up

Once we arrive and get settled in a relatively private area of the Merchant’s business, we quickly review what it is we are trying to accomplish that day with the test participant. We make it clear we’re not testing the user’s technical capabilities, but simply looking for feedback on our designs and how our product is working. It’s important to communicate this to participants to make them feel at ease and more comfortable with being honest.

We also have the Merchant sign a recording consent form that lets them know we plan on recording their screen activity and audio during the session to review later on and share with our team.

Once the consent has been signed, we fire up the laptop (or iPad), and set up the test. We use Quicktime to record the screen and audio.

2. Scenarios & Tasks

The bulk of the test is made up of three individual tasks we ask the Merchant to perform with our provided test materials. Each task is paired with a fictional but plausible scenario that aligns with our Merchant’s usage patterns and our test goal. As the Merchant performs the outlined task, the Belly employee who is moderating will ask the test subject to talk out loud and explain what they are thinking and why. The goal is to understand the thought process behind the actions they take on the screen. Another Belly team member is present to take notes while the test subject and moderator interact.

3. Wrap Up

Upon completion of the three tasks, the Merchant is compensated and the test is finished. After each session it’s important for us to ask the Merchant if they have any questions or additional feedback they would like to provide outside of what we have tested that session.

For a more detailed, step-by-step guide to this process, check out Rocket Surgery Made Easy.

4. Next Steps

Once we’ve completed a test with a single Merchant, we want to run that identical test with 2 to 4 other Merchants. Doing this will help us uncover the biggest usability issues quickly and allow us to move from the testing phase to redesigning.

After all of the Merchants have been tested, recordings and notes are reviewed and a prioritized list of potential improvements is created.

Closing Thoughts

Testing is, without question, an integral step in the design process. It’s critical to get feedback from people who use your products in order to improve your current solutions, or at the very least to confirm they are working. It is nice to think we could test every decision, design, and iteration, but that’s not realistic or, frankly, necessary. Rather, it is more important to focus on improving the largest, most impactful problems within your products and iterate to a better outcome.

Usability testing is often thought of a large, expensive, and time consuming production, but it doesn’t have to be. A little bit of planning, and a few steps, is all you need to start improving your digital products.