Stakeholder interviews, part one: Obtaining robust one-on-one feedback

Note: I’m Working Out Loud to document my learning journey around developing a digital learning course for the Master’s in Learning & Organizational Change program at Northwestern University. 

The first iteration of the digital portfolio course is built in the Canvas learning management system. This design is based on research, project KPIs, answers to key initial questions and a first round of stakeholder interviews.

We met with a group of MSLOC instructors, one-on-one, to collect initial feedback. Instances of overlapping feedback was integrated into the course design. This week, we are meeting with seven MSLOC students, also one-on-one, to document their reactions, thoughts and feelings around navigating the course.

It is important that the designer document as much real-time feedback as possible. Most participants are not accustomed to providing a stream of consciousness around what they are feeling and thinking when navigating a process, especially if it’s the first time they’ve seen the UX.

Here is guidance that we use to draw out as much robust response as possible.

To the focus group participant before the review is started:

  • As we go through this site, please think out loud
  • There is no observation that is too small to mention
  • Tell me what you’re trying to do
  • Tell me how you think you can do it.
  • If you get confused or don’t think you can understand me, please tell me
  • If you see something you like, please tell me

As the participant navigates the course, avoid asking “yes” or “no” questions and ask open-ended questions:

  • How would you decide to navigate X?
  • What is this [FEATURE] for?
  • What do you expect [FEATURE] to do? Why?
  • What goes through your mind as you look at [FEATURE]? Why
  • What are you looking for? Why
  • What would you do next? Why
  • Does the general navigation and flow make sense? Why or why not?
  • Describe, overall, what’s happening in the modules? (The interviewer is looking to determine if the modules are the correct size, if any parts seem overwhelming, etc.)
  • Does the quiz hit the mark as more of a checklist?

It’s important to take notes verbatim. There can be nuggets of insight used in particular phrasing that will be valuable when reviewing the notes.

This process is all about removing the friction from the process for the participant. Where are the points that feel overwhelming to someone who is not used to working with content management systems, not used to consistently blogging or not used to getting feedback about their writing?

We want to remove unnecessary obstacles that would keep the student from learning how to develop a WordPress site.