Three days before WordCamp San Francisco, during the weekly WordPress Dev Chat, I came up with and proposed the idea that we should use the conference as an opportunity to do user testing. It was a last minute idea, but it’s one that I found valuable and that I would encourage others to do as well. In fact, I think it’s something more WordCamps should try.

The Preparation

Before the event, I setup a test site with the current version of WordPress Trunk along with the feature plugins that we wanted to test. For WCSF, this was Focus, WP Session Manager, and Improved Author Dropdown. I set these up on a public url so that users could do the test on there own machines. I think it is incredibly valuable to test people in as close to there natural environment as possible. While sitting at a conference isn’t where most people blog, if they normally use linux and you only have windows, using there windows laptop with there browser settings will help you understand there problems.

Recruiting Participants

We didn’t do a great job of recruiting participants in large part since we put this together at the last minute. I tweeted about the user tests, put up a sign and John Blackburn asked the volunteers at the happiness bar to send people over. Due to us doing this primarily during sessions, we ended up with people who self selected out of the sessions.

The Tasks

Another key preparation point is deciding what you will be testing. I like to have a simple script of what I say to each participant. I started by asking participants there names, and if I didn’t know them, a tiny bit about them. I also got there email address so that I could create an account for them. This helped to frame the test and also put there experience in context. Once they were logged in to WordPress, I asked each participant to create a new post. As the big feature I was looking for the reaction to Focus. I wanted to see how they reacted to the change when you enter into the editor. I wrote down the first reaction that users had. I then had them change the author. This part of the test was two fold. I wanted to see how they transitioned out of focus, while also seeing if found the Author drop down more usable. Most users used the select2 based drop down in the same way that they use the normal drop down.

After that, we asked users “If you wanted to get an idea of all the places you were logged into WordPress, where would you look?”.   This was to test the new session manager UI.

The Outcome

Now that 4.1 has been released, we can look at the results and see how they guided us.  One thing that we learned was that we needed to focus on discoverability of the new distraction-free writing.  This helped lead to us adding a feature pointer.   Users not immediately seeing a benefit in the list of user sessions helped us recognize that we might want to scale the feature back.  In the end, we decided a simple log out of all other sessions button would be the best option.