Ive completed a lot of play-testing lately for my game Carnival. Its a fascinating experience to watch people work through the puzzles. And its a humbling experience as people stumble and flounder on failings in my designs. Without a doubt, focused user testing has made my game significantly. Here Id like to write about the principal things Im watching for.

A friend of mine spurred this article, as she discovered I keep extensive notes during the play-test. She also tested the game and was curious to know if I wrote anything bad about her. I assured her it was all good stuff, while silently discarding the evidence behind my back.

Play-testing Process

Carnival is played online in the browser with teams that meet in an audio/video chat. While the faces can be helpful for testing, its mainly the audio Im listening to. For the video, I have one participant share their screen. Itd be great if I could have all participants share their screens, but Id probably need more monitors to make sense of it.

Hearing the people play is super useful, in addition to watching what they do on screen. Where they move their mouse can be telling, but hearing their hmms and huhs, exasperated laments, and cries of fowl, tell even more. All of this helps me understand their thought process in solving puzzles and easing the play for future players.

In the early rounds of play-testing, I instruct people about what to expect and how the game works. This allows me to test prior to the game being finished, avoiding some redundant work. I try to reduce the verbal preamble quickly as possible for subsequent teams, replacing it with the in-games systems. By the late stages of testing, I start with only a few hellos, then send the teams off on their own.

I dont record these sessions. First off, I dont care to deal with the privacy aspect of archiving such data. But second, Id never watch them again. Its almost always better to test with a new group of people than labour endlessly on a single session. More people equals better testing, especially with a puzzle oriented game.

Cryptic Notes

As mentioned, I take a lot of notes for these sessions, where lots means 2-3 pages of cryptic scrawling for the roughly 90 minute test sessions. A symbol showing the type of comment prefixes each line . At least in theory, in practice, I have some lines that have nothing before them. If something occurs to me during the test, I note it down, even if Im unsure itll be helpful.

For clarity here, Ill replace my symbols with emojis. That way we can debate their semantic importance, rather than dissect my post-modern scribbling approach to art.

Red Herring

This became the most important symbol in testing. Red herrings a problem in puzzle games. These are things, an item, a graphic, some dialog, a pattern, virtually anything that misdirects the player. The player already has to resolve many important pieces of information in their head.

Im talking about unintentional red herrings. These are colour patterns, or objects, that reasonable look like clues to a puzzle. They arise naturally out of the graphic design by accident. I have nothing but ire for designers who intentionally add red herrings in their game. Its an undebatable poor design that frustrates players. There are enough unintended red herrings to deal with already.

For example, I have a puzzle in Carnival where you need to set a row of lights to the correct colours. I intend them to match another pattern on the screen, as hinted in some dialog. Lo-and-behold, some testers found another sequence of colours, in some flags, that seemed to match the lights as well. They then proceeded to match that pattern, which of course failed. This gets a big mark in the notes.

sequence of flags matches light pattern

Obvious thing to change

Many small, sometimes large, obvious ideas crop up during the tests. These may be ways to improve the puzzles, graphics improvements, overall UX ideas, or basically anything. The means I had a concrete thought and I should go back and improve it later at which point I add a to it.

Each circle is a clear opportunity to improve the game. Rather than theorizing about things to improve, these all come from actual players. Fixing them would have a direct impact on some future player. This doesnt mean they all get resolved, as priorities still play a role, and some of them are hard to fix.

One example I have is with a ticket in the game. The player acquires the ticket and must present it to get into the carnival. I thought itd be obvious that you show the ticket to the man in the booth, but one player tried to use the ticket on the entry sign. Its not ridiculous, since the sign, with a No Entry label hanging on top, is where you enter the park. What happens then is frustration. In their head the player has resolved the situation: they found a ticket and are using it to get inside. It not working is like throwing a wrench into their thought process.

The obvious resolution was to make the sign accept the ticket, and thats the note I made:

tried show ticket to sign, accept ticket

Though, I ended up using a more humourous fix. The booth agent cracks a joke about an inanimate sign. This has the effect of affirming the players action made sense, but was slightly off. Additionally, it made it clear where the ticket should be used instead. Oh, I could talk at length about this leading

Puzzle Solved

Whenever a player solves a puzzle, I note it with a little flag, and the time.

rabbit hand puppet, 23:03

This is the wall time, since its what I have most available. I record the start of the session as well, letting me calculate later the time offsets between puzzles.

My goal here is to establish a good pacing for the game. Droughts, long periods between solutions, demotivates the player, and increases the likelihood they dislike the game. Given my experience with the previous game, and numerous escape rooms Ive played, the pacing mostly worked as is.

However, there were some clear problems on some puzzles they took too long. In most cases, I didnt need to know the time to see that people were frustrated. But sometimes the time helped decide the frustration was okay. Perhaps the player was being impatient, rather than having an actual problem.

Timing can also deceive. In one game the players took nearly 15 minutes for one puzzle, well beyond the typical 5 I consider a maximum these arent like fundamentally fixed numbers, so dont quote me on them. But after the puzzle they both said wow, that was great or something to that effect, I cant always read my writing. This makes evaluation tricky, but at least the timed notes give some help.

Hint requested

Everybody gets stuck, but an experienced team should not need hints. Its why I treat hint requests during play-testing seriously. The game has a built-in progressive hint system, so all I do is note where they requested the hint, and how many they needed.

dart game

You might think requesting a lot of hints is bad, but for experienced teams, if they need one, they typically need many. This results from the hints being progressive; the players often already know the information in the first couple of hints, and only the later one reveals something new. Short of deciphering the players thoughts with electrodes plugged into their brain, theres no real way to avoid this.

Well, there is a way to avoid some hints, and I do that. Some hints can be tied to game actions, in particular with the inventory. But the general purpose hints cant be tied to in-game events. I can never really be certain the player already knows the hint.

Requesting hints is part of puzzle games. Its totally fine if players ask for them, but it shouldnt be the standard approach to solving a puzzle. And since people who play-test are mainly interested in this genre of game, it biases the results I assume the average player will require more. Its a good sign when some play tests have no 's on them. Past that point, I can consider each tension point more carefully.

What they said or did

To give context to my notes, and the players thought process, I take several notes about what they say, or what they do. The latter also uses quotes, because I couldnt think quickly what other symbol made sense, and in practice the notes are mixed.

Is this random? click on light

These notes primarily serve as anchors to the other ones

Ill also make notes of solutions theyve tried that have failed. These can often give ideas of how they are thinking, or, sometimes, I end up accepting alternate solutions if they seem equally valid. For some of these Ill end up using a CIRCLE- combination in the notes.

12345, hmm, doesnt work

Something is broken

I put this last since its not the point of this level of play-testing. I have already resolved most of the functional defects, and the game is fully playable before I begin play-testing. Several minor defects still appear, and if Ive recently changed something, engine defects are possible (the frightening and thankfully rare ).

B-girl font not converted path

The is for things that are 100% definitely technical defects. This could be a graphic that is missing, the wrong font used somewhere, or a typo in the text. I suppose I could use them for defects in the puzzles, though oddly, Ive not have that situation come up yet. As this is a multiplayer game played over flaky networks, Ive tried hard, from the start, to make the logical game state consistent. That appears to work. While defects are possible in the puzzles still, Ive likely worked them out prior to starting play-testing.

I think thats an important point: Ive played the game entirely many times before I do any play-testing. I want play-testing to focus on the things I cant find myself. Even the initial play-testers get a game that is working, albeit potentially without the hint system, and not all graphics in their final form, but functionally playable from beginning to end.

Scribbles

These symbols are a general guide to the notes I take. I still have other things written, sometimes in combination, or sometimes with no markings.

I have found though that trying to itemize my thoughts produces firm notes. Rather than write everything, and anything, I focus on specific action items:

- Something is misleading
- something is broken
- Point for improvement
- a puzzle may have a problem

Where and are then used to anchor those points, giving context when I go back later.

While not a perfectly defined process, thats about how I did play-testing of Carnival. Watching people play, think, and laugh is fascinating. I also love watching the streams of people playing my game, as it gives another insight. I take notes from those as well.