An essential aspect of market research involves randomizing your survey to ensure that any bias is eliminated across survey participants, thereby improving the data quality of your survey results. One way you can randomize elements of your survey is by choosing the display order for a given question’s answer choices.

In CheckMarket, there are multiple ways to configure the order in which a question’s answer choices are displayed, from simpler use cases to advanced randomization.

To configure the display order for your answer choices, select the Question tab in the survey editor and make a selection from the Response Order dropdown. See below for descriptions of each available option.

When you select Default order, the answer options will be displayed in the order in which they were set up in the question editor. Each respondent will see the same order of answer choices.

When you select Alphabetical order, the answer options will be displayed in alphabetical order, from A to Z. Each respondent will see the same order of answer choices.

When you select Randomly flip the order of answer options, half of respondents (on average) will see the answer choice order flipped/reversed. For example, the first respondent may see the choices listed as red, orange, then yellow while the second respondent would see the choices listed as yellow, orange, then red. Flipping the order of answer options for each respondent helps to eliminate bias when designing research studies.

When you select Random order, the answer options will be displayed in a random order for each survey participant. In theory, each respondent will see a unique order of responses and no two respondents would see the choices listed in the same order. Randomizing the order of answer options for each respondent helps to eliminate bias when designing research studies.

When you select Display random subset, only a specified number of the available answer choices will be randomly chosen and shown to respondents. You can specify how many responses you want to randomly display by selecting a number from the secondary dropdown that appears. By only displaying a select number of answer choices (rather than displaying all available choices), respondents are required to compare a fewer number of options before making their selections, which helps to eliminate respondent fatigue.

Researchers are also able to choose one or more answer options to always display from the randomly selected subset. This is particularly useful when conducting a competitive analysis, where you always want your brand to be shown to respondents, but you only want a few of your competitors to be displayed as the other answer options.

If you want to ensure that each answer option is displayed approximately the same number of times across all respondents, select the Balance choices checkbox. If the Balance choices checkbox is left unselected, the subset of answer choices to display will be selected using true randomization.

When you select Advanced Answer Choice Randomization, you are able to create groupings of answer choices. For each group you create, you are able to set a specific Response Order that applies only to that group. 

Use the checkboxes next to each answer choice to select all answer choices that you want to group together, then click Create group. Repeat this step to create as many groups as you need.

Once you have created a group, use the Response Order dropdown within the group to select the order in which you want the answer choices within that group to display.

Optional: Once you create a group, you can click the pencil icon next to the Group Number to rename the group. This is for your internal reference only; respondents will not be able to see that answer choices have been grouped, nor will they be able to see the group’s name.

To add a new answer choice to a group, click Add new answer choice. To move an existing answer choice into a group, select Add to Group and then choose which answer choices (whether they already exist within a different group or are ungrouped) you want to move into your current group.

When you select Advanced Answer Choice Randomization (thereby enabling you to create groups), the first (root) level of items (e.g., an ungrouped answer choice or an overall group of answer choices) will be displayed in random order. Any answer choices within a group will be kept together and displayed consecutively (yet could be shuffled around within the group depending on the display order that was selected for that group).

Q1 has the following structure with Advanced Answer Choice Randomization selected:

• • A1: red
  • A2: orange
  • A3: yellow
    Group 1: “Cool Colors” has ‘default’ selected as the display type for the group
    • A4: blue
    • A5: green
    • A6: purple

We will always display A4, A5, and A6 consecutively and in the same order (since the default display order is selected for the group). However, this grouping will be randomized among the other answer choices. So one respondent could see the following:

• A1: red
• A4: blue
• A5: green
• A6: purple
• A2: orange
• A3: yellow

And another respondent could see the following:

• A4: blue
• A5: green
• A6: purple
• A3: yellow
• A1: red
• A2: orange

Q5 has the following structure with Advanced Answer Choice Randomization selected: 

Group1 has Display random subset selected as the display order and 2 selected as the number of answer choices to randomly select and display.

• A1: Value for money
• A2: Performance
• A3: Warranty service
• A4: Broad selection of products
• A5: Quality

From Group 1, only two of the three answer choices will be randomly selected and displayed to each respondent. They will always be displayed next to each other in the list of answer choices. Answer choices 4 and 5 will always be displayed. The contents within Group 1, Answer Choice 4, and Answer Choice 5 will be displayed in random order. 

So one respondent could see the order as follows:

• A5: Quality
• A3: Warranty service
• A2: Performance
• A4: Broad selection of products

And another respondent could see the following:

• A2: Performance
• A1: Value for money
• A4: Broad selection of products
• A5: Quality

As you can see, any of the contents that are displayed from Group 1 will always be shown next to each other and the entire grouping will be shuffled among the rest of the answer choices (or if applicable, the rest of the groups). 

If you want to anchor the position of a group or an answer choice so that it is not randomized among the rest of the answer choices/groups, select the pin icon to the right of the group or answer choice to always display that answer choice/group in that position while randomizing the order of the un-pinned answer choices/groups.

In the following example, the answer choice Red and the group Cool Colors have been pinned, or anchored, in place. This means that the answer choice Red will always display in the first position for each respondent and the answer choice within the Cool Colors group will always be fixed to display in the 4th, 5th, and 6th positions. The answer options Orange and Yellow will be randomly shuffled but will each either display in the 2nd position or the 3rd position.