This three-part white paper explores some possibilities and ideas in creating multiple choice questions (MCQs). In Part I, we look at:

• Framing questions with the correct focus
• Best practices in MCQ creation
• Making distracters plausible

Section I: Focus

“Focus” here refers to what exactly it is that the instructor wants to test. It often happens that due to certain temptations, or because of not properly thinking out an answer choice, or a variety of other factors, one ends up not testing not the knowledge required of the learner, and testing something else instead.

Focus Technique 1: Avoid Extreme Adjectives

Extreme adjectives, such as “always” and “never”, mean that a higher degree of confidence is expected of the learner. Learners might be wary of choosing such answers because of the “amount of confidence” involved. Here are some answer choices for the question, “Can the Vice President become President?” Which choice would be selected most often (assuming most learners are not sure of the correct answer)?

Question: “In the USA, can the Vice President become President?”

Answer Options:

1. Always
2. Never
3. Sometimes
4. Only if the president likes him

Here, the fourth option would probably be dismissed because it is humorous. The choice “Sometimes” would be selected more often than one might expect. Why? Because the learner might know that the correct answer is “always.” He might be confident that “never” is an incorrect answer. However, if he is not fully confident, he chooses “Sometimes” just to be on the safe side.

So, use extreme adjectives with care. Only employ them when you want to specifically test for high confidence in some fact. In all other cases, avoid the extreme adjective that first comes to mind.

Next, consider the question “Can viruses can infect bacteria?” What would be the best choice for a correct answer? Assume that all the choices are factually correct. Also, assume that you do not want to check whether the learner is highly confident about the answer.

Question: Can viruses can infect bacteria?

Correct Answer Possibility #1: Since they can affect DNA, they can certainly infect bacteria.

Correct Answer Possibility #2: They almost always do infect a bacterium if there is one in the vicinity.

Correct Answer Possibility #3: It is possible but it doesn’t always happen.

Correct Answer Possibility #4: It is very highly probable that a bacterium close to a mass of viruses has been infected.

In the case of (1), the tone implies that it would be foolish not to select it. The key word here is “certainly.” It is an extreme adjective. Apart from avoiding extreme adjectives, also avoid an extreme tone, as in this example.

In (2), “almost always” is very close to the word “always”; avoid it.

Choice (3) is good. Even though the word “always” figures, the overall implied meaning is loose, not absolute.

In the case of (4), “very highly probable” means “always,” even though the word “probable” by itself is not extreme. Don’t go by the words alone!

Focus Technique 2: Do Not “Fool” The Learner

Do not draw attention to one aspect when actually checking the learner’s knowledge of a different aspect. Here’s an example.

Question: “Which of the following animals is not a mammal?”

Answer Options:

1. Horse
2. Whale
3. Porpoise
4. Person

Here, the technically correct answer is (D), because there is no animal called “person.” But the question does not test for zoological knowledge; it tests for knowledge of terminology. The test-taker might assume that “person” means “human being,” which is a member of the class of mammals. He might choose (C) just because he doesn’t know much about porpoises.

Here’s a similar example. Which of the following might be a good answer set for the question “Which of the following animals is not a mammal?”

Set 1: Rodent, Bird, Primate, Hare

Set 2: Rat, Echidna, Whale, Porpoise

Set 3: Rodent, Rhesus Monkey, Golden Kangaroo, Timber Wolf

Answer set (2) is the best, even though sets (1) and (3) are technically fine. In the case of (1), the learner might be tempted to select “Hare” because it is an animal while the rest are classes of animal. In the case of (3), the learner might be tempted to select “Rodent” only because it is a class of animal while the rest are animals. Here again, getting the answer right becomes a matter of words more than knowledge.

Focus Technique 3: Don’t Test For Unnecessary Knowledge

Testing for unnecessary knowledge is often tempting; perhaps you want to check how well the test-taker has been following the course. In fact, some (lazy) instructors frame fill-in-the-blank questions lifted from the recommended textbook. Many of these questions might test for arbitrary information. This is not good practice.

For example, asking for a precise date in history—as opposed to the year—is not a good idea. True, there are a few important precise historical dates (with year, month, date), but there are many more cases where only the year is important to remember.

As the instructor, you want to check whether the learner knows, at the time of the test, what he should have learnt during your course. You do not really want to test his sharpness, alertness, intelligence, or anything else.

Focus Technique 4: Avoid The Guessing Game

Remember that as the instructor, you are checking your learner’s knowledge, so don’t end up checking whether he can guess well! The example below illustrates the “guessing game” principle.

Question: Which of the following is NOT a county in the UK?

Answer Choices:

1. London
2. Londonshire
3. New York

“London” might confuse the learner thus: “London is a city, but is it a county? It might or might not be.”

“Londonshire” worsens the confusion: the learner might think, “Many UK county names end in ‘shire’, so maybe it’s a county. But if this is the correct answer, then ‘London’ has to be a county.” So what’s happened is:

The learner is grappling between two false options. This is not desired.

The third option is plausible, but it’s also obvious. Some learners might suspect that since this seems obviously the correct answer, it might not be. So there’s another guessing game going on:

The learner imagines that since an answer is obvious, it’s not correct.

Here’s another example:

Question: Which are the proper nouns or proper noun phrases in ‘Humpty Dumpty sat on a wall’?

Answer Choices:

1. “Humpty”
2. “Wall”
3. “Humpty Dumpty”
4. “Humpty” and “Dumpty”

The learner would be confused between (B) and (D) if he does not know what a proper noun is. He would be confused between (A) and (D) because he’s not sure whether they’re two separate words. In addition, option (A) makes the learner get the idea that “Dumpty” isn’t part of the name “Humpty Dumpty.”

It is tempting to make a question hard to guess, for the sake of making it hard to guess instead of testing what the learner knows. Avoid!

Focus Technique 5: Don’t (Inadvertently) Test For Extraneous Knowledge

A correct answer might sometimes be chosen because the learner has knowledge from an area different from that of the course. Suppose you were to ask, in the context of a course on brands:

Question: What is the form of the Blaupunkt logo?

Answer Choices:

1. Triangle
2. Dot
3. Square
4. Pentagon

The question would favour those who know some German (“Punkt” in German means “dot”). It is hard to ensure no mistakes of this sort, but one can be vigilant.

A similar example:

Question: What is the term, in astronomy, for the Great Bear constellation?

Answer Choices:

1. Ursa Major
2. Bovia Largo
3. Cassiopeia
4. Lorem Ipsum

Learners who know a bit of French will recognise that “Ours” means “Bear,” and might therefore realise that option (A) is correct. Be watchful!

A third example, assuming a basic course on music reading:

Question: In musical notation, what is the signature before the notes called?

Answer Choices:

1. Signa
2. Clef
3. Autograph
4. Trough

Those with some prior, more advanced knowledge of music (as opposed to music reading) will know that the piano is also called the Clavier, and can therefore guess “Clef” as the correct answer.

 

Section II: Best Practices In MCQ Creation

What follows is about best practices in MCQ creation. The need is to be able to identify what to avoid in the creation of MCQs, and implement the best practices listed herein.

Best Practice 1: Don’t Directly Lift Phrases

Some examiners frame MCQs using sentences lifted out of the courseware. As a result, those who memorise passages are more likely to get the answer correct than those who do not; worse, this can encourage memorising.

Here’s a sentence from a computer programming book: “Real-world objects share two characteristics: they all have state and behaviour.” It is possible, but probably not desirable, to frame this as a Fill-in-the-blanks MCQ with “state” as one of the choices: “Real-world objects share two characteristics: they all have _____ and behaviour.” Learners who have memorised the material would get this correct.

To rephrase this, try the following.

>> Reword the statement. You could re-phrase the sentence to: “All objects have a certain behaviour. What else identifies them?” The choices for this could be Colour, State, Existence, Position.

>> Add options. If you must use the same sentence, consider making it harder by increasing the number of answers, that is, putting in two (or more) blanks instead of one. So the question could be: Real-world objects share two characteristics: they all have _____ and _____. The answer choices would be word pairs in this case.

>> Create a different question. You could formulate a related question with the same correct answer, “state.” For example: “What is the word for the most fundamental property of an object apart from the way it behaves?” Or, “If a dog is a certain colour, and you consider a dog an object, what is the name for the property that its colour indicates?”

Best Practice 2: Avoid Double Negatives

A double negative is what it sounds like. Example:

Question: Which of the following is NOT true about Australia?

Answer Choice: “The country is not densely populated.”

This answer statement is factually true, so it is an incorrect answer choice. The “not… not” combination can confuse learners. It succeeds in making the learner think more, which is probably not what you want!

Here’s a complex example that illustrates how double negatives might creep into your questions. (For this example, let’s accept the rule of grammar says that adverbs should come before verbs.)

Question: Which of the following is NOT a case of poor grammar?

Answer Choices:

1. When the adverb is placed after the verb, and there are only one of each
2. When the number of the adjective and that of the noun do not agree
3. When, instead of standard speech, slang is used
4. When the appropriate apostrophes are missed

In (B), the answer contains the words “do not agree.” This might seem fine, but it really is a double negative. In (C), “Instead of standard speech” translates to “standard speech is not used,” so there’s a double negative. Finally, in (D), “Missed” implies “not used,” so a double negative exists. Answer choice (A) is the only one without a double negative.

Best Practice 3: Use Correct Phrase Forms

No key words in the main clause should be close in grammatical or structural construction to that of the answer, failing which the answer becomes easy to guess. An example:

Question: Which form of organ cancer can exist without symptoms for years?

Answer Choices:

1. Leukaemia
2. Stomach cancer
3. Halitosis
4. A brain tumour

It’s easy to guess “stomach cancer” as the correct answer because the question says “organ cancer.” Similarly, compare the answer choices for the question below:

Question: Mitosis is much more ____ than meiosis.

Answer Choices:

1. Comparatively elaborate
2. Harder to achieve
3. Dependent on mitochondrial processes
4. Complex

As answer choices, (A) and (B) do not make grammatical sense. By choosing (A), the learner would be saying “much more comparatively elaborate,” which has redundancy in words. Choice (B) is even worse because it would say “more harder.” Choice (C) introduces ambiguity. It might be taken to mean that mitosis is more dependent on mitochondrial processes than meiosis is, or that mitosis is more dependent on mitochondrial processes than on meiosis. Only choice (D) is well-worded.

Best Practice 4: Keep Choice Lengths Equal

All answer choices should be approximately the same length; this is almost a universal rule. If one of the choices is disproportionately long, it will be examined more carefully. It will draw attention. There are two “negative scenarios” here:

>> If the disproportionately long answer is the correct answer, the learner might choose it because he has looked at it more closely, or because it is more detailed.

>> If this disproportionately long answer is NOT the correct answer, you will be wasting the learner’s time and “fooling” him or her, too!

As a typical example:

Question: Why does ice float on water?

Answer Choices:

1. Because it has lower weight
2. Because it freezes at 0 degrees C
3. Because the equivalent mass of the water it displaces is greater, as in the explanation of the triple-point of water
4. Because otherwise it would melt

Answer (C) is the correct one. This is not a well-written question because (C), the correct answer choice, has more detail than the others, and it “seems” correct. It is more likely to be chosen regardless of whether the examinee knows his facts or not.

Choices like (D), which are humorous or meaningless, should be avoided. Some examiners use such answer choices only to “fill up” the list of four choices, but one meaningless choice means only three effective choices! (In turn, this raises the exam score of a test-taker who guesses many of the answers.)

Best Practice 5: Use “All Of The Above” Sparingly

“All of the above” as the last answer option is often not a good idea. Let’s look at two cases separately:

>> Suppose “all of the above” is the correct answer choice, and the test-taker knows that two options of the other three are factually correct. He is likely to select “all of the above” even if he has no clue about the third.

>> Suppose “all of the above” is NOT the correct answer choice, and only one of the three others (say option B) is factually correct. Also suppose that option A is obviously incorrect. The test-taker will dismiss “all of the above,” effectively reducing four options to three.

In the following example, “all of the above” is the correct choice.

Question: What are the properties of hydrogen?

Answer Choices:

1. It combines with oxygen to form water
2. It liquefies at 21 degrees K
3. It is a gas at room temperature
4. All of the above

Here, the learner should know from the course material that (A), (B), and (C) are all factually correct. But even if he knows that (A) and (C) are factually correct, and does not know about (B), he will get the answer correct.

“All of the above” can be a good choice in a few cases, though. For example, if the first three answer choices are factually correct, and one of them is a lesser-known fact, “all of the above” can be a good fourth choice.

Best Practice 6: Use “None Of The Above” Sparingly

“None of the above” as the last answer option is often not a good idea.

Let’s look at two cases separately:

>> Suppose “none of the above” is the correct answer choice, and the test-taker knows that two options of the other three are factually incorrect. He is likely to select “none of the above” even if he has no clue about the third.

>> Suppose “none of the above” is NOT the correct answer choice, meaning that at least one of the three others is factually correct. Also suppose that the test-taker knows option B to be correct while not knowing much about options A and C. He will dismiss “none of the above,” effectively reducing four options to three.

Question: Which of the following is a property of water at atmospheric pressure?

Answer Choices:

1. It is a gas at 99 degrees C
2. It can be a solid at 2 degrees C
3. It combines with hydrogen to form more water
4. None of the above

“None of the above” is the correct answer. Including this option can prevent guessing and demand that the learner be sure of his facts. On the other hand, in some contexts, it can make the question too difficult.

Question: Which of the following is true about France?

Answer Choices:

1. It lies to the north of Spain
2. It is connected to Ireland by the “Chunnel”
3. The Bois de Cézanne lies to the south
4. None of the above

Option (A) is the correct choice. You can imagine an examinee who is not sure about options (B) and (C), but is quite sure about option (A). As a result, he would disregard the “None of the above” option, effectively reducing it to three options.

Best Practice 7: Make Choices Similar

Keeping the choices very similar to each other is an oft-used technique in MCQ creation. To illustrate the principle, let’s use the example: “Which country is the biggest rice exporter in the world?” (Correct answer: Thailand.) You might use the following choices:

Set 1: Thailand, England, Saudi Arabia, Brazil

This is relatively simpler than in Set 2 below, where the options are similar (in terms of geographical proximity):

Set 2: Thailand, Vietnam, Malaysia, China

Similarity of options makes guessing harder.

Best Practice 8: Using “gradation” in answer choices

To lend depth to your MCQs, you can use choices that fit into a sequence (that is, with a gradation). An example from a physics course:

Question: In terms of calculations of mass, what is the difference between Newton’s Laws and Einstein’s principles of Relativity?” (The correct answer is (3).)

Answer Choices:

1. They are vastly different
2. They are different in some cases, similar in some others
3. They are similar in most cases, and vary when boundary conditions are approached
4. They are very similar even in the case of boundary conditions

Gradation allows the instructor to test for precise knowledge, while also discouraging the tendency to guess.

Section III: Making Distracters Plausible

A cardinal rule in good MCQ creation is that distracters must be plausible. We can define “plausible distracters” in many ways—a plausible distracter is one that seems correct; it’s an answer choice that the examinee might be tempted to select; it’s a distracter that the examinee would not immediately rule out.

Using “random,” poorly thought-out distracters, or very easy ones, is tempting, and they all result in wasted chances to test the learner. We can call these “implausible distracters” in general—that is, those that the learner would not consider choosing.

In a typical MCQ, you only have three options in which to “distract” the learner; use all three!

Plausibility Technique 1: Avoid Randomness And Easiness

Research your options properly, and avoid using random or very easy distracters. For the examiner, the challenge sometimes is to analyse whether an answer option is too easy. This will depend on the learner, on the course, and also on the exam.

Consider the question, “Which of the following substances is considered a precious metal?” Suppose you were to provide four options with only one of them correct.

“Gold” as an option would, in most cases, be too easy. Whatever the other three (incorrect) options are—say “Lead,” “Steel,” and “Arsenic”—the test-taker is likely to choose “Gold.” The point here is that while constructing the four options, rule out the easy ones, including (in this case) “Gold” and “Silver.”

“Oxygen,” if included as an option, would be a waste of a distracter. Your learner would most likely know that oxygen isn’t a metal. There are examiners who might construct the four options in this case as Zinc, Platinum, Mercury, and Oxygen—perhaps to make the question easier, or perhaps to make the four options very different from each other. The fact is, assuming an average learner, this question effectively has three options. In contrast to “Oxygen,” “Lead” would be a better fourth option.

Whether an answer option is too easy or not depends on the examinee as well. “Copper” would be a poor option if the examinee has some knowledge of engineering. Such students would know that copper is inexpensive because it is used in household wiring, electrical and construction equipment, and so on.

Plausibility Technique 2: Numerical Association

Think of a question that has four numerical values as choices. If all the distracters are plausible, then the closer a distracter is (numerically) to the correct answer, the more plausible it is. And the more plausible the distracter is, the more likely it is that the learner will choose it over the correct answer.

For example, the value of “g” in physics is 9.81. Would you use 9.82, 9.91, and 9.18 as distracters? All three are highly plausible, but using these does not mean a well-constructed question! You’d use these only if you want to test precision of recall.

The concept here is that plausible distracters do not guarantee a good question. In some cases, they might make the question too hard, or too specific.

Take a look at this question addressed at a beginning physics class: “How many particles exist in the universe?” The correct answer is 10⁷², that is, the digit “1” followed by seventy-two zeroes. Which of the following four is an implausible distracter?

10²⁷;  10⁷⁰;  2⁷²;  10⁷

The first option, 10²⁷, is numerically far from the correct answer, but learners who have memorised “10⁷²” might confuse 72 with 27, so this option is plausible.

The second option, 10⁷⁰, is a plausible distracter. Still, it is numerically very close to the correct answer, so it should be used only if you’re checking for precision.

Option 3 is highly plausible. Many learners might confuse 10⁷² with 2⁷² because “72” stands out more than “2” or “10.” This option discourages those who memorise.

“10⁷” (option 4) is an implausible distracter because it is numerically too far from the correct answer.

Plausibility Technique 3: Verbal Association

Let’s now look at verbally plausible distracters. Creating these can be intuitive or hard depending on the situation. The factors involved are many more than those in numerical plausibility. For example, rhyming words are often good distracters, as are words ending in the same syllable. Students with some knowledge of geography verbally associate “Brisbane” and “Melbourne” with “Australia.” Those with some knowledge of technology associate “CPU” and “Disk” with “Computer,” and so forth.

On the basis of verbal associations such as these, you can decide what would “distract” your learner. Here’s an example question for a non-UK student with some knowledge of UK geography:

Question: Which of the following is a county in the UK? (Correct answer: A)

Answer Options:

1. Cumbria
2. Wessex
3. Bois de Boulogne
4. Cottinghamshire

Option (A) is the correct answer.

“Wessex” (option B) is the most plausible distracter, because Sussex, Middlesex, and Essex actually are counties in the UK, while “Wessex” is not.

“Bois de Boulogne” (option C) is the most implausible of the distracters, because students familiar with English place names would think of this option as a French name.

Option (D) is plausible, too. There is a county in the UK called Nottinghamshire. An examinee who knows that fact might not be tempted by this choice. But others might be tempted by this option because it rhymes with “Nottinghamshire.” Only you, as the examiner, can truly gauge what your learners will try and guess versus what they are sure of!

Plausibility Technique 4: Keeping The Choices Divergent

You might now think that keeping the answers “similar” or “close” would be a better technique than keeping them far apart; however, in some situations, the converse is true. Suppose you need to create a question that tests (theoretical) grasp of a concept. Let’s look at an example where the options, instead of being all of one type (like city names, or numerical values) are different explanations, employing diverse concepts.

Question: In which case is mass a true representation of weight?

Answer Choices:

1. When the objects to be compared are experiencing the same gravitational pull (This is the correct answer. It uses the concept of “gravity.”)
2. When the frame of reference for the objects to be compared varies with time (This choice uses a different concept, “frame of reference.”)
3. When Newton’s First Law holds for all the objects involved (This choice uses the name of a law of physics.)
4. When g = 9.81 (A numerical value is used here.)

There are diverging themes here, and all the answers are plausible. Getting the correct answer is more difficult than just ruling out the distracters one by one. Only someone with an understanding of the principles involved would get the answer correct.

Bear in mind that keeping the answer choices very different from each other is not the same as keeping the themes divergent. The following is an example of where the answer choices are widely different from each other, but the theme is the same—so it’s not really a good question.

Question: What is the value of the gravitational constant g on earth? (Correct answer: B)

1. 3.99
2. 9.81
3. 2000
4. 128

Similarly, in the following geography question, the answer choices are geographically diverse but not economically diverse.

Question: Which of the following countries has the fewest natural resources? (Correct answer: B)

1. Australia
2. Japan
3. Germany
4. Bangladesh

To sum up this principle: Divergent, or diverse, answer choices make it difficult to guess the answer. Such choices can be used to ensure that the learner understands core principles. However, the answer choices just being very different from each other does not serve this purpose.

Plausibility Technique 5: The Simplest Choice As The Correct Answer

This technique is straightforward: set the correct answer as the choice that is perceived as the most obviously correct, and/or the simplest. In other words, make the correct answer seem implausible.

Question: What is produced when an alkali reacts with an acid?

1. More alkali
2. More acid
3. Water
4. Hydrogen sulphate

The correct answer is (C). Those who do not know the correct answer might dismiss (C) as being too simple. That is, the distracters seem more plausible.

Here’s an example from astronomy that demonstrates the same idea.

Question: Which of the following planets or satellites has iron oxide at its core?

1. Deimos
2. Earth
3. Poseidon
4. Jupiter

“Earth” is the correct answer and the simplest choice.

Plausibility Technique 6: A Two/Two Method

The most typical MCQ has four options. You can make the distracters more plausible—and therefore the question more difficult—by grouping the choices into two pairs of two, and in each of the pairs, keeping the answers close to each other.

Question: What is the (rounded) per capita GNP of the United States in dollars, 2005?” (Correct answer: C)

1. 110
2. 2,200
3. 44,000
4. 660,000

In this case, a learner who has a rough idea of the answer would choose the correct answer. He might just know that the figure lies between 10,000 and 100,000. This makes it an easy question.

The following example is a much more difficult question:

Question: What is the (rounded) per capita GNP of the United States in dollars, 2005?” (Correct answer: D)

1. 6,600
2. 66,000
3. 4,400
4. 44,000

In this case, what might happen is:

>> Someone who has memorised the answer might remember the digit “4” as part of the answer. He would have to guess between (C) and (D),

>> Someone who knows that the answer is (say) between 10,000 and 100,000 would need to guess between (B) and (D).

Either way (and perhaps in other ways), the question is more difficult; memorisers are discouraged, and also those who have only a rough idea.

Here’s a non-numerical example that illustrates the same principle.

Question: What is the Latin name for gold?

1. Aureodymium (A fictitious word.)
2. Aurum (The correct answer.)
3. Paleostrum (A fictitious word.)
4. Plumbum (The Latin name for lead.)

>> A memoriser might know that the answer begins with the letters “aur-”. He will be confused between (A) and (B).

>> An examinee who has some idea of the Latin names of elements would know that (A) and (C) are fictitious. But since he has only a partial idea, he would be confused between (B) and (D).

 

 

These general guidelines for creating multiple-choice questions will be continued in Part II, Make The Right Choice, which addresses how to test for specific types of knowledge.

© Focalworks 2008

 

In the tips and ideas above, did you find something particularly useful? Is there a technique you disagree with? Let us know!