Testing Higher-Order Cognition: MCQs Versus MEQs

The popular perception of multiple-choice questions (MCQs) is that they can only test recall. They are also more often than not used for this purpose. For more “serious” testing—that is, for testing analytical ability and the ability to apply learned material—educators often look to the modified essay question (MEQ). This seems to be true across all testing scenarios from academia to business environments.

Issues with MEQs

The MEQ as a testing tool, assuming that it is properly implemented, poses important problems in the grading process and also for test-takers. The more obvious problem areas include:

• Grading takes up more time and resources overall
• Grading involves subjective judgments. Examples: (a) Not all graders will agree on the points to be awarded for a certain answer (b) The structure of the answer can have an effect on interpretation by the grader
• For the test-taker, time management during the test is difficult. (If he wants to come back to a certain question later, he remains in doubt as to begin the answer or skip it for the time being.)
• Uncertainty/confusion can arise for the test-taker, on account of the space for the answer being fixed. (Examples: (a) If the provided space seems too short, he needs to condense his answer. It might therefore not accurately reflect his knowledge. (b) The correct answer according to the test-taker might consist of only one sentence, in which case he is in doubt about whether he needs to supply more information as part of the answer.)

Despite these problems, the MEQ (as opposed to the MCQ) is consistently employed across testing scenarios because MCQs are deemed inadequate in testing higher-order aspects. But this, in turn, arises primarily because it is more difficult to formulate a good MCQ.

By “higher-order,” we mean the three higher levels (levels four, five and six) of Bloom’s Taxonomy: Analysis, Synthesis, and Evaluation. Our argument here is that MCQs can indeed be used for testing higher-order cognition. The corresponding benefits are less confusion and better time management for the test-taker, and, easier grading along with time savings.

MEQ Question Formulation

It is relatively difficult to formulate a good MCQ to test at, for example, the Analysis level of Bloom’s Taxonomy. To the question “Which of the following,” the natural answer choices that follow are facts or distinct possibilities. If we want the test-taker to perform an analysis, the natural expectation is that he should frame his response in words.

A 2007 study at the University of Adelaide, Australia (citation) points to the converse problem—that of improper formulation of MEQs. The study involved an analysis of (MCQs) and (MEQs) used for summative assessment in an undergraduate curriculum over two years. The effectiveness of the question was defined by the questions’ “ability to measure higher cognitive skills,” and their quality by “the presence of item writing flaws.”

A key observation was that half of all the MEQs tested only factual recall. Conversely, the MCQs in many cases did test higher-level aspects.

From the paper: “The analysis has shown that it is possible to produce an MCQ paper that tests a broad spectrum of a curriculum, measures a range of cognitive skills and does so, on the basis of structurally sound questions.” And: “The criticisms levelled at MCQs are more a judgement of poor construction … and the present study suggests that a similar criticism should be levelled at MEQs.”

Conditions and requisites being ideal, an MEQ can test more easily for higher-level cognition. However, in the real world, exams may not go through rigorous review before being published, and not all authors may be properly trained in examination design. These imply that

• The relative merits of the two types need to be considered.
• The notion that “MCQs cannot test more than recall” and “MEQs can always test aspects beyond recall” should be dispelled, which is hinted at by the study cited above.
• The way questions are constructed is at least as important as the question type itself.

MCQ Question Formulation

How can MCQs be formulated to test for Analysis, Synthesis, Evaluation? We present a quick summary, with elements from an earlier whitepaper and ideas from Designing and Managing Multiple Choice Questions. The latter handbook was written by John Carneson, Georges Delpierre, and Ken Masters.

Testing for Level 4 (Analysis)

Approach 1: The MCQ can check whether the student understands the matter sufficiently well to be able to analyse it. The examiner frames a sentence that represents one possible result of the analysis; this is presented as the correct answer (along with plausible distractors). The examinee’s actual analysis of the material is not recorded (as in an MEQ), but unless the examinee performs the analysis correctly, he will be unable to pick the correct answer.

Example question for Approach 1:

Read the following paragraph about Bernoulli’s Principle of non-sufficient reason, then choose the correct analysis from the options that follow.

If there is no known reason for predicating, of our subject, one rather than another of several alternatives, then relatively to such knowledge, the assertions of each of these alternatives have an equal probability.

(A) If we have a reason for supposing that alternative X is likely, we must assign X a high probability.

(B) If we only suspect that alternative X is a likely outcome, the other outcomes must be given equal probabilities.

(C) If we have no known reason for supposing that alternative X is more likely, we cannot consider X a viable alternative.

(D) All of the above

Approach 2: In contrast to the above, where the answer options were the result of analysis, the examiner can construct answer choices that need to be analysed. The examinee will be able to point out the correct answer choice only if he has properly analysed all the choices.

Example question for Approach 2:

Read the following statements about basic probability and choose the option that is logically correct.

(A) If event Q can happen only after event P, the probability of Q is unknown until after P occurs.

(B) If the probability of events P and Q are known, the degree of their interdependency can also be known.

(C) If event Q can happen only after event P, the probability of Q depends on the probability of P.

(D) If the degree of their interdependency between events P and Q is known, their respective probabilities can also be known.

Approach 3: Similar to approach 1 (above), the object of analysis is the presented sentence or paragraph, not the answer choices. The examinee is asked to choose from proposed changes to the paragraph. Making the correct choice implies an analysis of the paragraph in the question.

Example question for Approach 3:

Read the following factually incorrect sentence about probability, then decide which of the four answer options rectify the sentence.

Suppose there is a reason for predicating, for an event, one particular succeeding event(s) out of ten possibilities. Then, based on that reason, the probabilities of all the succeeding events are equal.

(A) “Based on” should be replaced by “independent of”

(B) “One” should be replaced by “two”

(C) “Equal” should be replaced by “unequal”

(D) “A reason” should be replaced by “negative reason”

Testing for Level 5 (Synthesis)

“Synthesis” implies creative aspects such as: reorganization of learnt material into a new construct; practical application of learnt concepts; formulation of a new idea or plan. By definition, this cannot be directly tested by MCQs. However, the examinee’s ability to perform at the synthesis level, relative to other examinees, can be predicted by a properly-constructed MCQ.

Here are two examples of how such MCQs can be constructed:

1. Suppose the examinee has read several literary works—short stories, essays, or books—with nothing obviously in common. An MCQ can ask about which of these have parallels between them. (Within world literature, there are parallels between Dante’s Inferno and Goethe’s Faust, and also between Faust and Ibsen’s Peer Gynt. On the other hand, there are few or no significant parallels between King Lear and Peer Gynt.) In answering an MCQ that asks about parallels between literature, it is not merely analysis that will be tested—though that is essential—but also Synthesis.

2. Suppose the examinee is familiar with a taxonomy within a subject area (types of object classified into a structure). Drawing on his familiarity with the subject, he might be able to devise a different taxonomy. The MCQ could present plausible as well as implausible taxonomies as answer choices. Getting the answer right implies some Synthesis on the part of the examinee.

3. Examinees may be tested on systems modelling by a set of MCQs. While the examinee cannot present his model scenario, a model can be presented to him, and MCQs can ask questions about the behaviour of the model under different circumstances. Suppose the goal is for the examinee to design an electronic circuit, one that will control five light bulbs in a certain way. He can be asked questions about the model, such as: “If a sixth light were added, you change the circuit by…” “The circuit would malfunction if…” To answer these questions would require that the examinee form a mental model.

Testing for Level 6 (Evaluation)

“Evaluation” implies not only comprehending (and possibly analysing) an entity, but also having a sufficient grasp on it to make critical statements or value judgments upon it. Here are three approaches for constructing MCQs to test whether an examinee is competent at the Evaluation level.

Approach 1: The examinee is required to Evaluate, or critically appraise, a piece of text. The question is phrased as an assertion, and the answer choices are phrased as reasons. Choosing the correct answer implies having arrived at a correct assertion/reason pair for the text.

Example question for Approach 1:

Being familiar with Schopenhauer’s On the Wisdom of Life: Aphorisms, how would you defend the view that Schopenhauer is not a “philosopher of pessimism”?

(A) He never explicitly espouses a pessimistic view

(B) His looks at things as they are, thus going beyond pessimism

(C) He encourages a strictly neutral outlook, precluding pessimism

(D) He often elaborates on positive aspects such as harmony

The correct answer is B. Note that a mere analysis of the matter would not lead the examinee to the correct answer; a full evaluation is demanded.

Approach 2: Similar to approach 1, the examinee is required to Evaluate a piece of text. The answer choices are phrased as assertion/reason pairs. Being able to choose the correct assertion/reason would mean performing an Evaluation of the text.

Example question for Approach 2:

Being familiar with Schopenhauer’s On the Wisdom of Life: Aphorisms, which of the following pairs of assertion/reason best describes the fragment?

(A) It is pessimistic because it suggests the impossibility of achievement

(B) It is positive because it recommends action towards progress

(C) It is pessimistic because it is critical of our positive aspects

(D) It is positive because it sees “both sides of the coin”

The correct answer is (C). As in Approach 1 (above), a mere analysis of the matter would not lead the examinee to the correct answer. Such a question is more difficult to construct than that in Approach 1, and the distractors are correspondingly more plausible—meaning higher score discrimination.

Approach 3: An interesting possibility is for the examiner to construct a partial evaluation of the subject matter, and ask the examinee to rate that evaluation. For our example, we use the same literary work (Schopenhauer’s On the Wisdom of Life: Aphorisms) as above.

Example question for Approach 3:

Being familiar with Schopenhauer’s On the Wisdom of Life: Aphorisms, read the following appraisal of the work, then rate the appraisal by choosing from the options that follow.

“Aphorisms deals primarily with our illusions and our lack of real understanding. It is universally critical, but offers little constructive advice—even though its unstated claim is to edify.”

(A) The appraisal is correct but lacking in scope

(B) The appraisal is incorrect and lacking in scope

(C) The appraisal is complete in scope but only partially correct

(D) The appraisal is incorrect but the scope is complete

Approach 3 requires even more time from the examiner than approach 2, but it affords the highest score discrimination of the three we have described here.

Issues With The MCQ Format

While MCQs can be used in testing higher cognitive skills, there are some intrinsic difficulties (as opposed to the use of MEQs). Some are listed below:

• Problems associated with subjectivity are reduced during the grading process, but conversely, examiner subjectivity in the creation of the questions can be a concern.
• If a set of questions is to be validated by peer review, more time would be required in the case of MCQs than in the case of MEQs, simply because the questions are more complex.
• In the context of higher-level testing, students may perceive the MCQ format as restrictive and authoritarian.
• The higher the cognitive level, the greater the negative impact of guessing. Possible solutions include negative marking, the “Don’t Know” or “Skip Question” options, and weighted-answer schemes.
• There is a greater demand on the examiner in terms of precise wording of questions, which can prompt the need for peer review.