I have been growing more interested in Multiple-choice Questions or (MCQs). It started off as a QuickKey interest but it has become much more than that. If I thought I might be a edugeek, this pretty much confirms it. An unhealthy interest in assessment and MCQs is unlikely to make me a highly sought after dinner guest.
Reassuringly, I am not the only MCQ enthusiast. Exams watchdog Ofqual suggested that A levels could move away from their “exclusive focus on reading and interpreting traditional forms of text” and introduce more “multiple choice questions.” Shock horror. What is more there are over eight million hits on “how to write effective multiple choice questions,” and just shy of seven million hits focusing on MCQs purely on “higher order skills.” Unsurprisingly in this hyper-connected edu-landscape we work in, MCQs gets a lot of attention from online educationalists and assessment gurus.
Right, Multiple-choice Questions (MCQs) are a subset of “objective questions.” I didn’t even know that there were “sets” of question types at this point, let alone “subsets.” Objective questions are basically those questions which have a correct answer (usually, though not always, one correct answer.)
The term “objective” in this set of questions means there is complete objectivity in marking the test. Wrong is wrong. Correct is correct. However, that does not make MCQs infallible, of course, as the construction of question itself is fallible (hence the five million hits on how to write effective MCQs) even if the answer should not be.
So, to press on, this is what I have learnt so far…
- MCQs are predominantly used to test factual material and the understanding of concepts (discounting ARQs the new MCQ on the block – more on those later).
- MCQs are rapidly and automatically marked (especially with QuickKey) – therefore there is no prior knowledge required to be an assessor.
- Scoring is objective and reliable – no examiner bias or halo effect.
- MCQs assessment highlights students knowledge and Question Level Analysis can also be used to highlight misunderstanding and common misunderstanding (ineffective teaching). QLA in principle is straightforward, however there is debate on how to most effectively apply QLA.
- The validity, internal reliability of MCQ quizzes can be statistically assessed; Kuder-Richardson formula, Cronbach’s (alpha), a measure of internal consistency, distribution scores, scaled scores (z and T-scores).
- MCQ quizzes can be given a difficulty rating or (p-value). The higher the value the easier the question, or quiz.* Having discussed this with the staff at Boundary Oak School – I have posted an answer.
- Question Level Analysis also provides information on how an exam itself can be improved (more of that in Part II).
Disadvantages
- MCQs take time to set.
- MCQs provide no credit for partial information or correct processes.
- MCQs accept that some correct answers can be guessed.
- High quality objective test require skilled question writers.
While the following notes refer specifically to the most common form of objective tests (MCQs), many of the comments are relate to all objective questions.
Characteristics of multiple-choice questions (MCQs)
MCQs are usually composed of four parts: the stem (the question or incomplete statement), the options (the suggested answers or completions), the incorrect responses, sometimes distractors in more subtle questions, and the key or correct response.
Up until recently, most MCQ stems I had seen were written questions, however working the Science department has shown me that graphs, diagrams or sets of results are common. Using these MCQ props leads to abilities other than recall being tested. Who said MCQs were easy? Not me, that is for sure.
As for the use of distractors, MCQs with carefully selected distractors, or closely matched keys and distractors certainly calls for more sophisticated thinking.
If you are debating four or five options. It is hardly surprising that five options provides the more reliable test results. The drawback, you have to think of another plausible distractor.
Five Basic MCQ types (plus one late entrant)
There are a wide range of MCQs that can be created both on and offline. Given that QuickKey and most teaching is offline, I have focused on the five basic offline MCQ types, setting aside rank ordering, drag and drop and free response questions.
Best answer
Composed of a “stem’ and generally 5 “distractors”, 1 of which is the best response.
Right answer
Composed of a “stem’ and generally 4 “distractors”, and 1 key response.
Multiple right answer
Composed of a “stem’ and generally 2 or 3 “distractors”, 3 or 2 key responses. Sometimes the answers are predefined. For example;
- 1 and 3 are correct
- 1 and 4 are correct
- 1, 2 3 are correct
- All are correct
- Only 3 is correct
Or there is this option taken from a Dylan Wiliams seminar back in 2011.
Ask MCQs with more than one correct answer. Do your students really know the answer? In this example, if there are two correct answers that probability of getting it both correct decreases from 1 in 6, to 1 in 15 (2 in 6 x 1 in 5) chance. I later learn from Dylan Wiliam’s, not knowing how many correct answers there are decreases the chances of guessing to 1 in 64 (2^6). (Lets not forget the ACE up your sleeve. If you really want to know, if your students fully understand the topic, don’t inform them that there are two or more correct responses and simply wait for the insightful student with a secure understanding to ask you…
Err, Sir… aren’t both A AND E right?!
Reason-assertion or ARQ
Each item in an assertion/reason questions consists of two items an assertion and a reason that are linked together. The student is required to first decide whether assertion is valid (true / false), and then decide whether the reason is valid (true/false). If one or both of the statements is false then the answer will be either c, d, or e. If both statements are true, a third step is required whereby the students must determine whether the second statement provides an accurate explanation for the first.
- True True – Reason is correct explanation
- 1 and 4 are correct – Reason is NOT correct explanation
- True False
- False True
- False False
Here is an example using the Hulstrom curve that shows the relationship between particle diameters and entrainment velocities for water taken from GCE Geography.
| Assertion | Reason | ||
| Pair 1 | Sand has a lower critical entrainment velocity than clay | BECAUSE | Sand particles are lighter than clay particles |
I have not personally used ARQs but these seem tough. In researching ARQs most exemplar appear to be scientific, medical or business / economics and linked to higher education courses. Which makes sense, as doctors, pharmacists and economists make reasoned assertions.
According to Williams (2006) “traditional MCQs usually test only one issue/concept. ARQs, on the other hand, test two per question (the assertion and the reason statements) plus the validity of the ‘because’ statement in the event assertion and reason are both correct statements. On the basis that judging the correctness of two statements must be harder than judging the correctness of one, it would follow that ARQs present more of an intellectual challenge than traditional MCQs.” Simple. these questions are good for testing whether students can see or reason out the relationship between different concepts or events. However, they are hard to write because they depend on natural connections between events.
Analogy
Here the student is expect to deduce a relationship between two parts. I first came across these when I sat an intelligence test for graduate school in the US. (MAT)
FRYING PAN : (a. iron, b. ice, c. fire, d. pot) :: ROCK : HARD PLACE
The familiar phrase “between a rock and a hard place” describes a dilemma in which a person must decide between two equally unattractive choices. The familiar phrase “out of the frying pan and into the fire” is another metaphor involving two unappealing situations. So frying pan is analogous to rock, while fire is analogous to hard place. (Analogy type: association). The correct answer is (c).
Have I moved your opinion about MCQs yet?
Late entrant – Matching
Matching questions might be a fast assessment tool, even using the same possible five answers for each questions for example. 5 questions using the same 5 responses.
A Knowledge B Comprehension C Application D Analysis E Evaluation
- Recall of information. Remembering facts, dates, names, events and places
- Put ideas, theories and concept into practice in new situations
- Interpreting information in own words as in making a summary or grasping meaning as in reading an article and being able to understand the content
- Assessing values of ideas and theory and make judgement or compare and discriminate between ideas
- Recognition of components and patterns and understanding their relationship
With this post, I have one extra reference to support the 5 millions already available. There is a relief.
Recall of information. Remembering facts, dates, names, events and places = Knowledge
Interpreting information in own words as in making a summary or grasping meaning as in reading an article and being able to understand the content = Comprehension
Put ideas, theories and concept into practice in new situations = Application
Recognition of components and patterns and understanding their relationship = Analysis
Assessing values of ideas and theory and make judgement or compare and discriminate between ideas = Evaluation
Pingback: Alternatives to MCQs on the fly – Edventures