How difficult is hard? Question or item analysis

How difficulty is a question?

Multiple Choice Questions MCQ quizzes can be given a difficulty rating or (p-value). The higher the value the easier the question, or quiz.* Having discussed this with the staff at Boundary Oak School – here is my understanding.

Your carefully construct you objective assessment items. Our conversation focused on MCQs. What we want to establish is whether or not the items effectively tested or differentiates students attainment, who do well on the overall test and those who do not?

Difficulty Index

To determine the difficulty of a test item, we use a difficulty index or Facility (F).

The difficulty index = correct answers / total answers

A general guide is if the item difficulty is more than .75, it is an easy item; if the difficulty is below .25, it is a difficult item. Elsewhere I read that questions that are answered correctly by more than 85%, or less than 25%, of the students are of questionable validity. Perhaps simpler and easier to remember.

Furthermore by looking at each distractor we can also identify weak distractors or event possibly reveal student misconceptions.

Here are 2 MCQ with three distactors and one correct answer.*

What can you tell me from reviewing Question 1?

A was a weak distractor. (24 / 30 = .8) moderately easy.

What can you tell me from reviewing Question 2?

Q2 is much more difficult (12/30 = .40). In fact, more students selected an incorrect answer (B) than selected the correct answer (A). What might the reasons be for such an outcome? A teaching error? Is B is an appropriate distractor?

Applying this calculation to a given distractor or set of distractors is referred to as distractor analysis or distractor efficient (DE).

Discrimination Index

There is another measure, the Discrimination Index, which refers to how well an assessment differentiates between high and low attainers.

You should expect that the high-attaining students select the correct answer for each question more often than the low-performing students.  If this is true, then the assessment is said to have a positive discrimination index (between 0 and 1). Students who received a high total score chose the correct answer for a specific item more often than the students who had a lower overall score.

If, however, you find that more of the low-performing students got a specific item correct, then the item has a negative discrimination index (between -1 and 0).