Following yesterday’s read looking at the sequence effects of generative activities and retrieval practice, I am now reading a few “starred” Prof. Dr. Julian Roelle papers. Unsurprisingly, a paper on adaptive practice quizzing caught my attention, which led to Svenja Heitmann dissertation “On Optimising the Use of Practice Quizzing: Benefits Increase by Adapting and May Depend on Learners’ Achievement Motives,” – Prof. Dr. Julian Roelle her Supervisor. This dissertation then is the basis for a series of three Heitmann et al., publications connected with “adaptivity on practice quizzing.”
- “Testing is more desirable when it is adaptive and still desirable when compared to note-taking” (Heitmann et al., 2018).
- “The quizzing effect depends on hope of success and can be optimized by cognitive load-based adaptation” (Heitmann et al., 2022).
- “Adaptive practice quizzing in a university lecture: A pre-registered field experiment” (Heitmann et al., 2021).
Dissertation Summary
“Practice quizzing is a well-established retrieval-based means to foster learning outcomes,” both in the “laboratory and the field” classroom. The focus of the dissertation is the interaction between practice quizzing and learner characteristics, in this case their “cognitive prerequisites,” or “prior knowledge” to you and me. (Later research papers will look at two other learner characteristics).
Her premise, that adjusting the quiz questions’ level of complexity would lead to higher learning outcomes, that providing learners with questions that trigger effortful yet successful retrieval is, should be targeted. Second, and importantly, Heitmann highlights that when it comes to learning, “no two learners” are the same. That the benefits derived from answering the same set of questions can therefore be expected to differ greatly between individuals.
A solution to this problem would be to use an individually adapted (personalised) quiz question sequence, based on each learner’s cognitive prerequisites in lieu of a pre-fixed quiz question sequence.
Two adaptive mechanisms were investigated:
- performance-based – performance on the quiz questions as an indicator of retrieval success – the focus of Heitmann et al., (2018)
- cognitive demand-based – perceived cognitive demand when answering the quiz questions as an indicator of retrieval effort – the focus of Heitmann et al., (2022)
The dissertation looked at learners achievement motives, “hopes of success” and “fear of failure.” An area of test enhanced learning that has been signposted for further investigation by Dr Bob Bjork. We will cover this in more detail in subsequent posts.
Question 1
Does providing learners with quiz questions that match their cognitive prerequisites increase retrieval success? In the case of Heitmann et al., (2018), the learners’ self-scored performance on the quiz questions (harnessing the diagnostic power of the quiz questions) was used to determine the next question’s complexity. Would this adaptation or personalisation improve learner outcomes?
Method summary – Heitmann et al., (2018)
Pre-test – to measure the participants’ knowledge on the topic.
The initial study phase consisted of a 30 minute e-lecture (26 narrated slides) with three subtopics. Participants assigned to (a) adaptive testing, (n = 43)
(b) non-adaptive testing, (n = 49) (c) note-taking, (n = 46) or (d) focused note-taking groups (n = 49). Two testing conditions across four levels of complexity. After each test question, the participants evaluated their responses by using a feedback form.
- Level 1 – reproduce main content from the e-lecture one subtopic at a time
- Level 2 – test questions asked for the analysis of the situation with respect to only one social norm or technique
- Level 3 – the test questions required learners to simultaneously consider two social norms or techniques at a time
- Level 4 – test questions, all three social norms or techniques were intertwined
Once the participants had evaluated their answers, a score was automatically computed for the test question. This score provided the basis for the adaptation in the adaptive testing group.
The control group strategy – a note-taking condition rather than restudy or business-as-usual. The note-taking condition received the e-lecture and were instructed to write down notes in text-boxes that were located next to them.
Learning outcomes were assessed via a posttest one week later.
The research led to a twin track of key questions.
Track 1:
- Is practice quizzing optimisable by taking an adaptive approach based on performance on the quiz questions (performance-based adaptation effect)? (Heitmann et al., 2018).
- Is practice quizzing optimisable by taking an adaptive approach based on cognitive demand induced by answering the quiz questions (cognitive demand-based adaptation effect)? (Heitmann et al., 2022).
- If adaptive practice quizzing exerts a positive effect, does that effect translate into the field? (Heitmann et al., 2021).
Track 2:
- Do the benefits of practice quizzing increase with increasing hope of success and decreasing fear of failure compared to note-taking? (Heitmann et al., 2022)
- If the benefits of practice quizzing increase with increasing hope of success and decreasing fear of failure compared to note-taking, does it translate into the field?(Heitmann et al., 2021).
Results summary from Heitmann et al., (2018).
Learners in the performance-based adaptive condition performed better in the posttest than learners in the non-adaptive condition, and that learners in both practice quizzing conditions had higher learning outcomes than learners in the note-taking condition.
We will come back to achievement motives and perceived cognitive demand in the next post.
What do we learn?
The conclusion is clear in the title: “Testing Is More Desirable When It Is Adaptive and Still Desirable When Compared to Note-Taking.” That these benefits were mediated by the adaptive learners’ higher testing performance and lower perceived cognitive demand during testing. The inference being that testing as learning, freed up thinking spaces when the real test, the exam, came along.
Increasing the fit between the learners’ current state of knowledge and the test questions increased the benefits of testing (d=0.29) compared to non-adaptive testing (d=0.09). Therefore, the desirability of testing evidently can be improved by providing adaptive test questions. Secondly, our results indicate that non-adaptive specific testing leads to better learning of tested content than note-taking, even if the latter is supported by focus guides. These findings strengthen the case for implementing specific testing in real educational environments.
Heitmann et al., (2018)
Remember this improved performance was compared to note-taking (not restudy or business-as-usual) on a one week posttest.
Next up Heitmann et al., (2022).
If you happen to be interested in personalised, spaced retrieval practice – you might find RememberMore helpful.
Pingback: Adapting quizzing: perception of cognitive load #retrievalpractice – Edventures
Pingback: Adaptive quizzing in the field – Edventures