Dig around the retrieval practice and test enhanced learning research and you will invariably bump into the research of Assistant Professor Steven Pan: interleaving, retrieval, feedback, transfer and metacognitive beliefs. His research has covered the full gamut of test enhanced learning. Most recently, I cited his work with reference to pretesting.
What caught me off guard was when he swung by my Twitter feed and shared a post. So I asked if he was currently working on anything that really excited his professional curiosity? Generously he replied:
Pretesting/prequestions as a way to improve attention during lectures is a topic that I remain quite interested in (https://sc-pan.github.io/pdf/PSSB_2020.pdf…), as an example.
Twitter
Steven even offered to feedback and answer any questions.
Pretesting Reduces Mind Wandering and Enhances Learning During Online Lectures
What is really helpful is, the paper has both an abstract and a “General Audience Summary.” That immediately makes the paper more accessible to educators.
What is the key message: pretesting “significantly” reduces “mind wandering” leading to better final test performance.
Why mind wandering rather than pretesting improves focus or attention. Are they in fact different?
Let’s unpack both Mind Wandering and Pretesting
Shall we start with mind wandering or the “occurrence of thoughts that are not tied to the immediate environment.” As teachers we know keeping and directing attention is important essential. We learn that a third to well over half of the students attending a lecture mind wander and with increasing frequency as the lecture progresses. Would it be fair to suggest that mind wandering may be amplified in a school classrooms, as opposed to lecture theatres, where there are potentially more distractions and the learners are younger. There is not a teacher in the land that has not been usurped by weather, a wasp or helicopter.
Next, as Steven recommended, to “disambiguate retrieval practice from pretesting a bit more.”
…in that both are ways of using low-stakes tests to enhance learning, but the former comes after learning and has been more thoroughly researched.
Steven Pan
I would also add one further categorisation.
Again, as I work through pre and post, I think teachers would benefit from one a further classification. Pretesting is very much about exposure and potential indirect benefit. You know or do not yet know about this knowledge. Whereas, retrieval practice, is more retrieving information, relearning / consolidation. You were exposed to this knowledge, can you access it.
Next, I learnt the correct term for interim retrieval practice, or testing that occurs during the teaching process – what Pan et al, (2020) term interpolated testing. In fact interspersing practice tests throughout a lecture or other learning materials, is ranked as one of the most promising mind wandering interventions investigated to date. It is right there on Roediger’s list of “Ten Benefits of Testing and Their Applications to Educational Practice. – Benefit 3.” A list that recently celebrated in’t 10th birthday.
Now – what is interesting, is Pan et al, (2020) commentary on incorrect responses. Indirectly, learning following both a non or errorful response.
Owing to their lack of preexisting knowledge, learners often generate many incorrect answers during pretesting and only learn of the correct answers upon subsequent study or when they receive feedback. Crucially, pretesting followed by studying of target materials or correct answer feedback yields improved long-term memory—also known as the pretesting effect—relative to conditions that lack pretests and in which information is simply studied.
Richland, Kornell, & Kao, (2009).
I would certainly encourage teachers tempted to utilise pretesting, to outline the rationale for doing so with students, and also to managing students expectations of pretesting performance. Performance is likely to be low or poor. And what better rationale than for both the teacher and the learner to ascertain what the learner already knows – and to be able to teach him accordingly, adapted from Ausubel (1968). What Pan et al (2021) refer to as the “metacognitive “reality check.”
Highlighting the gaps in knowledge and facilitating a search for the relevant information during subsequent study two indirect benefits. Or to “stimulate curiosity” which in turn improves attention for the information that follows a potential third and forth.
Now to The Present Study
Across two experiments, 105 undergraduate students viewed a lecture that was accompanied by pretesting or a control algebra problem-solving activity. Pretesting either between portions of the lecture (interpolated) or entirely before the lecture. Knowing that as the length of the video extends, mind wandering increases, it is worth know the vdeo was 26 mins in duration, divided the video into four approximately equal parts of 5–6 minutes in length.
Measures for mind wandering and metacognitive judgment of learning after viewing the entire lecture, and in the second experiment, we also assessed participants’ memory for the pretest questions.
Eight multiple choice questions for each of the four parts of video. Half of those questions appearing as pretest questions and all of them appearing on the final test
Results
Now there is a lot of data to get through, so here are my key observation.
Experiment 1: Interpolated pretests
Test performance was significantly higher for the pretested condition than the control – 83% vs 72%.
Performance was significantly higher of the pretested questions – 77% vs 73%.
Participants in the pretest condition reported paying more attention during the lecture and a significant relationship between reported attention and test performance. No judgement accuracy was benefit was reported.
Experiment 2: Pretesting
Participants in the interpolated pretest condition performed significantly better on the pretests than participants in the conventional pretest condition – 48% vs 38%.
Both interpolated 76% and the conventional 81% pretest conditions demonstrated significant learning improvements between the pretest and final test.
Attention was significantly lower in the control condition (50%) versus the interpolated pretest condition (67%) and the conventional pretest condition (71%).
Participants in the conventional pretest condition recalled significantly more pretest questions compared to those in the interpolated pretest condition.
Participants’ ability to predict how well they would score on the final test did not differ with respect to their assigned condition. If I am honest, I was not that surprised that pretesting did not enhance metacognitive calibration – from my classroom experience, pupils (11-16) are less invested in pre as opopsed to posttesting.
What does this mean?
Although attention waned in both the pretest and control conditions as the lecture progressed participants in the pretest conditions reported greater attention at all measured time points (8–21% higher than the control condition when averaged across the entire lecture). That improved level of attention translated into better final test performance: 11% for pretested questions in Experiment 1, and up to 14% for pretested and new questions in Experiment 2.
What may have attributed to the performance gains?
Increased attention, motivation to search for the answer, curiosuty, or any combination. Did the questions raise the cognitive demand? Or the eognitive demand and effort of watching the videos?
The fact that both interpolated and conventional pretesting yielded similar benefits in the present experiments suggests that the method of pretesting may not always be critical for achieving beneficial outcomes. Or it could be seen as 4 conventional pretesting experiments?
Self-reported rates of mind wandering were predictive of final test performance.
Similarly, judgments of final test performance were predictive of the final test results in all conditions, but those judgments were not more accurate following pretesting.
That pretesting is likely to have dual benefits: Students will pay greater attention to the lectures and learn more from them.
Pan et al, (2020)
If you enjoyed this post you might like Making Chemsitry Easier and Priming effects of testing.
Pan, Steven C; Schmitt, Alexandra G; Bjork, Elizabeth Ligon; Sana, Faria.Journal of Applied Research in Memory and Cognition; Washington Vol. 9, Iss. 4, (Dec 2020): 542-554. DOI:10.1016/j.jarmac.2020.07.004
Roediger, Henry & Putnam, Adam & Sumeracki, Megan. (2011). Ten Benefits of Testing and Their Applications to Educational Practice. 10.1016/B978-0-12-387691-1.00001-6.
Pingback: Attention and retrieval (part 1) – Edventures