Predictive validity of FFT and CATs

The predictive validity of FFT and CATs is basically an extension to the baseline versus aptitude test I have been reviewing the past few days. It is a question of whether a student’s ability to pass a known knowledge test for which they are intensely prepared for (or not given their school experience) is more accurate a predictive measure than student’s aptitude? So before I wade into this minefield, let me start by saying that these two methods explore different intelligences and therefore represent different models predictive validity. It does not mean one or more accurate than the other. In fact Strand (2006) positions that

both pupils’ general transferable learning abilities, and measures of specific curricular attainments at the end of primary school have unique and distinct value at the start of the secondary phase. – Strand (2006)

Let’s take a good at the predictive validity, or correlation, of CATs and Keys Stage Attainment (Average Points Score). Strand (2006) present a table of the Y7 CAT and KS2 points scores with GCSE outcomes – Autumn Package (2002).

GL assessment makes this information available here. In summary, most GCSE outcomes tend to have their highest correlation with mean CAT score. The exceptions are English Language (0.70) and English Literature (0.61) where CAT Verbal Reasoning score alone gives a slightly higher correlation than mean CAT score (p<0001).

Lowest correlations are; Art and Design 0.49, Drama and Theatre Studies 0.45, Media, Film and TV Studies 0.48 and Spanish 0.49. Mathematics 0.80, Science 0.70 and English Language (0.69). Both GCSE Capped point scores and 5+ A* – C including English & Maths(0.65).

Multiple regression analyses reveals that reasoning scores and KS2 test scores account for somewhat different parts of the variation in KS3/GCSE outcomes. Adding the KS2 test points scores along with reasoning scores accounts for a small but significant additional part of the variance in KS3/GCSE outcomes. This confirms the opening point, that both KS2 tests and CATs explore different intelligences, and that a combination of both measures provides a better indication of future KS3/GCSE outcomes. It might also be positioned that CATs score are more resilient; impacted less by the students primary school experience and that the finer differentiation available in CAT scores makes them more accurate? Though this is most certainly not the only reason. Whereas KS2 tests are a better predictor of the students ability to sit and focus on  taking a test? Of their ability to retain knowledge?

Strand (2006) goes on to warns us that even a correlation as high as 0.70, over half the variation in outcomes is attributable to factors other than their age 11 score. Motivation and effort, the quality of teaching, the level of parent and carer support, life changes and so the list goes on. Of course FFT takes into account some of that variance, gender, month of birth, school context, student context (EAL, FSM, SEN, Ethnicity and Mobility) pending your chosen estimates model.

From everything I have read listened to, to date, all that I have read today and yesterday, whilst FFT estimates form the core of our targeting model, reasoning tests have a lot to offer. For example, filling missing data gaps, identifying over/under achievement, corroborating teachers assessments. I also want to find out more about how FFT estimates are constructed, their predictive validity, and despite my above average Google Kung Fu I can’t find that much really.

[qr_code_display]