the competence of trainee CBT therapists is routinely assessed using the Cognitive Therapy Rating Scale-Revised (CTRS-R), but a just published study by Roth et al (2019) has shown poor levels of agreement on the performance of IAPT trainees, using this measure. The levels of agreement were no better when an alternative measure of competence the University College London CBT Scale was used. On both measures of competence the intra-class correlation coefficients were less than 0.5, indicating poor reliability (on a scale poor, moderate, good, excellent). The UCL Scale is rooted in the competence framework developed by Roth and Pilling (2008) as part of the IAPT programme.
The chaos is underlined by a study conducted by Liness et al (2019), published in Cognitive Therapy and Research which assessed the competence of IAPT trainees using the CTRS-R with client outcome assessed, mainly with the PHQ9 and GAD7, and no relationship was found, either at the end of training or 12 months later, see link below:
But the same set of authors as in the Liness et al (2019) study, have published a further paper in Behavioural and Cognitive Psychotherapy, again of IAPT trainees, evaluated using the CTRS-R. But this time, in the abstract, they reported that ‘CBT competence predicted a small variance in clinical outcome for depression cases’ with no reference to the findings of their other paper! In the body of their Behavioural and Cognitive Psychotherapy report one discovers that for depression cases the CTRS-R explained 1.3% of the variance in outcome, it is extremely doubtful if this is of any social or clinical significance. There is also a failure to mention in the abstract that CTRS-R did not at all relate to anxiety. The abstract is dominated by the message that training helped trainees score highly on the CTRS-R, without acknowledging that this might be without meaning. Three of the 6 authors have links to IAPT and spin is not therefore unexpected.
Liness et al Behavioural and Cognitive Psychotherapy (2019), 47, 672–685
Roth et al Behavioural and Cognitive Psychotherapy (2019), 47, 736–744 doi:10.1017/S1352465819000316
Roth, A. D. and Pilling, S. (2008). Using an evidence-based methodology to identify the competences required to deliver effective cognitive and behavioural therapy for depression and anxiety disorders. Behavioural and Cognitive Psychotherapy, 36, 129–147. doi: 10.1017/S1352465808004141
Dr Mike Scott
One reply on “IAPT’s Unreliable Assessment of Competence, Incompetence and Spin”
Lies, damned lies, and psychotherapy research !