CBT Outcome Studies – An Example of The Problems of Generalisation

The problems of generalising from CBT randomised controlled trials can be illustrated by examining the following just published study from Clark et al (2022):


  1. All patients were referred by an Improving Access to Psychological Therapies (IAPT) Service if they were thought to be suffering from social anxiety disorder(SAD). But following detailed diagnostic assessment only one half were found to be suffering from the disorder. Thus it seems likely that IAPT staff, left to their own devices, would have provided inappropriate treatment for one in two patients.
  2. IAPT staff do not make assessments using standardised diagnostic interviews, so that there can  be no certainty that the results of this, or any other credible randomised controlled trial, would translate into routine practice.
  3. The three treatment  clinicians involved in the trial involved were experienced, well known clinical psychologists, unlike the therapists in routine practice. It is unlikely the latter would achieve the same outcome as the former.
  4. In the study internet delivered CBT and standard CBT were the active comparisons and the results set against a waiting list control condition. But patients on a waiting list do not expect to get better and this comparotor has therefore been termed a nocebo. Given such able clinicians, it is possible that, if they had provided an alternative treatment with a credible rationale e.g ‘managing shyness/better mixing’ with equal attention the results would not have been appreciably different. The chances of this are increased by Clark et al (2022) finding that many of the standard components of the original CBT, such as accompanying the patient to social experiments were found to be redundant.   
  5. The study authors were evaluating, the computerised CBT program that they had developed. But no mention that the study  requires independent replication because of the possibility of allegiance bias. Interestingly the authors report no conflict of interest. 
  6. The study required patients to have internet access and almost two thirds had higher education. The results may not be applicable in forgotten towns.
  7. Patients were required to have SAD as their main problem, but patients typically see themselves as having a range of difficulties and want treatment for all. In this study 30% were found to be suffering from depression. But strangely, overall the pre-treatment mean on the PHQ9 was below the cut off of 10 usually used to denote a case of depression. Further this score is much lower than the initial mean of PHQ9 scores in the IAPT population.  This creates doubts about the level of functional impairment in this population. 
  8. On the surface the study results are remarkable with 70%+ recovering from SAD and avoidant personality disorder. But the primary outcome measure was a composite of loss of diagnostic status and achieving below cut-off scores on several SAD self-report measures. There could be no certainty that the components of the composite were equally important to each patient, nor that the frequencies of say loss of diagnostic status and below cut off scores matched. The hazards of using composites has been highlighted by McCoy (2018).
  9. Patients see recovery as being free of a disorder for a meaningful length of time. Given that persistent SAD is defined in the DSM as having the disorder for at least 6months. It therefore seems reasonable to suggest that a primary outcome measure should have been being free of SAD for at least 6 months. This would have been a real world change.

This study is a salutary tale about the marketing of CBT – the takeaway message in the title is that the cost of one intervention(internet CBT) is half the cost of standard CBT. This is not to say that CBT is not of limited utility with depression and the anxiety disorders (including OCD and PTSD) but we need to assess what is of real world importance to the patient.


Dr Mike Scott

7 replies on “CBT Outcome Studies – An Example of The Problems of Generalisation”

You need to investigate undiagnosed female
autism. I was persecuted, experimented on. Deliberately aggravated to get a psychosis and another tried to elicit a psychotic response. I was lifelong denied autism support, denied correct benefits had spinal misdiagnosis. I was persecuted and bullied in the school community and work. it seems likely I’m ADHD I’m in the list for this too. I’ve been thrown out of practices because they were invalidating me. Labeling me hypomanic, paranoid,PDs, low mood when I had depression. Basically I was having meltdown after melt down. I did end up with psychosis and was sectioned. Peoples committed suicidal here I suspect 1 was female autism undiagnosed. I was diagnosed at 52. Seemingly no one had autism training in CNTW health authority. I have complained and they say they will train clinical staff mandatory now as a result of my complaint. I was a served TA class 1 combat medic I worked 22 years in and out of jobs was being scapegoated as the outsider. Who is responsible for diagnosing autism. My man died not knowing her own dss as ughter was autistic and I was persecuted at home. No one but us knows the gull cost to us as fellow humans and to society of misdiagnosing autism like this . The impact of it for females is we become vulnerable isolated, no mates out of work and lonely and the benefit system crucified us . And we get overload. Ramming vet down everyone’s throat and not treating us as equal humans needs to stop . In my notes it stated I’m academic but my repeat active behaviors study it. I wouldn’t of had this had I been supported and diagnosed as a child. The uk nhs mental health services persecuted me and destroyed my life and I’ve physical disability now ..

It’s really interesting to find your work Mike. My husband and I both had pretty shockingly bad experiences with IAPT recently which mirror most of the thoughts you share in your writing. We both had intuited pretty correctly I think, how this show is really run. I also work in mental health and have heard identical experiences from clients. I’m so glad you’re shining a light on this. I find the lack of care and skill in this area very worrying, particularly the incorrect approaches to treatment of people at a vulnerable time in their lives. We had correctly worked out our issues and even then the therapists actually employed ‘incorrect therapy’ so who knows what experiencs people have when they don’t yet know. I know the therapists themselves are stuck in a flawed system but the therapists often seem oblivious to the potential harm or actual they’re causing directly in their sessions despite some level of quantative feedback from the GAD tests etc.

Thanks so much Jade. If you would like to detail your experience suitably anonymised that would be great. Alternatively if u would like to chat backchannel michaeljscott1@virginmedia that would be be fine.
take good care both

Leave a Reply

Your email address will not be published. Required fields are marked *