IAPT’s Unreliable Assessment of Competence, Incompetence and Spin

the competence of trainee CBT therapists is routinely assessed using the Cognitive Therapy Rating Scale-Revised (CTRS-R), but a just published study by Roth et al (2019) has shown poor  levels of agreement on the performance of IAPT trainees, using this measure.  The levels of agreement were no better when an alternative measure of competence the University College London CBT Scale was used. On both measures of competence the intra-class correlation coefficients were less than 0.5, indicating poor reliability (on a scale poor, moderate, good, excellent). The UCL Scale is rooted in the competence framework developed by Roth and Pilling (2008) as part of the IAPT programme.

The chaos is underlined by a study conducted by Liness et al (2019), published in Cognitive Therapy and Research which assessed the competence  of IAPT trainees using the CTRS-R with client outcome assessed, mainly with the PHQ9 and GAD7, and no relationship was found, either at the end of training or 12 months later, see link below:

https://www.dropbox.com/s/e26n191ie09sngs/Competence%20and%20Outcome%20IAPT%20no%20relation%202019.pdf?dl=0

But the same set of authors as in the Liness et al (2019) study, have published a further paper in Behavioural and Cognitive Psychotherapy, again of IAPT trainees, evaluated using the CTRS-R. But this time, in the abstract, they reported that ‘CBT competence predicted a small variance in clinical outcome for depression cases’ with no reference to the findings of their other paper!  In the body of their Behavioural and Cognitive Psychotherapy report one discovers that for depression cases the CTRS-R explained 1.3% of the variance in outcome, it is extremely doubtful if this is of any social or clinical significance. There is also a failure to mention in the abstract that CTRS-R did not at all relate to anxiety.  The abstract is dominated by the message that training helped trainees score highly on the CTRS-R, without acknowledging that this might be without meaning. Three of the 6 authors have links to IAPT  and spin is not therefore unexpected. 

Liness et al  Behavioural and Cognitive Psychotherapy (2019), 47, 672–685
doi:10.1017/S1352465819000201

Roth et al  Behavioural and Cognitive Psychotherapy (2019), 47, 736–744 doi:10.1017/S1352465819000316

Roth, A. D. and Pilling, S. (2008). Using an evidence-based methodology to identify the competences required to deliver effective cognitive and behavioural therapy for depression and anxiety disorders. Behavioural and Cognitive Psychotherapy, 36, 129–147. doi: 10.1017/S1352465808004141

 

Dr Mike Scott

 

 

IAPT No Better Than Treatment As Usual

that is the conclusion of a just published study in the British Journal of Psychiatry by Serfaty et al, (2019).  In the study manualised IAPT-delivered CBT was compared to treatment as usual in treating depressive symptoms in people with advanced cancer. These authors concluded: 

‘our results suggest that resources for a relatively costly therapy such as IAPT-delivered CBT should not be considered as a first-line treatment for depression in advanced cancer. Indeed, these  findings raise important questions about the need to further evaluate the use of IAPT for people with comorbid severe illness’

Interestingly the IAPT therapists were all High Intensity Therapists  with mean Cognitive Therapy Rating Scale Score of 47.6  “at the upper end of the ‘proficient range’.” The primary outcome measure was the Beck Depression Inventory II and both the treatment as usual and the CBT group showed a mean reduction of 5 points. Curiously the MINI diagnostic interview was used to assess whether people were in fact depressed and would thereby qualify for the study but it was not re-administered at the end to determine how many were no longer depressed. Nevertheless even using a diagnostic interview trying to determine whether symptoms such as sleep disturbance and fatigue should count as part of a depression or as a feature of the illness   is realistically probably an impossible task.  The appropriate model with severe illnesses is probably support ( both tangible e.g heating allowance and emotional) rather than psychological therapy.

Serfaty et al Effectiveness of cognitive-behavioural therapy for depression in advanced cancer: CanTalk randomised controlled trial British Journal of Psychiatry (2019) Page 1 of 9. doi: 10.1192/bjp.2019.207

Dr Mike Scott

The Treatment of PTSD Has Been Destabilised……

by the advent of stabilisation groups and overvaluing trauma focussed CBT. In the wake of an extreme trauma IAPT clients can be referred to stabilisation groups. Such groups will often meet weekly for 6 weeks and participants are encouraged not to talk about the trauma but rather about its effects. However there is no empirical evidence that such groups make a real world difference. In support of such groups the work of Judith Herman  [ Group Trauma Treatment in Early Recovery (2019) Guilford Press] is often cited, her groups are for those in ‘early recovery’ but there is no specification of what is meant by ‘early’ or from what the person is recovering. IAPT’s assessment process is as vague as Judith Herman’s.

 

Sienna, a Civil Servant had a horrendous rta and after an IAPT telephone assessment was referred to a stabilisation group, she assumed it was for PTSD. The group made no difference to her functioning, nor did the 3 individual sessions of trauma focusssed cbt afterwards. Sienna dropped out of the TFCBT because it was too painful but she never did have PTSD!

 

But the problems in the treatment of PTSD are not confined to IAPT. Although trauma focussed CBT (TFCBT) is the NICE recommended treatment for PTSD, inspection of the randomised controlled trials reveals that on average only one in two people recover. NICE’s guidance can be overvalued, with clinicians continuing to pursue TFCBT when it is clearly not working. With a parallel insistence that they confront the scene of their trauma. Client’s are often more pragmatic thinking that they could get by without re-exposure to the scene, but with the therapist urging the client not to be ‘defeated’. Given the power imbalance the client is unlikely to be able to effectively voice their opinion. There is a pressing need for creative solutions when TFCBT doesn’t work and for a re-examination of the theory on which the latter rests.

I am proposing to run a ‘Getting Back To Me’ workshop next year.

 

Dr Mike Scott 

PWP’s Floundering – Problem Descriptors Are Unreliable

whilst they may describe to a greater or lesser extent the psycho-social context in which the client is operating, different therapist would disagree about the relative importance of the psychosocial stressors and have a different collection of them.  One may emphasise the clients current relationship another a harsh/ ? abusive childhood another poverty. 

At a recent workshop I gave a PWP (Psychological Wellbewing Practitioner)defended reliance on the use of problem descriptors on the basis that they could be complemented by the therapists intuition. But this was precisely the therapeutic task centred approach adopted by social work in the 1970’s, it failed to demonstrate effectiveness  and by the 1990’s social work had become confined to largely a policing role, replete to this day with meaningless checklists. I speak as a former social worker, consumer of social services for over three decades and as a psychologist.

With my psychologist hat on I am very aware of the the work of Daniel Kahneman on the use of rules of thumb (heuristics) in decision making, for example the use of the availability heuristic – the vividness of a description giving a mistaken impression of its’ likelihood, so that a therapist hearing the horrific details of a trauma assumes it must be PTSD. Loretta whom I saw recently simply had a specific phobia about driving/travelling as a passenger in a car as a result of very serious rta. Nevertheless the PWP directed her to a 6 week stabilisation group that did nothing at all for her  difficulties. But the stepping up procedure offered no protection, she attended 3 individual sessions in which she was asked to talk about and write about the trauma, she dropped out because she found the procedure too toxic.   Loretta’s difficulties in driving and travelling as a passenger were not addressed at all. I broke the good news that her problems could be simply addressed.

The PWPs were totally unaware that Beck’s first paper was on the unreliability of the standard interview. This led to the inclusion of standardised diagnostic interviews in CBT outcome studies. In my view the PWP training however quick and simple is not fit for purpose.

 

Dr Mike Scott

Ps Do listen to Radio 4 on Tuesday Sept 24th at 8.0pm  for its’ investigation into ‘The Therapy Business’

BBC Investigates ‘The Therapy Business’

 with recordings  from an IAPT (Improving Access to Psychological Treatment) client, IAPT therapist, myself and the British Medical Association.  The 37 minute programme will be broadcast on Radio 4 on Tuesday, September 24th at 8.0pm and repeated on Sunday, September 29th at 5.0pm.

hopefully this will be a springboard for the expression of the views of those most effected by IAPT, and will lead to a transformation of the Service.

Dr Mike Scott

 

 

When You Do Nothing, Repeated Testing Will Likely Indicate ‘Recovery’ or ‘Remission’ At Some Point

IAPT shouts eureka at this point, discharges the client and claims the improvement is due to the therapists efforts.  It is like declaring  that a person who is terminally ill is cured because they have had a good refreshing day gardening. In effect IAPT has gone fishing for flashes in the pan – I make  similar points in a forthcoming BBC Radio 4 investigation into IAPT. 

But studies of depression and the anxiety disorders  e.g Bruce et al (2005) require a 2 month period of symptoms not significantly impairing functioning (see also DSM-5 criteria for recurrence of depression). Bruce et al (2005) point out that anxiety sufferers naturally only have symptoms 80% of the time. With regards to depression Stegenga et al (2012) point out that for 40% of depression sufferers their depression naturally takes a variable course.  Without independent assessment of the period for which the person is without significant symptoms talk of remission/ recovery is meaningless. IAPT’s clients have not been assessed using this metric. 

Bruce et al (2005) https://www.dropbox.com/s/9powmto8miw60a2/Natural%20recovery%20in%20Social%20Phobia%20Panic%20Disporder%20and%20Generalised%20Anxiety%20Disorder.pdf?dl=0

 

Stegenga et al (2012) https://www.dropbox.com/s/k0x2fm0ds01no0k/natural%20course%20of%20depression%20stegenga%202012.pdf?dl=0

 

Dr Mike Scott

 

IAPT’s Below Intensity CBT Revolution

IAPT’s low intensity CBT should be re-branded ‘below intensity CBT’, as all the methodologically rigorous CBT outcome studies were conducted  on full dose CBT.  Guided self-help (GSH) interventions were first recommended by a NICE committee in 2007 and 2009 for depression and the anxiety disorders. In its’ wake IAPT enthusiastically adopted GSH such that by 2018, 70% of clients were being given it. But recently therapists have been told not to use the term ‘GSH’ but talk to clients instead of ‘low intensity CBT’. This re-labelling appears to have occurred because of the difficulties of engaging the public in this more obviously cheap option (see previous post).

But NICE did not conduct a systematic review of the outcome literature, rather its’ recommendations were simply the advice of its’ committee. It failed to acknowledge that there were no studies of ‘guided self-help (GSH)’ with a hard outcome measure i.e studies involving an independent blind assessor using a standardised diagnostic interview. Thus there was no evidence that the man/woman in the street would recognise that the GSH had returned them to normal functioning. However the recommendation of NICE was that the low intensity interventions had to be matched to the particular depression or anxiety disorder. But IAPT took what it wanted from the NICE guidance, jettisoned making a diagnosis and proclaimed that appropriate treatment could follow a problem descriptor, without any empirical evidence for the latter.  The upshot is that for a decade IAPT clients have largely been subjected to ‘below intensity cbt’.

There has been a decade of ‘the below intensity CBT’ revolution and it has failed. This is not to say that there may not be cheaper effective options for service delivery such as group CBT, but the scope for such interventions is limited to depression and some anxiety disorders and much more methodologically rigorous outcome studies are necessary to confirm its place.

Dr Mike Scott 

IAPT’s Failure To Engage

the IAPT Annual Report (2018)/2019] see link below, reveals that a third (31.2%) of new referrals drop out before treatment and approximately two thirds (61.1%) do not complete a course of treatment (using IAPT’s liberal definition of treatment as attending 2 or more session) with almost a third (29.54 %)  attending only one treatment session.

https://www.dropbox.com/s/hwn9ncuuyds8qfa/IAPT%20Annual%20Report%202018-2019.pdf?dl=0

IAPT’s disengagement is illustrated by Jock’s records which revealed that at age 6 he had behaviour problems and threatened to stab himself.  By age 14 he was diagnosed with oppositional defiance disorder and was short tempered. At age 19  he was diagnosed as having an anxiety state low mood drinking 10 units in a binge once or twice a fortnight  and cannabis 2-3 times a week. Despite his extensive history he was assessed by IAPT and assigned to a step 2 (low intensity) workshop, unsurprisingly he DNA’d. Two years later he is referred to them again for depression and unsurprisinly he does not respond to their opt in letter. Five years later the GP notes that he is struggling with an online CBT course has had to enlist his father to help because he is not computer literate. Then after a major negative life event he develops a depressive psychosis. Had IAPT bothered to listen this troubled soul of longstanding, the results could have been very different.

Institutional Disengagement

Engagement difficulties are built into the fabric of IAPT. Daniel consulted his GP 2 years after a major trauma and was found to have PTSD and depression and was promised a referral to IAPT. 4 weeks later he was prescribed an increase in medication and a different GP gave him IAPT’s telephone number to ring. Daniel was furious, he felt that he had explained that his mood was very up and down and that he could not be relied on to ring them. His interpretation of the organisational setup was that no one was really interested. This perception was likely to be compounded if and when he underwent a telephone assessment as had already had lots of acrimonious telephone conversations with the housing Dept and DWP since his trauma.   

It is surely time f or the Care Quality Commission and the National Audit Office to take note of the near universal disengagement of clients, voting with their feet, and institute an independent review of IAPT to determine what if any real world difference it makes. There is considerable media interest in these failings.

Dr Mike Scott

 

 

The Cognitive Therapy Rating Scale – a rite of passage?

as an External Examiner I found it painful to watch videos of trainees trying to ensure their interview with a client was CTRS (or its’ successor the Revised Cognitive Therapy Rating Scale) compliant. I vividly remember one Course Leader giving a student a score above the competence threshold of 36 (see R-CTRS Manual, link at end of blog ), despite the student barely making eye contact with the client – the latter was busy leaving through the 12 item (each scored 0-6) scale on his lap! Unfortunately institutions were obliged to use it, and students groaned in silence. Before another cohort of trainees endures this rite of passage, those that have the courage to, should consider that the R-CTRS may cause more problems rather than it solves i.e that it is, iatrogenic.

Life Before the CTRS

In the seminal study of CBT for depression by Rush, Beck, Kovacs and Hollon (1977) https://link.springer.com/journal/10608 they did not use the CTRS. Later when Steve Stradling and I conducted a randomised controlled trial comparing individual CBT, group CBT and treatment as usual, [Behavioural Psychotherapy, 18, 1-19]  we simply followed Beck’s protocol, p409-411 (1979) Cognitive Therapy of Depression, by Beck, Rush, Shaw and Emery., published by John Wiley and Sons to achieve good results.

The Poor Predictive Value of the R-CTRS

The CTRS has only been evaluated in a sample of depressed clients undergoing cognitive therapy [Shaw et al (1999)] , therapists scores on this did not  predict outcome on self-report measures the Beck Depression Inventory or the SCL-90 (a more general measure of psychological  distress) however it did predict outcome on the clinician administered Hamilton Depression Scale predicting just 19% of the variance in outcome, but it was the structure parts of the scale (setting of an agenda, pacing, homework) that accounted for this 19% not items measuring socratic dialogue etc. The authors concluded: ‘The results are, however, not as strong or consistent as expected’When the CTRS was first evaluated the results were not compelling’ . I enlarged on this previously in my blog ‘The Mis-Selling of the R-CTRS’ http://www.cbtwatch.com/mis-selling-cognitive-therapy-rating-scale/.

The R-CTRS and IAPT

Since my earlier blog I have more recently blogged ‘Jump Through Our Hoops an Make No Difference To Client Outcome’, http://www.cbtwatch.com/iapt-training-jump-through-our-hoops-and-make-no-difference-to-client-outcome/ in which I noted an IAPT study that had used the R-CTRS to predict outcome and found that there was no relationship to outcome.

Cavalier Usage of The R-CTRS

Studies using the R-CTRS tend to be cavalier. In a study comparing BA and CBT, Richards et al (2016) used the R-CTRS but these authors did not report how this or indeed the competence measure for BA related to outcome. Richards et al said that though both modalities were equally effective in treating depression, but BA was to be preferred because it was cheaper to train therapists in BA. They further claim that the CBT therapists were competent with a mean score of 37.9 on the R-CTRS ( but this score is almost identical to the threshold of 36 in the R-CTRS Manual deemed necessary for a competent therapist) so on this metric half of the CBT therapists were not competent. Thus there has been a meaningless implementation of CBT. Paradoxically it may be that the CBT therapists performance had been made worse by having to use the R-CTRS.

Spinning The R-CTRS

Given the paucity of evidence for the utility of the R-CTRS for depression and possible negative side effects one would expect that it would not have been applied to other disorders. Unfortunately trainees are asked to apply it to whatever the client’s complaint or ‘problem descriptor’ as IAPT would have it. Little wonder that trainees are stressed by its’ usage.

Revised Cognitive Therapy Rating Scale

https://www.dropbox.com/s/760c9i1i3hlhcd5/Cognitive%20Therapy%20Scale%20Revised.pdf?dl=0

Richards et al (2016)

https://www.dropbox.com/s/3kp53zqz61ewzrw/cbt%20v%20ba%20Richards%202016.pdf?dl=0

Dr Mike scott