The Mismatch Between Clients Global Judgements and Changes on Psychometric Test

casts doubt on the wisdom of the Improving Access to Psychological Therapies IAPT) services sole reliance on test results.  A recent study by Hobbs et al (2020) compared subjects global judgements on improvement/deterioration with changes on the PHQ9 and found poor agreement. For those who reported ‘feeling worse’ PHQ9 scores showed no change or improvement for 76% of them.  Further for those who reported ‘feeling a lot worse’  for 81% of them the PHQ9 showed no change or an improvement.  

Hobbs et al (2020) conclude that test results tell only part of the client’s story and there is a need for clinical assessment.

The interview assessments in IAPT go no further than the open ended interviews that are the first part of all standardised diagnostic interviews. Open-ended interviews furnish at most differential diagnoses and highlight candidate disorders for further investigation.  Further inquiry is then made of all of the symptoms in a diagnostic set ( controlling for information variance) and thresholds are used to determine whether a symptom is present  at a level that would constitute functional impairment (controlling for criterion variance).  Reliable diagnosis makes it possible to determine which evidence based treatments are likely to be appropriate in a particular case, bearing in mind the client’s social context and cognitive capacities.

Ultimately global judgements have a more real world feel than changes on psychometric test. A client knows whether they are back to their old selves post treatment and whether they would regard this as enduring. Similarly a client with a lifelong history will now whether they are back to their ‘best functioning’ post treatment and whether it is just one more ‘flash in the pan’. Similarly an independent assessor of a Service can make g global judgement (using a standardised diagnostic interview) whether the person has lost their diagnostic status post treatment and whether there is evidence of permanence. Psychometric tests have been grossly overvalued by IAPT for quality control purposes, but they are fantastic for marketing to the unwary.

Tests Misleading for Diagnostic Acccuracy

It is common for advocates of psychometric tests to quote high sensitivities/ specificities of the order off 80% for instruments such as the PHQ9. But this does not mean that using the PHQ9 on all clients coming through the IAPT door that using its 10 or greater cut off 4 out of 5 clients will be correctly classed.  Tests are validated in a particular context, thus if the proportion of clients with diagnosed disorder (using a diagnostic interview)  differs  by context then so to will the appropriate cut offs and sensitivities/specificities.  Further a psychometric test does not indicate what other disorders are present nor which is the primary disorder. Treatment that fails to address comorbidity is likely to fall short and comorbidity is the norm not the exception. 

Dr Mike Scott

PROMS – Track and Trace for Mental Health Without Knowing What Is Being Tracked

a just published study in the British Medical Journal has found that  ‘There is insufficient evidence and mostly of low quality, that routine monitoring  with PROMS (Patient reported outcome measures) … leads to improvement in outcomes’.  Of the 5 studies reviewed one was of the Improving Access to Psychological Therapies (IAPT) Service in which the PHQ-9 and GAD-7 self report measures were used. 

Strangely the authors of the study Kendrick and Maund (2020) are surprised by the negative findings. It seems not to have occurred to them, that if it is not known with any certainty what the patients were suffering from in the first place then using the most available psychometric test to measure outcome is unlikely to yield any positive findings. In none of the studies was a standardised diagnostic interview used to establish diagnosis and determine any accompanying diagnostic comorbidity.  Thus it cannot be reliably known which is the outcome measure of primary interest, and should becomes the established yardstick before treatment begins and what secondary analyses should be declared in advance. This is akin to the need to pre-register how the results of a randomised controlled trial are going to be analysed rather than going on a post hoc fishing expedition highlighting some positive finding or other to justify a service.

Last Night of The PROMS?

The use of PROMS appears to be fuelled by the need to quickly process patients, using surrogate outcome measures. Rather than taking the time to properly listen to them and use a real world outcome measure such as loss of diagnostic status for say 8 weeks, as assessed by an independent evaluator using a standardised diagnostic interview. Psychometric tests completed for the benefit of a treating clinician are subject to demand characteristics, including wanting to please the therapist and not wanting to feel time has been wasted in engaging in psychological therapy. These concerns are amplified when tests are administered (as in IAPT) on a weekly basis and clients can easily remember their last score.

For all the deficiencies of track and trace over COVID-19,  the target is at least not a ‘fuzzy’ , rendering the process meaningless. Ironically since the demise of Public Health England Baroness Dido Harding is in charge the Covid-19-19 Track and trace. I e-mailed her asking if she was also going to assume responsibility for IAPT but have had no reply. Any QUANGO such as IAPT is likely to rejoice at the absence of accountability but to the detriment of the public. There has to be clarity about exactly who IAPT is accountable to now.

Monitoring Is Necessary But  Never Sufficient 

Just as monitoring the spread of the coronavirus is critical to triggering some preventative measures, it is likely going to be insufficient until there is an evidence based treatment protocol including a vaccine and treatment of the effected. So to only an informed monitoring of mental health problems can highlight appropriate treatment interventions. Monitoring by itself is descriptive rather than prescriptive. Unfortunately there is nothing in the Kendrick and Maund (2020) approach that is likely to make it reliably prescriptive, making their proposed developments in monitoring rather pointless.

Dr Mike Scott


Is Evidence Based Treatment Possible Without Evidence Based Assessment?

‘no’, this is the take home message from a just published study by Moses et al in the Journal of Anxiety Disorders An evidence based assessment includes a diagnostic interview, as well as a clinical interview and psychometric tests. Moses et al (2020) summarise the literature that the inclusion of a diagnostic interview improves outcome, by minimising missed diagnosis and misdiagnosis. These authors bemoan their finding that only a small minority of Australian psychologists use a diagnostic interview, but the position is even worse in the UK, as the largest provider of services the Improving Access to Psychological Therapies (IAPT) explicitly excludes the making of diagnosis/diagnostic interviews.   IAPT cannot improve access to evidence based psychological therapies because it does not operate the admission gate of an evidence based assessment.

The absence of an EBA leads to a revolving door, demoralising clients in search of a credible explanation of their difficulties. An EBA is a necessary part of evidence based practice (EBP) in that it highlights candidate evidence supported treatments (ESTs). But clinical judgement is still required to ascertain whether there is a sufficient match between client and the subjects in the EST. Most ESTs have admitted clients to the study with a limited range of comorbid disorders and have not been cognitively impaired, or suffering debilitating pain. Further the clients in the EST have been in a safe environment. 


Dr Mike Scott

IAPT and BABCP Duck Key Questions

‘what proportion of IAPT clients have maintained recovery from the primary disorder for which they first presented?’ . The Improving Access To Psychological Treatments (IAPT) Service prides itself on its’  large comprehensive database, as if this was somehow a guarantor of the effectiveness of the service.  But it is not possible to interrogate this database to determine the  extent of restoring clients to their normal functioning, as they don’t do diagnosis.

Not only don’t they do diagnosis, they refuse to share a platform with anyone known to be critical of them.  To date IAPT has not published written rebuttals of its’ critics charges. IAPT uses the muscle of the British Association of Cognitive and Behavioural Psychotherapies (BABCP) when challenged. Later this month the BABCP has its Annual Conference. I have had no indication from the President Elect as to how they are going to address my concerns over conflicts of interest and editorial freedom, but I do know that pride of place is to be given to IAPT’s leading light. BABCP is IAPT’s apologist. It might better spend its’ time investigating why the IAPT documentation indicates that its therapists, who are invariably BABCP members, make it up as they go along, sprinkling their notes with CBT terms, without any evidence of fidelity to an evidence based protocol for anything.

Dr Mike Scott



National Institute for Health Protection to Control IAPT?

in a blog written just before the demise of Public Health England I noted  the’Breathtaking Naivety of Public Health England On Mental Health’, My hope is that its’ replacement the National Institute of Health Protection (NIHP)  will question why £4billion of the taxpayers money has been spent on the Improving Access to Psychological Therapies (IAPT) Programme, without any publicly funded independent evaluation of the service. My own independent finding was that only 10% of  those going through the IAPT service recover and that the public are very dissatisfied ,. By contrast IAPT claims a 50%  recovery rate, but my just published paper in the British Journal of Clinical Psychology,  casts serious doubts on the Services claim.

I have written to Baroness Harding of Winscombe, Dido Harding, the head of NIHP  to clarify whether the NIHP is indeed going to be the monitor of IAPT’s performance and if not who is? I have also stressed that no agency, including IAPT, should be allowed to mark its’ own homework.   It is imperative that a the metric for gauging the effectiveness of a service is one that the general public would recognise as meaningful, such as being independently assessed as no longer suffering from the disorder that they first presented with, as opposed to a surrogate measure, such as a change of score on a psychometric test completed in the presence of the therapist.

As MPs resume sitting in Parliament it is critical to ask who will now be in charge of ensuring IAPT does what it says on the tin and how will this QUANGO be made accountable?

Dr Mike Scott