casts doubt on the wisdom of the Improving Access to Psychological Therapies IAPT) services sole reliance on test results. A recent study by Hobbs et al (2020) https://doi.org/10.1017/S0033291719003878 compared subjects global judgements on improvement/deterioration with changes on the PHQ9 and found poor agreement. For those who reported ‘feeling worse’ PHQ9 scores showed no change or improvement for 76% of them. Further for those who reported ‘feeling a lot worse’ for 81% of them the PHQ9 showed no change or an improvement.
Hobbs et al (2020) conclude that test results tell only part of the client’s story and there is a need for clinical assessment.
The interview assessments in IAPT go no further than the open ended interviews that are the first part of all standardised diagnostic interviews. Open-ended interviews furnish at most differential diagnoses and highlight candidate disorders for further investigation. Further inquiry is then made of all of the symptoms in a diagnostic set ( controlling for information variance) and thresholds are used to determine whether a symptom is present at a level that would constitute functional impairment (controlling for criterion variance). Reliable diagnosis makes it possible to determine which evidence based treatments are likely to be appropriate in a particular case, bearing in mind the client’s social context and cognitive capacities.
Ultimately global judgements have a more real world feel than changes on psychometric test. A client knows whether they are back to their old selves post treatment and whether they would regard this as enduring. Similarly a client with a lifelong history will now whether they are back to their ‘best functioning’ post treatment and whether it is just one more ‘flash in the pan’. Similarly an independent assessor of a Service can make g global judgement (using a standardised diagnostic interview) whether the person has lost their diagnostic status post treatment and whether there is evidence of permanence. Psychometric tests have been grossly overvalued by IAPT for quality control purposes, but they are fantastic for marketing to the unwary.
Tests Misleading for Diagnostic Acccuracy
It is common for advocates of psychometric tests to quote high sensitivities/ specificities of the order off 80% for instruments such as the PHQ9. But this does not mean that using the PHQ9 on all clients coming through the IAPT door that using its 10 or greater cut off 4 out of 5 clients will be correctly classed. Tests are validated in a particular context, thus if the proportion of clients with diagnosed disorder (using a diagnostic interview) differs by context then so to will the appropriate cut offs and sensitivities/specificities. Further a psychometric test does not indicate what other disorders are present nor which is the primary disorder. Treatment that fails to address comorbidity is likely to fall short and comorbidity is the norm not the exception.
Dr Mike Scott
3 replies on “The Mismatch Between Clients Global Judgements and Changes on Psychometric Test”
I used to have to use the “problem descriptors” in PCMIS for everyone I saw. Many came under “mixed anxiety and depression” descriptor, however we were stopped from using this as it messed up the figures elsewhere, due to there being no NICE guidelines for mixed anxiety and depression.
Presumably also the PHQ9 is designed to be accurate when someone is rating the frequency of symptoms over the previous fortnight (?) and it is standard in IAPT that people will be rating over the past week only as therapy sessions are normally weekly.
IAPT’s sole reliance on the PHQ9 means that 50% of those found to be positive on the test (greater or equal to 10) are actually false positives.[ see Brooke Levis et al (2019) BMJ] Thus half of those IAPT would deem need treatment for depression are actually incorrectly treated. IAPTs results must therefore fall far short of the results of randomised controlled trials in which all patients were known to have depression initially with regard to a standardised diagnostic interview, yet the recovery rate in the acts was only 50% thus the recovery rate at the very most in IAPT for depression cases can only be 25% and this is probably an overestimate as it assumes IAPT therapists are as good as the highly trained therapists in rcts!
As Kojay rightly points out the PHQ9 is inappropriate for weekly assessment as in IAPT because the test specifies previous 2 weeks.
Mike