IAPT’s Hall of Mirrors Feedback

negates improved practice. The Organisation’s usage of the Patient Experience Questionnaire (PEQ) violates all the key requirements for eliciting feedback, which has to involve:

a) a response to questions that have an unambiguous real world meaning e.g ‘are you back to your old self  since treatment? i.e questions have to have validity

b) drawing upon a consecutive series of clients

c) completion anonymously and not in the presence of the therapist

d) a representative sample.

IAPT makes a token gesture of soliciting feedback using the Patients Experience questionnaire,  but its’ modus operandi is such that it cannot furnish a body of evidence that would demonstrate that it is not a ‘world beater’. In effect it has operated with a confirmatory bias only seeking information that would confirm its’ prior belief.

In 2016-2017, of 219 Clinical Commissioning Groups in England, 55 (25%) produced no PEQ data at all, i.e that a quarter of CCGs were funding a service in the absence of any evidence that the IAPT Service was eliciting feedback. In 2017-2018 of the 554,709 clients who completed a course of treatment  only 22% Moller et al (2019) https://doi.org/10.1186/s12888-019-2235-z completed the treatment questions on the PEQ.

There is no evidence that IAPT took active steps to ensure completion by consecutive cases or that therapists were blind to the results of the PEQ. The questions on the PEQ are themselves of doubtful validity e.g ‘how satisfied are you with your assessment’, this  question presupposes that the client has the knowledge base as to what constitutes a good assessment, imagine  if asked this question as a possible COVID patient most of us would believe ‘it is outside my expertise to judge the matter’. The question on the PEQ ‘on reflection, did you get the help that mattered to you’ is ambiguous, is it asking about the emotional significance of the encounter with the therapist, for example did they really listen/seem to care or about whether as a result of treatment you had returned to normal functioning.

Government and Clinical Commissioning Groups have turned a blind eye to IAPT’s failure to systematically elicit feedback from clients. At a cost of £4billion over the last decade and an incalcu cost to mental health. sufferers.

Dr Mike Scott

 

 

‘CBT Is Not A Treatment Or Cure for ME/CFS’

announced NICE in its’ draft guidance, https://t.co/OWJjzbQGqK?amp=1 on November 10th 2020. This is likely to come as a shock to Improving Access to Psychological Therapies (IAPT) Service clinicians charged with delivering treatment to patients with Medically Unexplained symptoms, a term which embraces chronic fatigue syndrome (CFS) sufferers, those with musculo- skeletal pain and those with irritable bowel syndrome.  Will the IAPT clinicians explain to CFS sufferers that they are not providing treatment, just helping them ‘manage their symptoms’ as NICE now recommends?.

Will they tell their client that they don’t know what the odds are that the programme will make them ‘manage their symptoms’ better?  

It will no longer be credible for IAPT workers to present the CBT model of MUS which posits that the latter is a product of an interaction of personality factors like perfectionism, illness beliefs (attributions), cognitions (catastrophising), and behaviours such as symptom focusing (somatising) and avoiding activity (due to fear-avoidance or anxiety), (albeit that biological factors are acknowledged but given no explanatory powers). If indeed this CBT model is at the core of the pathology underpinning CFS, then CBT treatments should have resolved the disorder and not simply have helped sufferers ‘manage their symptoms’.  

IAPT assesses the competence of its CFS clinicians, in part, by whether they explain the CBT model of MUS. They need an urgent rethink.  How many of its’ therapists have they stressed out by asking them to promulgate a model that doesn’t deserve the term. It is not surprising that when Keith Geraghty and I (2020) reviewed IAPT data doi: 10.1186/s40359-020-0380-2 we found that only 4.6% of the MUS patients IAPT taken into treatment, complete intensive CBT.

NICE have taken seriously the views of sufferers and the experiences of clinicians and have been unimpressed by the quality of studies on the effectiveness of CBT for CFS. Unfortunately consumer and clinicians views on CBT meted out in routine IAPT practice,  for supposedly depression and the anxiety disorders, have yet to be elicited and what we have been offered is research evidence that is far from compelling,  Scott (2020) https://doi.org/10.1111/bjc.12264. 

Professor Jonathan Edwards submission to the NICE guideline Committee are a must read he highlights the importance of support over treatment:

much more emphasis is needed on ongoing support, rather than the use of short treatment courses. Perhaps the greatest missed opportunity of trials focussing on stereotyped treatment modalities is a failure to recognise the value of support from experienced carers that simply aims to make life easier to cope with, rather than address a theoretical model’

His is a clarion call  for basic humanity, being with people rather than believing that everything is fixable, walking the road with them. This should not be put under a CBT umbrella, to do so is a gross imperialism. Professor Edwards strictures  on the research are exemplary:

 ‘all the studies that claim to show benefit from therapist-delivered treatments in ME/CFS are unblinded trials that make use of subjective outcome measures… ….unblinded trials with subjective outcomes are specifically considered unreliable’ 

He goes on to  detail why such methodological rigour is required such as finding that repeating an experiment with unblinded assessors has discredited the initial positive findings and the problems of not taking seriously allegiance bias and none declaration of conflicts of interest. professor Edwards addresses the issue of honesty thus on p10 of his submission:

Patients are entitled to be fully informed, based on reliable evidence. As I understand it, patients receiving CBT are not told that the explanation given for the illness is speculative, that there is no reliable evidence for the efficacy of the treatment or that objective measurements and long-term follow-up suggests there is none of consequence. Convincing the patient that the explanation of the illness is correct, and that the treatment is effective, appear to be inherent to the way CBT is supposed to work. It would therefore appear to be unethical to continue development and use of CBT based on the current approach of altering beliefs about e.g. disease causation or effects of exercise’.

But exactly the same critique can be applied to IAPT’s low intensity CBT interventions.

 

 

Dr Mike Scott

 

Spinning Low Intensity CBT and Treatments for Refugees

How seriously should we take claims for the efficacy of an intervention?  The ‘Risk of bias” assessment tool developed by the Cochrane Collaboration (2011), see slide, helps distinguish clever marketing from the genuine article. All LiCBT intervention studies fail the Tools requirement for the blinding of outcome assessment, suggesting a high risk of bias.  

In LiCBT studies, there are no independent assessments to determine whether those treated were no longer suffering  from the primary disorder for which they first presented.  LiCBT interventions is the most common treatment modality in the Improving Access to Psychological Therapies (IAPT) service.

But the LiCBT studies are not only subject to this detection bias, but in many instances they are also subject to allegiance bias, with the main author of the study evaluating their own manualised intervention e.g Williams et al (2018) evaluation of the Living Life To The Full Classes doi: 10.1192/bjp.2017.18

But the purveyors of LiCBT interventions do not have a monopoly on proclaiming effectiveness were non exists. A just published study by Kip et al (2020) DOI: 10.1002/cpp.2446 in Clinical Psychology and Psychotherapy  claims that ‘psychological interventions can effectively reduce symptoms of both PTSD and depression in adult refugees’. But of the 14 studies on adult refugees  only in 4 studies was outcome assessed by a ‘gold standard’ semi structured interview (the CAPS). The method of determining the diagnostic status of the refugees at entry to the study is unclear, but of the 9 questions asked to determine the quality of the included studies  the lowest score   was for whether the diagnosis was determined by using a semi structured interview. The next lowest score was for the item referring to fidelity checks i.e did the clients actually receive the treatment that it was purported they receive. Five out of six studies on Narrative Exposure Therapy  involved at least one author of the published manual on NET.  In total half of the trials researchers  were involved in utilising manuals whose development they were involved in. There is clearly a pressing need for independent replication of these findings on refugees. There is also a need to higher the methodological bar, as it is impossible to say from this review what proportion of refugees were no longer suffering from PTSD at the end of treatment, nor how long that improvement persisted.

Like IAPT the understandable wish to further dissemination takes precedence over a determination about whether treatment makes a real world difference to clients lives.

Dr Mike Scott

 

Mental Health Sufferers Vote With Their Feet and Government Does Nothing At All

 of those who undergo an initial assessment with the Improving Access to Psychological Therapies (IAPT) Service 40% do not go on to have treatment, and about the same proportion (42%) attend only one treatment session, according to a just published study by Davis A, Smith T, Talbot J, et al. Evid Based Ment Health 2020;23:8–14. doi:10.1136/ebmental-2019-300133. These findings echo a study  published last year by Moller et al (2019) https://doi.org/10.1186/s12888-019-2235-z, on a smaller sample, which suggested that 29% were non-starters and that the same proportion attended only one treatment session. Further scrutiny of the data reveals that about 3 out of 4 people drop out of treatment once begun. Unsurprisingly the authors’s independent study, of 90 IAPT clients, Scott (2018) revealed that only the tip of the iceberg (9.2%) recovered                 DOI: 10.1177/1359105318755264, raising serious questions about why the Government has spent over £4 billion on the service.

What Has Gone Wrong?

Kline et al (2020) consider that at an assessment by a clinician is supposed to: a) provide a credible rationale for the proposed treatment b)  detail the efficacy of the envisaged treatment and c) ensure that the clients preferences are acknowledged. IAPT’ assessments fail on all counts, taking these in turn:

a. If the problem is ill-defined e.g low mood/stress it is not clear what rationale should be presented. It is doubtful that a 30-45 minute telephone conversation can provide sufficient space to define the primary problem and other problems/disorders that may complicate treatment. Initial assessments  of patients for randomised controlled trials of psychological interventions are typically 90 mins plus, if this is the time deemed necessary to reliably diagnose a patient by a highly trained clinician, how can a much less trained PWP do it in less than half the time? Under time pressure a PWP may consider providing a credible rationale is part of treatment not assessment and in such circumstances it becomes more likely that a client will default. 

b. How often do PWPs present clients with evidence on the efficacy of an intervention? Take for example, computer assisted CBT, does the therapist tell the client that only 7 out of 48 of NHS recommended e-therapies have been subjected to randomised controlled trials, ( see Simmonds-buckley et al J Med Internet Res 2020;22(10):e17049) doi: 10.2196/170490 and even in these a gold standard semi-structured diagnostic interview conducted by a blind assessor was not use to determine diagnostic status post treatment, i.e there was no determination of the proportion of clients who were back to their old self after treatment and for how long. Further the e-therapies had average dropout rates of 31%.  They are not evidence based treatments in the way the NICE recommended high intensity treatments are. But approximately three quarters Of IAPT interventions (73%) are low intensity first, with 4% stepped up to high intensity and 20% in total receiving a high intensity intervention Davis A, Smith T, Talbot J, et al. Evid Based Ment Health 2020;23:8–14. doi:10.1136/ebmental-2019-300133

c. Client’s preferences are a predictor of engagement in treatment, but how often is a client given a choice between a low intensity intervention and a high intensity intervention. If both options are juxtaposed choice is likely skewed by informing the client that the high intensity intervention has a much longer waiting time.

Defining A Dropout

The generally accepted definition of a dropout is attending less than 7 sessions [see Kline et al (2020) https://doi.org/10.1016/j.brat.2020.103750], it is held that clients attending below this number will have had a sub-therapeutic dose of treatment and are therefore unlikely to respond]. Applying this metric to IAPT’s dataset is difficult as they only report data for those who complete 2 or more sessions and for which the average number of sessions attended is 6, thus the likely dropout rate from IAPT treatment, as most would understand the term, is about 75%.  But IAPT has developed its’ own definition of a completer as one who attends 2 or more sessions. This strange definition serves only to muddy the waters on its haemorrhaging of clients. It makes no sense to continue to fund IAPT without an independent  government inquiry into its’ modus operandi.

 

An Alternative Way Forward

Such has been the marketing power of IAPT over the last decade, that professional organisations such as the British Association for Behavioural and Cognitive Psychotherapies (BABCP) and the British Psychological Society (BPS) have sat mesmerised, as the Services fellow travellers have dominated accreditation and training.     In ‘Simply Effective Cognitive Behaviour Therapy’  published in (2009) by Routledge, I detailed a very different way of delivering services, that represents a faithful translation of the CBT treatments delivered in the randomised controlled trials (rcts) for depression and the anxiety disorders. Unfortunately it is IAPT’s fundamentalist translation of the rcts that has held sway and has brooked no debate either in journals or at Conferences.

 

Dr Mike Scott

 

Psychological Wellbeing Practitioners (PWPs) Ill-equipped, Yet On Sentry Duty

John, a client of mine,  had a telephone assessment by a PWP at the IAPT (Improving Access to Psychological Therapies) Service and was told that he might have bipolar disorder. IAPT referred him to Secondary care but without any indication of how long he would have to wait for an appointment, nor any indication of the possible consequences. Within minutes of talking to him it was apparent that he had never had an elevation of mood that lasted more than a day. Albeit, that on his best days he felt he could do anything, but others had never reported that his behaviour was strange or bizarre at those times. John did get low but not most of the day, most days.

The PWP hadn’t picked up at all that he was troubled by obsessive thoughts of engaging in embarrassing behaviour. John in fact had OCD but without overt ritualistic behaviour. It takes little imagination to realise that a client is likely to Google any suggestion that comes from clinician, making it wholly unacceptable to hint at severe mental illness without due care and attention. PWPs are simply not equipped for the purpose of guarding entry into the mental health services. Nor is there any credible evidence that they deliver evidence based treatment. 

Consider how PWPs operate, armed with the results of a PHQ9, they interpret a score of 10 or more as indicative of depression. As a result it is quite likely they will be placed on what they consider a suitable trajectory for a depressed client. But Zimmerman (2019) [Using the 9-item Patient Health Questionnaire to Screen for and Monitor Depression, JAMA, 322, 2125-2126] has pointed out that the instrument over diagnoses people as being in the severe depression category and misses people in the mild category. He cautions that the instrument is not diagnostic and should only be used in the context of a standardised semi-structured interview such as the SCID, measuring change. Both BABCP and the British Psychological Society have been enthusiastic backers of IAPT, but have taken no steps to ensure that PWPs are aware of the limitations of self-report measures. They bear the responsibility for the current mess – for their own reasons they have put dissemination of services ahead of everything else.

A study by Chris Williams (2018) Williams, C., McClay, C., Matthews, L., McConnachie, A., Haig, C., Walker, A., & Morrison, J. (2018). Community-based group guided self-help intervention for low mood and stress: Randomised controlled trial. The British Journal of Psychiatry, 212(2), 88-95. doi:10.1192/bjp.2017.1 shows the bizarre conclusions that can flow from reliance on the PHQ9. He and his colleagues recruited patients from the community with possible depression, with a mean PHQ9 score of 15.2 (and standard deviation of 5.4) thus over 80% (83.8%) of cases would have been regarded as depressed.  But according to the MINI diagnostic interview none of those in the immediate treatment group met criteria for depression and only 4% of those in the delayed access had depression. Despite the colossal mismatch between the PHQ9 results and the MINI  he and his colleagues concluded that ‘low intensity class based CBT delivered within a community setting is effective for reducing depression, anxiety and impaired social function’ .  This highlights the weak to non-existent evidential base for low intensity cbt.

 

Dr Mike Scott

 

IAPT’s Below Intensity CBT – Rhetoric vs Reality

 

The Improving Access to Psychological Therapies (IAPT) Service clients receive on average 6 sessions (amongst those who complete 2 or more sessions), compared to the typical 13  sessions in the 29 evidence based, ‘gold standard’, randomised controlled trials, considered by OST (2008) doi:10.1016/j.brat.2007.12.005. The discrepancy in the dosage of treatment creates a suspicion that, in routine practice, clients receive a sub-therapeutic dose of treatment.

In practice the Service haemorrhages clients, with 60% of all referrals not completing treatment in 2017-2018,  Moller et al (2019) https://doi.org/10.1186/s12888-019-2235-z.  In this period a third (35%) of clients were given an unspecified diagnosis making the delivery of a NICE approved evidence based treatment impossible. The IAPT services claim to a 50% recovery rate, comparable to that in randomised controlled trials, is preposterous in this context.

Service providers, such as IAPT, are necessarily opportunistic and overstate the power of a simple and cheap mode of service delivery, thereby doing wonders for their power and bank balance. But clients are the losers, particularly those from disadvantaged backgrounds, Moller et al (2019) https://doi.org/10.1186/s12888-019-2235-z.

The rhetoric was that the Service would pay for itself because of the positive effects on employment but Moller et al (2019) have found no evidence of this https://doi.org/10.1186/s12888-019-2235-z.Employed: Start of treatment 316,604; end of treatment, 302,746; Unemployed and seeking work: Start of Treatment, 54,580; End of Treatment 49,803; Long term sick or disabled or in receipt of benefits; Start of Treatment, 43,275; End of Treatment, 43,671. Using IAPT’s own data for 2017-2018,  there is a 17% difference in recovery rate depending on whether the client was from the most deprived area , with a 41.0% recovery rate compared to a 58.1% recovery rate for the least deprived area.

 

Dr Mike Scott

 

 

The Mistreatment of IAPT Clients – The Smoking Gun

What right has the Improving Access to Psychological Therapies (IAPT) Service to routinely label each client with a diagnostic code (ICD of the International Classification of Disorders, World Health Organisation) when the Organisation’s Manual states that it does not do diagnosis. Fearful of litigation, it states that its diagnoses should not be used for medico-legal purposes, but as the code is the determinant of treatment, IAPT should be in the dock!

The Improving Access to Psychological Therapies (IAPT) service screens clients for treatment using the PHQ-9 but a study published in the British Medical Journal by Brooke Levis et al last year http://dx.doi.org/10.1136/bmj.l1476  indicates that half the  deemed depression cases  have been incorrectly diagnosed. 

In high quality randomised controlled trials of the treatment for depression all clients admitted have been diagnosed as having depression according to a ‘gold standard’ diagnostic interview such as the SCID. The recovery rate in the rcts is 50%. But IAPT claims that it approaches the recovery rate of rcts. This is preposterous! Consider 100 IAPT cases which score above the PHQ-9 cut off of greater or equal to 10. One half of them i.e 50 will not actually have depression and therefore cannot recover from the disorder. Of the other half, 50 cases, if the IAPT clinicians were as good as in the rcts 25 would recover. Thus the maximum possible recovery rate for depression in IAPT is 25% and this is assuming its clinicians are as good as the highly trained clinicians in rcts. More plausibly the recovery rate for depression in IAPT is the 14.9% I found in my independent study of IAPT, http://DOI: 10.1177/1359105318755264 https://doi.org/10.1177/135910531875 using the SCID.

In primary care 22% of patients score over 10 on the PHQ9, so what are the treatment implications for the likely 3 out of 4 IAPT clients who score below 10? For these the PHQ-9 offers no direction.

But IAPT has its’ own answer, IAPT Manual, p 24  (2019), a) come up with a problem descriptor then choose an ICD 10 that that ‘matches’ the descriptor and  then b) a NICE treatment that matches the ICD 10 code.  Consider an IAPT client who reports that they are feeling emotionally numb at work, detached from others and fatigued after little exercise. The therapist could plump for either depression, burnout, chronic fatigue syndrome or the effects of COVID-19, with no guidance as the appropriate label! 

Using IAPT system Delgadillo et al (2020) http://dx.doi.org/10.1037/ccp0000507 classified over 40% of clients as having ‘Affective Disorder’  and over 20%  as having a ‘mixed disorder’. But there are no randomised controlled trials for ‘affective disorder’ or ‘mixed’, so that for 60% of IAPT’s clients there cannot be an appeal to an evidence based treatment (i.e one based on a randomised controlled trial). Considering again a sample of 100 IAPT clients who score less than 10 on the PHQ9 60 of them will have been labelled with a disorder for which there can be no evidence based treatment, this leaves 40 clients who in principle could be treated  with an evidence based treatment. Again assuming that for this population of 40 that allegedly covered GAD (10-12%), panic disorder (4-6%), social anxiety disorder (4-6%), specific phobia (0.5-1.0%), OCD (4-5%), PTSD (6-8%) and other (2-3%) there was an overall recovery rate of 50% only 20% of the allegedly ‘non-depressed’ clients would recover. This 20% would have to regarded as an upper limit because it assumes the IAPT therapist would be as skilled as the highly trained therapists involved in the rcts for anxiety disorders. A more realistic estimate of recovery for the IAPT ‘anxious clients’ would be the 14.2% found in my study of IAPT clients http://DOI: 10.1177/1359105318755264 https://doi.org/10.1177/135910531875

The other metric employed  by IAPT is the GAD-7, a measure of the severity of anxiety, but as according to IAPT it has only been relevant to one in 10 of its service users, any effect of the treatment of this disorder will only effect the above picture minimally. Assuming a 50% recovery the effect will be even less and less still when one compares the training of therapists in GAD acts with the training of the routine IAPT therapist.

IAPTS sole reliance on psychometric tests and fudge has backfired badly, but it is the client who suffers most, with therapists suffering from the recoil.  

Dr Mike Scott

 

 

 

Number Theatre and Routine Mental Health

the National Institute for Health Research has just published a review of studies of the psychological treatment of Medically Unexplained Symptoms (MUS) https://doi.org/10.3310/hta24460 [Leaviss J, Davis S, Ren S, Hamilton J, Scope A, Booth A, et al. Behavioural modification interventions for medically unexplained symptoms in primary care: systematic reviews and economic evaluation. Health Technol Assess 2020;24(46)] but in all studies the primary outcome measure was an improvement of symptoms on some psychometric test. No categorical measure was used such as no longer suffering from a ‘disorder’ such as fibromyalgia, irritable bowel syndrome or chronic fatigue syndrome post treatment. Likewise the Improving Access to Psychological Treatment (IAPT) markets its success on a change in score on psychometric tests the PHQ-9 and GAD-7. Further whether or not an IAPT clinician is to be subjected to a formal review of competence is based on a change of score on these measures. No categorical measure is used such as the proportion of cases of depression, panic disorder, generalised anxiety disorder etc that have lost their diagnostic status. Sir David Spiegelhalter the Statistician has coined the term ‘number theatre’ to describe the way in which the UK Government has promulgated statistics in relation to the Pandemic, but this drama been playing for years in the mental health arena.  I am reminded of a line from a song somewhere, ‘I am more than a number in a little red book’, although intended for a very different context, it seems particularly apt for IAPT.

Lies
Damned Lies
and
IAPT
 

Number theatre in the mental health field has it seems been driven by the desire of psychologists to colonise. It is a reaction against the categorical labels employed by psychiatry. But the truth of the matter is both are needed simultaneously. To take a medical example, if I have a heart problem I need to know what the problem is but also my blood pressure today.

IAPT will topple because it pivots on psychometric tests. Inspection of of its’ main pillar, the PHQ-9 exposes a crumbling structure:

  1. Client’s judgement of their functioning does not match changes on the PHQ-9 https://doi.org/10.3310/hta24460. Thus an IAPT therapist might report to his supervisor the ‘improvement’ on his/her clients score on the PHQ-9 and at the same time report that the latter said they are ‘the same old’. The overall judgement of the client is likely to be dismissed in favour of the alleged ‘moving towards recovery’ or ‘recovery’ on the PHQ-9.
  2. In the initial validation study of the PHQ-9  by Kroenke and Spitzer it was not validated against a ‘gold standard’ that it was sufficiently different to to make it an acceptable diagnostic aid according to the AMSTAR
  3. The findings of the progenitors of the PHQ-9 Kroenke and Spitzer were not replicated by independent researchers using a ‘gold standard’ diagnostic interview  such as the SCID.
  4. The diagnostic accuracy of an instrument depends very much on the prevalence of the disorder in which it was first evaluated. In the case of the PHQ-9 psychiatric outpatients in the United States. There is no reliable evidence (as assessed by a standardised diagnostic interview)  on the prevalence of disorders amongst those attending IAPT (which include both self referrers and GP referrals).  Thus the clinical utility of the PHQ-9 in this context is unknown.
  5. The PHQ-9 is purportedly a measure of the severity of depression, but there is poor concordance between it and alternative measures of the severity such as the HAD i.e a person would be in a different category of severity depending on which measure is used.

5. The use of a psychometric test with a summary score assumes that each of the items (9 in the case of the PHQ-9) contribute equally to the total score. But this is implausible an item about suicidal ideation (item  9 on the PHQ-9) is likely to  be more significant than an item about fatigue. 

6. Two patients on the PHQ-9 could have the same score, but arising from one patient endorsing all intermediate scores whilst the second endorses several items at the highest score. The same score but arguably a quite different meaning.

7. The PHQ-9 assumes that is the frequency of a symptom  that is the determinant of severity rather than the intensity. 

8. Unless the mechanism by which a PHQ9 score is changed is known it cannot determined that an evidence based treatment was in fact used. Thus those getting a supposed ‘result’ may be more at fault than those acknowledging none response, the latter may simply be more honest. 

These considerations on the PHQ-9 may not be prohibitive of its use, if employed in the context of a standardised diagnostic interview that has established the person has depression. But such an interview would likely also yield the presence of one or more coexisting disorders. The trajectory of these additional disorders would have to be tracked by other psychometric tests that are pertinent to the disorder. The idea that the  PHQ-9 can stand alone as judge and jury on a client’s mental health is absurd.

However politicians, public health bodies and clinical commissioning groups like to be told that there is a simple solution to a problem and that they can make a difference by implementing the chosen solution. Enter stage right IAPT proclaiming ‘give the PHQ9 reduce it below 10, job done and woe betide any clinician who does not manage this routinely’. As an encore IAPT uses numbers e.g throughput of clients, waiting lists to placate politicians and funders.  Exhaustion, numbing and detachment [burnout] are an inevitable consequence of these working conditions. No amount of self-reflection as advocated by Psychological Wellbeing Practitioner in the current issue of CBT Today, is going to make a real world difference. It is a shame that CBT Today has become IAPT’s comic.

Dr Mike Scott

 

 

 

 

The Mismatch Between Clients Global Judgements and Changes on Psychometric Test

casts doubt on the wisdom of the Improving Access to Psychological Therapies IAPT) services sole reliance on test results.  A recent study by Hobbs et al (2020) https://doi.org/10.1017/S0033291719003878 compared subjects global judgements on improvement/deterioration with changes on the PHQ9 and found poor agreement. For those who reported ‘feeling worse’ PHQ9 scores showed no change or improvement for 76% of them.  Further for those who reported ‘feeling a lot worse’  for 81% of them the PHQ9 showed no change or an improvement.  

Hobbs et al (2020) conclude that test results tell only part of the client’s story and there is a need for clinical assessment.

The interview assessments in IAPT go no further than the open ended interviews that are the first part of all standardised diagnostic interviews. Open-ended interviews furnish at most differential diagnoses and highlight candidate disorders for further investigation.  Further inquiry is then made of all of the symptoms in a diagnostic set ( controlling for information variance) and thresholds are used to determine whether a symptom is present  at a level that would constitute functional impairment (controlling for criterion variance).  Reliable diagnosis makes it possible to determine which evidence based treatments are likely to be appropriate in a particular case, bearing in mind the client’s social context and cognitive capacities.

Ultimately global judgements have a more real world feel than changes on psychometric test. A client knows whether they are back to their old selves post treatment and whether they would regard this as enduring. Similarly a client with a lifelong history will now whether they are back to their ‘best functioning’ post treatment and whether it is just one more ‘flash in the pan’. Similarly an independent assessor of a Service can make g global judgement (using a standardised diagnostic interview) whether the person has lost their diagnostic status post treatment and whether there is evidence of permanence. Psychometric tests have been grossly overvalued by IAPT for quality control purposes, but they are fantastic for marketing to the unwary.

Tests Misleading for Diagnostic Acccuracy

It is common for advocates of psychometric tests to quote high sensitivities/ specificities of the order off 80% for instruments such as the PHQ9. But this does not mean that using the PHQ9 on all clients coming through the IAPT door that using its 10 or greater cut off 4 out of 5 clients will be correctly classed.  Tests are validated in a particular context, thus if the proportion of clients with diagnosed disorder (using a diagnostic interview)  differs  by context then so to will the appropriate cut offs and sensitivities/specificities.  Further a psychometric test does not indicate what other disorders are present nor which is the primary disorder. Treatment that fails to address comorbidity is likely to fall short and comorbidity is the norm not the exception. 

Dr Mike Scott

PROMS – Track and Trace for Mental Health Without Knowing What Is Being Tracked

a just published study in the British Medical Journal  https://doi.org/10.1136/bmj.m3313 has found that  ‘There is insufficient evidence and mostly of low quality, that routine monitoring  with PROMS (Patient reported outcome measures) … leads to improvement in outcomes’.  Of the 5 studies reviewed one was of the Improving Access to Psychological Therapies (IAPT) Service in which the PHQ-9 and GAD-7 self report measures were used. 

Strangely the authors of the study Kendrick and Maund (2020) are surprised by the negative findings. It seems not to have occurred to them, that if it is not known with any certainty what the patients were suffering from in the first place then using the most available psychometric test to measure outcome is unlikely to yield any positive findings. In none of the studies was a standardised diagnostic interview used to establish diagnosis and determine any accompanying diagnostic comorbidity.  Thus it cannot be reliably known which is the outcome measure of primary interest, and should becomes the established yardstick before treatment begins and what secondary analyses should be declared in advance. This is akin to the need to pre-register how the results of a randomised controlled trial are going to be analysed rather than going on a post hoc fishing expedition highlighting some positive finding or other to justify a service.

Last Night of The PROMS?

The use of PROMS appears to be fuelled by the need to quickly process patients, using surrogate outcome measures. Rather than taking the time to properly listen to them and use a real world outcome measure such as loss of diagnostic status for say 8 weeks, as assessed by an independent evaluator using a standardised diagnostic interview. Psychometric tests completed for the benefit of a treating clinician are subject to demand characteristics, including wanting to please the therapist and not wanting to feel time has been wasted in engaging in psychological therapy. These concerns are amplified when tests are administered (as in IAPT) on a weekly basis and clients can easily remember their last score.

For all the deficiencies of track and trace over COVID-19,  the target is at least not a ‘fuzzy’ , rendering the process meaningless. Ironically since the demise of Public Health England Baroness Dido Harding is in charge the Covid-19-19 Track and trace. I e-mailed her asking if she was also going to assume responsibility for IAPT but have had no reply. Any QUANGO such as IAPT is likely to rejoice at the absence of accountability but to the detriment of the public. There has to be clarity about exactly who IAPT is accountable to now.

Monitoring Is Necessary But  Never Sufficient 

Just as monitoring the spread of the coronavirus is critical to triggering some preventative measures, it is likely going to be insufficient until there is an evidence based treatment protocol including a vaccine and treatment of the effected. So to only an informed monitoring of mental health problems can highlight appropriate treatment interventions. Monitoring by itself is descriptive rather than prescriptive. Unfortunately there is nothing in the Kendrick and Maund (2020) approach that is likely to make it reliably prescriptive, making their proposed developments in monitoring rather pointless.

Dr Mike Scott