‘Psychometric Tests, Administered In Isolation, Are Not Footprints of Anything’ – IAPT’s Big Mistake

IAPT uses psychometric tests to identify ‘cases’ and changes in test score to gauge effectiveness.  This is not an evidence based assessment and without it there can be no evidence based treatment.

Image result for wrong direction

A psychometric test can’t exist in a vacuum it has to refer to something tangible i.e it must have criterion related validity. For example in last month’s British Journal of Psychiatry, Quinlivan et al [‘Predictive accuracy of risk scales following self-harm’] assessed the ability of risk scales to predict whether a person will make a further suicide attempt  (the criterion). It was found that the much used scales, did not in fact predict self-harm, i.e they lacked criterion validity. Thus when psychometric tests such as the PHQ-9 (an intended measure of depression) and GAD-7 (an intended measure of generalised anxiety disorder) are used, individual test results are only meaningful if they are actually the ‘footprint’ of the construct under examination. Imagine seeing a footprint in the snow:

 

 

does it relate to the abominable snowman, a polar bear, a human being or the great yeti? Without a specification of what it refers to changes in the footprint found are meaningless.  Thus when IAPT use the PHQ-9 and GAD-7 in isolation it is not known to what they refer, as no reliable diagnostic interview has been performed. Is the person simply stressed, depressed, worried well or what? The myriad possibilities likely have very different trajectories e.g the stressed improving as the stressor passes. Lumping them altogether, creates confusion, prevents any evidence based assessment, which is the foundation for evidence based treatment. Clients cannot be reliably signposted to anything, resulting in the wrong tools being used:

Worryingly, I wrote a rejoinder to a paper by Ali et al in this month’s  Behavior Research and Therapy, on relapse after IAPT low intensity intervention, making the point that they had abused psychometric test results in just this way, it was rejected, the reviewers pointed out that I hadn’t included a reference supporting criterion related validity!  I despair. The reviewers tried to justify the approach of Ali et al on the grounds that the PHQ-9 is a reliable instrument, identifying 80% of those who are depressed (sensitivity) and 80% of those who are not depressed (specificity), which is true. But this provides no basis on which to judge whether Mr X who scored say 25 on the PHQ-9 should a) be regarded as a ‘case’ of depression and relatedly b) whether his progress should be charted with this measure, a) and b) can only be determined by a reliable standardised diagnostic interview, which is absent from the IAPT assessment protocol. If you found your electrician was measuring current with a voltmeter you would, forgive the pun be ‘shocked’, we need to create a similar state of alarm about the quality of audit in IAPT. There is a pressing need for independent rigorous assessment.

Dr Mike Scott

The Silencing of Dissent and IAPT

 

This month’s Behavior Research and Therapy features a paper by Ali et al in which IAPT data on relapse after low intensity (Li) interventions is reviewed, and it is concluded that further attention to relapse prevention may be needed. I submitted a rejoinder essentially saying that Li-interventions have been a false economy and complaining that it had not been declared that Ali headed the Northern IAPT Research network, but it was rejected.

The editor began her letter of explanation with ‘Each of the reviewers is a highly experienced researcher in the area of low-intensity treatments for depression anxiety’.  But that is precisely the problem, researchers in low intensity see no pressing need for independent assessment using a ‘gold standard’ diagnostic interview (unlike their forbearers who conducted the bench-marking studies that gave CBT its’ credibility), although they pay lip service to it.

In practice, low intensity researchers find it ‘reasonable’ to conduct research on outcome solely on the basis of changes in a psychometric test. This strategy enables research to be done on the cheap, produce lots of papers and get brownie points in academia. There is a mutually beneficial groupthink amongst low intensity researchers and the IAPT hierarchy. Low intensity interventions fail an evidence based assessment test with a shameful lowering of the bar of methodological rigour. I will return in future blogs to editors/reviewers scant regard for criterion related validity and the misuse of Jacobsen’s Reliable and Clinically Significant Change Index, an abuse that is rampant in IAPT.

Dr Mike Scott

Bias in CBT Journals

When the organs of communication are controlled by a single ideology we are on a short road to hell. Recently I protested to the Editor of Behavior Research and Therapy (BRAT), that no conflict of interest had been declared in a paper authored by Ali et al published in this month’s issue of the Journal, focusing on IAPT data on relapse after low intensity interventions. I pointed out that the lead author headed the Northern IAPT research network, not only did the editor ignore the conflict of interest but so to did the two reviewers, of a rejoinder to the paper that I wrote. But it is not just BRAT, IAPT sponsored papers regularly appear in Behavioural and Cognitive Psychotherapy without declarations of conflicts of interest.  I have protested to the editor about this, but again to no avail. Unfortunately it is not just a matter of what Editors of CBT Journals allow through the ‘Nothing to Declare’ aisle but also their blocking of objections to the current zeitgeist that is a cause for concern. More about this anon.

Dr Mike Scott

A Screen for Mental Health – The First Step Questionnaire Revised

Clients often do not improve, not because of a lack of therapeutic skill, but because of something else  going on that they never thought to ask about. Screening clients for all common disorders is a protection against missing an important therapeutic target. The First Step Questionnaire published in Towards a Mental Health System that Works Scott (2017) London: Routledge, is such a screen, covering all the common disorders and importantly asking clients whether or not they want help with a particular difficulty, but also with a ‘don’t know’ option, so that ambivalence can be recognised from the outset. There is also an interview version the 7 Minute Interview. [ The validity studies on the Questionnaire/Interview are considered  in the Simply Effective trilogy Scott (2009), (2011) and (2013).]  I have now revised the Questionnaire/ Interview to take into account the changed diagnostic criteria for PTSD in DSM-5 and added a screen for borderline personality disorder (BPD)

The symptom questions of the PTSD screen are from the Primary Care PTSD Checklist for DSM-5,   from the US National Centre for PTSD, a positive response to 3 or more symptom questions is a positive screen for PTSD.  The  BPD screen is based on a paper by Zimmerman et al (2017) Clinically useful screen for borderline personality disorder in psychiatric outpatients, British Journal of Psychiatry, 210, 165-166. Of those with BPD over 90% endosed the affective instability question in item 11 below, but only 38% of those with affective instability had BPD i.e most of those with affective instability don’t have BPD. This illustrates that screening questions are only ever a starting point, if you don’t ask further clarifying questions in terms of the full DSM-5 criteria they can be very misleading. [Adding the anger item, see item 11 to the BPD screen meant that 97% of those with BPD answered ‘yes’ two both symptom questions according to Zimmerman et al (2017)].  It remains to be seen how much the question about wanting help adds to diagnostic accuracy, it is known that it does so for the depression screen.

 

Name:                                                                                      Date:

 

D.o.b:

 

The First Step Questionnaire – Revised

This questionnaire is a first step in identifying what you might be suffering from and pointing you in the right direction. In answering each question just make your best guess; don’t think about your response too much, there are no right or wrong answers.

 

1. Yes No Don’t know
During the past month have you often been bothered by feeling, depressed or hopeless?
During the past month have you often been bothered by little interest or pleasure in doing things?
Is this something with which you would like help?

 

 

2. Yes No Don’t know
Do you have unexpected panic attacks, a sudden rush of intense fear or anxiety?
Do you avoid situations in which the panic attacks might occur?
Is this something with which you would like help?

 

 

3.

In your life, have you ever had any experience that was so frightening, horrible or upsetting that, in the past month, you

Yes No Don’t know
i. Have had nightmares about it or thought about it when you did not want to?
ii. Tried hard not to think about it or went out of your way to avoid situations that reminded you of it?
iii. Were constantly on guard, watchful, or easily startled?
iv. Felt numb or detached from others, activities, or your surroundings?
v.  Felt guilty or unable to stop blaming yourself or others for the event(s) or any problems the event(s) may have caused?
Is this something with which you would like help?

 

4. Yes No Don’t know
Are you a worrier?
Do you worry about everything?
Has the worrying been excessive (more days than not) or uncontrollable in the last 6 months?
Is this something with which you would like help?

 

 

5. Yes No Don’t know
When you are or might be in the spotlight say in a group of people or eating/writing in front of others do you immediately get anxious or nervous
Do you avoid social situations out of a fear of embarrassing or humiliating yourself?
Is this something with which you would like help?

 

 

6. Obsessive Compulsive Disorder Yes No Don’t know
Do you wash or clean a lot?
Do you check things a lot
Is there any thought that keeps bothering you that you would like to get rid of but can’t?
Do your daily activities take a long time to finish?
Are you concerned about orderliness or symmetry?
Is this something with which you would like help?

 

7. Yes No Don’t know
Do you go on binges were you eat very large amounts of food in a short period?
Do you do anything special, such as vomitting, go on a strict diet to prevent gaining weight from the binge?
Is this something with which you would like help?

 

 

8. Yes No Don’t know
Have you felt you should cut down on your alcohol/drug?
Have people got annoyed with you about your drinking/drug taking?
Have you felt guilty about your drinking/drug use?
Do you drink/use drugs before midday?
Is this something with which you would like help?

 

9. Yes No Don’t know
Do you ever hear things other people don’t hear, or see things they don’t see?
Do you ever feel like someone is spying on you or plotting to hurt you?
Do you have any ideas that you don’t like to talk about because you are afraid other people will think you are crazy?
Is this something with which you would like help?

 

 

10. Yes No Don’t know
Have there been times, lasting at least a few days when you were unusually high, talking a lot, sleeping little?
Did others notice that there was something different about you?

If you answered ‘yes’, what did they say?

 

Is this something with which you would like help?

 

11. Yes No Don’t know
Do you have a lot of sudden changes of mood, usually lasting for no more than a few hours?
Do you often have temper outbursts or get so angry you lose control?
Is this something with which you would like help?

 

‘How Do I Deliver Effective CBT Where I am?’

The contexts in which CBT Practitioners work vary enormously, from independent practice to secondary care, from low intensity IAPT to a specialised trauma unit.  For the most part we are Engineers struggling to work within the organisational constraints we are given.  Drawing on our knowledge and skills, working with a diverse population, trying to make a real world, socially significant difference in client’s lives.  The pressing question is how can I deliver effective CBT where I am?

A practitioner working in secondary care in Ireland,  told me he faces the challenge of cases come to him via psychiatrists, there is a preliminary assessment within 4 weeks of referral, a maximum of 12 sessions of CBT are offered.  There is a progress review about the 6th session and a decision is made as to whether another 6 sessions would be beneficial. He asks is this best practice? The managerial edict he believes is to throughput as many clients as possible.

Another practitioner, from IAPT High Intensity told me that she had taken up her post on the understanding that the 6 session maximum was flexible and clients could be quickly re-referred back in for more sessions, but this has proved to be very rarely the case. What should she do?

 

Dr Mike Scott