Sometimes even a 98% success rate isn't enough
Lies, damned lies, and statistics*
The following is offered as an illustration of the sometimes dubious power of numbers in investigative psychology, particularly those relating to ‘success rates’ in profiling.
A while back I came across a commercial company offering psychological profiles of offenders. The main target audience for this service was the corporate sector. The profiles were exactly what you would expect, some pseudoscientific psychobabble describing the personality characteristics of offenders (which is always useful in identifying offenders from amongst the wider population). The profilers themselves were also exactly what you’d expect, a mix of ex-detectives and ex-academics. What made the company interesting was that they claimed their success rate in identifying offenders was about 98%, which really does sound very impressive.
Except it’s not. Let me explain.
Let’s imagine that there are 100 suspects to be profiled in an investigation and we have been contracted to identify the offender. We administer our patented profiling system to those 100 suspects. Note, it really doesn’t matter what we do here, as long as it looks suitably exciting enough: We must maintain appearances otherwise our very large invoice won’t get paid.
Let’s now imagine that the offender goes by the name Davo. Davo is a very bad and unpleasant type, so we really want to identify him as the offender. Except that’s not who we identify as the offender, instead, we identify Shane. Shane is completely innocent, but we out him as the offender anyway (regarding the names, don’t worry, it’s an Australian thing).
We then hand over our report identifying Shane and our suitably big invoice.
Job done, we now sit and wait.
Short term: Shane gets fired/divorced/arrested etc., Davo pretty much carries on as before.
Long term: Eventually it comes out that that the real offender was Davo, meaning Shane was innocent all along.
Clearly we got our profiles of Davo and Shane wrong (no, you can’t have your money back) but all in all, this was another good day at the office and our 98% success rate was maintained.
Confused? You should be.
How can we claim a 98% success rate when we falsely accused one person (poor Shane) and falsely exonerated another (the awful Davo)?
It’s actually quite easy. Sure, we got the profiles of Shane and Davo wrong, but you forget that we accurately profiled the other 98 suspects as being innocent…...
And so there you have it, our success rate in this case was 98%. Yes, you can go have a lay down now and let that one sink in.
This was exactly the logic used by the commercial company I referred to at the outset of this Blog to justify their claims to having a 98% success rate.
It’s worth bearing these statistical contortions in mind when reading about the apparent “success rates” of profilers, lie detection experts, etc.
Sometimes, even a 99% success rate isn’t good enough.
* Mark Twain, possibly…. probably.
Note: This Blog above was based on the ideas first proposed in Moston, S., & Engelberg, T. (2015). Science 1, Religion 5: A reply to Petróczi et al. (2015) ‘A call for policy guidance on psychometric testing in doping control in sport’. International Journal of Drug Policy, 26(11), 1140-1141. https://doi.org/10.1016/j.drugpo.2015.05.013
I’d love to hear what you think about this post. Do you agree with me? Do you disagree? You can write to me - firstname.lastname@example.org - and let me know what you think.
If you’ve enjoyed this Blog post, I invite you to check out www.Forensii.com. Forensii.com is the essential resource for anyone at any level of interviewing skill.