The challenge of proving a negative

A quick comment on the challenge of proving a negative as related to the effectiveness of interventions (if it is even possible is debatable). But lets look past proving, and shoot for knowledge as justifiable true belief (still difficult).

These thoughts have emerged after months of reading through and incorporating the ”Minimum Required Skills” for a PT graduate (from 2004, revised in 2009) and wondering when those with justifiable evidence that they do not work should be removed?

I understand completely the need to keep those without justifiable evidence that they work if there is no justifiable evidence that they do not work. You have likely heard: a lack of evidence for effectiveness is not the same as evidence of a lack of effectiveness.

Then I realized - what is “justifiable evidence” or “knowledge” (justified true belief) that an intervention does not work. This is trying to prove a negative. Very difficult indeed.

Are we destined to continue to teach such skills? If so, what are the implications for the profession? Health care? Society?

Do we need something beyond the hierarchy of evidence for adjudicating such decisions?

As a quick example for those wondering why it is difficult to prove a negative. If I said that a “black swan” exists, I can put that hypothesis forward and continue searching. All I need to find is a single example that a black swan exists and I have proven the affirmative - a black swan exists. But how much searching do I need to do to prove that a black swan does not exist? Are we ever satisfied with the search for evidence, or might there be one around the corner that we have not looked? Or perhaps - like standing cows in the classic Far Side cartoon  - they disguise themselves when observed.

As a specific example, let’s take Continuous Passive Motion. There is now a large review of years of evidence that support the claim that it’s use is not justified (see here). But, are we comfortable yet in throwing it aside and not recommending it’s use - or do we find reasons why a particular person might benefit from it beyond the mean of several large samples and relying on the central limit theorem for a homogenized estimate of effect?

In other words - does this review on CPM warrant justified true belief (knowledge) of the negative - that CPM does not work? Or does the “There is weak evidence that continuous passive motion reduces the subsequent need for manipulation under anaesthesia.” continue to raise questions about better approaches to patient examination and evaluation for identifying those that will benefit from it’s use?

When we break this down to  the use of logical (material) implication it may seem possible to prove the negative “If - Then” implication. After all, with material implication, the implication is “False” whenever the antecedent (“If” condition) is true and the consequent (“Then” condition) is false. And for material implication that is the only time the If - Then statement is proven false (true antecedent and false consequent). However, things get muddy when attempting to justify a universal based on this approach. It is easy to justify a particular situation. If the cause occurred and the effect did not occur, you are justified to say that the cause did not cause the effect (the effect did not even occur) in a particular situation. But the challenge is whether this is a universal truth - and here we go from logical inference to statistical inference.

When considering this from the perspective of statistical inference, and attempting to justify a universal causal association we must consider the complexity of the system under study. In reality, none of the connections between the cause and effect we consider this simple:

dagitty-model.png

And they likely are not even this simple - but with this structure there are questions raised about what we need to know in order to understand the causal structure enough to know when the Cause will result in the Effect, such as whether the other variables in the path are possible (X1, X2, Y for example).

dagitty-model (1).png

With this causal structure the variability in the effect being measured for the use of CPM (Cause) is impacted by  X1 and X2 and Y,and a minimum their existence, perhaps even their nature or own causal structure. In which case we do not want to close the door on Cause -> Effect until we understand that structure and have studied the system effectively.

Ok, that is enough laps around the track for this morning. I realize I have come full circle -and still have not decided, after months of pontificating, whether anything on this list can be removed due to justifiable evidence of the negative.

Leave a Comment