“Significantly………”

We are entering a period on the blog where I have now completed an explanation of the title: Cause, Models and Inference and how each component contributes to a knowledge based practice (Note, in the migration from the original blog to the current platform the posts that established how each component contributes to a knowledge based practice have been pulled out into a collection called Collections). The next step is for me to try to fully explain why critical realism over empiricism as a foundation for the clinical epistemology, and what the implications are of recognizing that foundation. But this will take me some time - particularly with the semester starting soon and CSM in just a few weeks.

In the meantime I will attempt to post on topics that are helpful to the development, but not specifically developmental.

Yesterday I got the updated set of newly “online first” articles for PTJ. This article caught my attention: Cardiovascular Stress Induced by Whole-Body Vibration Exercise in Individuals With Chronic Stroke

Just looking at the abstract the following sentence jumped out at me and resonates as I am preparing my notes for my upcoming lectures on statistical inference (statistical inference uses probability and statistics to go from data to quantify propositions for consideration during induction as “general premises”) in research methods for the spring semester.

From the results: “Low-intensity and high-intensity WBV induced significantly higher VO2 by an average of 0.69 and 0.79ml/kg/min respectively (P≤0.001) than the control condition.” (Emphasis added)

From the conclusion: “Addition of high- and low-intensity WBV significantly increased the VO2 and HR, but the increase was modest. WBV thus should not pose any substantial cardiovascular hazard in people with chronic stroke.”(Emphasis added)

The purpose of the study is mixed between demonstrating the demand is not so large as to pose a CV risk to patients with stroke versus demonstrating the demand is enough to serve as a “useful adjunct treatment for cardiovascular exercise training.” Note, if the protocol was not registered ahead of time we do not know whether both of these purposes were actually part of the study design, or whether one was added later to accommodate the findings.

Let’s isolate our thoughts for a moment on the phraseology of the results and subsequent conclusion in the abstract. It highlights one problem people have had with statistical significance, keep in mind it is “JUST” statistical significance, and with using it for hypothesis testing, it is just testing against one hypothesis (the null).

The p value is the conditional probability of the data given the null hypothesis and all other assumptions: p(datanull hypothesis and all other assumptions); if the p value is less than the accepted alpha (alpha = P(reject null hypothesisnull hypothesis is true and all other assumptions hold)) then we say there is statistical significance.

In the results and the conclusion the word significantly needs to be specifically identified as “statistically significantly” to avoid giving an interpretation that has hidden meaning (the hidden meaning being the that “effect size” is meaningful, has clinical or practical significance. This study set an alpha of 0.05 and adjusted based on the number of tests performed (which is great), and yes, the VO2 results are statistically significant. As they point out in their conclusion they are modest. To say that are modest may even be overstating a bit. What they are packaging with the word “significant” and then only modifying with the adjective “modest” is a change that amounts to about 0.2 METS. Since this is not a large enough “effect size” to make WBV a “useful adjunct treatment for cardiovascular exercise training” the conclusion focused on whether the demand is enough to pose significant risks. But keep in mind that hypothesis was not tested. The hypothesis tested by this statistical test was the conditional probability of the data given the null hypothesis is true, the null hypothesis (unless otherwise stated) is that there is no effect (no difference, no change, no association). A perfectly reasonable hypothesis given their overall methodology.

Whether the study uses the right methodology for concluding “WBV thus should not pose any substantial cardiovascular hazard in people with chronic stroke” requires a different set considerations for the person interpreting this study. I am not saying whether this study meets or does not meet such considerations, I simply want to point out how the language of statistical significance is used. You cannot just look for “statistically significant” p values and then assume that the results are correct, or that the author’s conclusions are appropriate. Just reading an abstract, in particular, can give the wrong impression about the underlying meaning of data when looked at in isolation (so for my research methods students, you need to keep in mind that you cannot just read the abstract…….).

Leave a Comment