Assessing the Validity of Statistical Inferences in Public Health Research: An Evidence-Based, ‘Best Practices’ Approach

Abstract

Like many fields, public health has embraced the process of evidence-based practice to inform practice decisions and to guide policy development. Evidence-based practice is typically dependent upon generalizations made on the bases of the existing body of knowledge – assimilations of the research literature on a particular topic. The potential utility of scientific evidence for guiding policy and practice decisions is grounded in the validity of the research investigations upon which such decisions are made. However, the validity of inferences made from the extant public health research literature requires more than ascertaining the validity of the statistical methods alone; for each study, the validity of the entire research process must be critically analyzed to the greatest extent possible so that appropriate conclusions can be drawn, and that recommendations for development of sound public health policy and practice can be offered. A critical analysis of the research process should include the following: An a priori commitment to the research question; endpoints that are both appropriate for and consistent with the research question; an experimental design that is appropriate (i.e., that answers the research question[s]); study procedures that are conducted in a quality manner, that eliminate bias and ensure that the data accurately reflect the condition(s) under study; evidence that the integrity of the Type-I error – or false-positive risk – has been preserved; use of appropriate statistical methods (e.g. assumptions checked, dropouts appropriately handled, correct variance term) for the data analyzed; and accurate interpretation of the results of statistical tests conducted in the study (e.g., the robustness of conclusions relative to missing data, multiple endpoints, multiple analyses, conditions of study, generalization of results, etc.). This paper provides a framework for both researcher and practitioner so that each may assess this critical aspect of public health research.