Enhancing Reproducibility through Rigor & Transparency Guidance
January 2016 Update from the NIH
NIH Training Module
In an effort to promote scientific integrity through enhanced reproducibility, the NIH has announced four NEW required components in grant applications. For grant applications due on January 25, 2016 and beyond, applicants must address four areas deemed important for enhancing rigor and transparency: 1) scientific premise, 2) scientific rigor, 3) biological variables, and 4) key biological and/or chemical resources. The information below is meant to help you navigate these required changes.
| Focus Area | Where in the grant will I need to make changes? | What will I need to change? | Where can I find more information about this? |
|---|---|---|---|
| Scientific Premise | PHS 398 Research Strategy/Significance | Section 5.5.3 Demonstrate that critical cited literature in the significance section discusses the strengths and weaknesses of the experimental design within the cited literature (low # sample numbers, age/sex of subjects, etc). | NIH NOT-OD-15-103: Scientific Premise and NIH FAQs |
| Rigorous Experimental Design | PHS398 Research Strategy/Approach | Section 5.5.3 Document inclusion of rigorous experimental design in your approach (sample size, power, confidence, interval, controls, masking, sample handling) | NIH NOT-OD-15-103: Rigorous Experimental Design and NIH FAQs |
| Consideration of Sex and Other Relevant Biological Variables | PHS398 Research Strategy/Approach | Section 5.5.3 Document control within experimental design of biological variables (sex, age, strain, etc) | NIH NOT-OD-15-103: Consideration of Sex and Other Relevant Biological Variables and NIH FAQs |
| Authentication of Key Biological and/or Chemical Resources | PHS398 Other Research Plan Sections/Authentication of Key Resources | Document quality of key reagents, antibodies, cell lines and other (BEI Resources, Critical Reagents Program) | NIH NOT-OD-15-103: Authentication of Key Biological and/or Chemical Resources and NIH FAQs |
More Details on the 4 New Focus Areas: as found in NIH NOT-OD-15-103
Scientific Premise
All research builds upon prior research, which may include observations, preliminary data, or published literature. The scientific premise for an application is the research that is used to form the basis for the proposed research question; NIH has always strived to fund projects that are based on a strong foundation. Moving forward, NIH expects applicants to describe the general strengths and weaknesses of the prior research being cited by the investigator as crucial to support the application. It is expected that this consideration of general strengths and weaknesses could include attention to the rigor of the previous experimental designs, as well as the incorporation of relevant biological variables and authentication of key resources. For example, basing one’s proposed research on previous publications that lacked statistical power, were not blinded, lacked detail on the sex of animals or authentication of cell lines would be considered a weakness of the application if it does not identify these weaknesses and propose ways to improve going forward. Likewise, conclusions drawn from prior research that used a small sample size may not adequately support the next phase of research, such as moving to a higher species of animals or to humans. See Landis, et al., 2012 (http://www.nature.com/nature/journal/v490/n7419/full/nature11556.html) for a description of the value of scientific premise and core reporting standards important for evaluating the premise.
It is important to note that some funding opportunities have required an evaluation of the scientific premise in applications for some time. The new application instructions and revised review criteria will ensure that all areas of research emphasize the principle that meritorious science is built on a rigorous foundation.Rigorous Experimental Design
Scientific rigor is the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results. This includes full transparency in reporting experimental details so that others may reproduce and extend the findings.
NIH expects applicants to describe how they will achieve robust and unbiased results when describing the experimental design and proposed methods. Robust results are obtained using methods designed to avoid bias and can be reproduced under well-controlled and reported experimental conditions.
A number of NIH Institutes have published guidelines for reporting elements of rigor in experimental design to be included in applications considered for funding, namely: National Institute of Neurological Disorders and Stroke (NOT-NS-11-023), National Institute of Mental Health (NOT-MH-14-004) and National Institute on Drug Abuse (NOT-DA-14-007). Similar guidelines have been adopted by journals for preclinical research across many disciplines; see http://www.nih.gov/about/reporting-preclinical-research.htm and http://www.nih.gov/about/endorsing-jounals.htm. Many of these principles can be adopted or adapted in all areas of biomedical research.
Need additional training in data analysis? CSU is now offering a Certificate in Data Analysis.
Consideration of Sex and Other Relevant Biological Variables
Biological variables, such as sex, age, weight, and underlying health conditions, are often critical factors affecting health or disease. In particular, sex is a biological variable that is frequently ignored in animal study designs and analyses, leading to an incomplete understanding of potential sex-based differences in basic biological function, disease processes and treatment response. NIH expects that sex as a biological variable will be factored into research designs, analyses, and reporting in vertebrate animal and human studies. Strong justification from the scientific literature, preliminary data or other relevant considerations must be provided for applications proposing to study only one sex. Please refer to NOT-OD-15-102 for further consideration of NIH expectations about sex as a biological variable.
Similarly, investigators should consider other biological variables, as appropriate, in the design and analyses of their proposed studies. Research plans and findings should clearly indicate which biological variables are tested or controlled. Clear justification should be provided for exclusion of variables that may be relevant but are not considered in the research plan. For example, studies using young adult animals should clearly describe their study population and not generalize findings to juvenile or aged animals.
Authentication of Key Biological and/or Chemical Resources
The quality of the resources used to conduct research is critical to the ability to reproduce the results. NIH expects that key biological and/or chemical resources will be regularly authenticated to ensure their identity and validity for use in the proposed studies. Key biological and/or chemical resources may or may not be generated with NIH funds and: 1) may differ from laboratory to laboratory or over time; 2) may have qualities and/or qualifications that could influence the research data; and 3) are integral to the proposed research. These include, but are not limited to, cell lines, specialty chemicals, antibodies and other biologics.
There are not clear consensus guidelines on authenticating different types of resources, and NIH encourages the research community to inform the development of such guidelines. In the absence of clear guidelines, researchers should transparently report on what they have done to authenticate key resources, so that consensus can emerge.