How should systematic reviews consider evidence on harms?

Systematic reviews that attempt to assess the risk of harms (adverse effects) associated with specific therapies should consider a broad range of study designs, including both systematic reviews and observational studies. These are the findings of a new study, led by Su Golder of the Centre for Reviews and Dissemination, University of York, UK published in this week's PLoS Medicine.

There is increasing focus on the importance of using rigorous methods to assess the effectiveness and harms associated with the use of new drugs and other therapies, and recognition of the role of systematic reviews in this process. A systematic review uses predefined, explicit methods to find and appraise all relevant evidence to answer a specific question in healthcare. However, there has been considerable debate as to whether systematic reviews should use evidence from randomized controlled trials or observational studies (or both) in order to collect all the relevant evidence on risk of harms. Some groups have argued that observational studies may produce biased estimates of harm, while randomized trials may be too small to generate useful data on the risk of rare adverse effects.

In the study, Golder and colleagues identified systematic reviews that had compared the risk of specific harms in evidence from randomized controlled trials versus the evidence from observational studies. They found that there was no difference on average in the estimates produced by these two approaches. The authors conclude: "Instead of restricting the analysis to certain study designs, it may be preferable for systematic reviewers of adverse effects to evaluate a broad range of studies that can help build a complete picture of any potential harm and improve the generalisability of the review without loss of validity."