Back to Annual Meeting Posters
The Quality of Hand and Elbow Related Randomized Controlled Trials: A Critical Analysis of Current Literature
Jaehon Kim, MD1; Ryan M. Zimmerman2; Christopher M. Jones1; Norman H. Dubin1; James P. Higgins1; Kenneth R. Means1
1Curtis National Hand Center, Baltimore, MD; 2Massachusetts General Hospital, Boston, MA
Hypothesis: Proper research design, data analysis and unbiased reporting are increasingly emphasized in modern orthopaedic literature. The purpose of our study was to determine the quality of current hand and elbow randomized controlled trials (RCTs) using standardized metrics, and identify deficiencies.
Methods: We selected the six most frequently cited journals that regularly publish hand and elbow related manuscripts based on five-year average impact factors from the 2011 Journal Citation Reports. Using journal-specific search query, we identified and screened 2114 articles. A total of 63 RCTs met the inclusion criteria for analysis. Two authors were blinded and randomly assigned to each paper. The reviewers used the Modified Coleman Methodology Score (MCMS) and Jadad scale (5-point validated quality measure) to assess manuscript quality and the Consolidated Standards for Reporting of Trials (CONSORT) statement to assess the completeness of reporting. We compared study characteristics and methodology variables with the manuscript quality using Fisher exact test and two-tailed Student t-tests in univariate analysis. Pearson coefficient (R) determined the strength of correlation between the number of citations and the quality of the studies.
Results: There was a strong correlation between Jadad scale and MCMS (R=0.759, p<0.001), and between CONSORT statement completeness and MCMS (R=0.808, p<0.001). Based on MCMS, ten studies were good, 22 fair, and 31 poor. None were graded excellent. RCT quality has significantly improved between 2001-2006 and 2006-2012 (p=0.030), and the top journal based on impact factor had a higher number of good quality RCTs (p<0.001). Important methodological deficiencies in poorly scoring RCTs include lack of power analysis (p<0.001), lack of withdrawal and drop out description (p=0.011), no blinding (p=0.027) and failure to use validated outcomes assessments or an independent investigator (p=0.021). Individual study quality was not associated with geography, funding, conflict of interest, or multicenter trials. Among studies published more than 3 years ago the quality of RCTs did not correlate with the number of citations (r=0.138, p=0.390), although this may be underpowered.
1) Despite an overall improvement in RCTs over the last decade, a large number of studies were of poor quality based on MCMS and Jadad scale.
2) Even with high level of evidence, the study design and execution of RCTs should be critically assessed.
3) Common methodological deficiencies include lack of power analysis, lack of withdrawal and drop-out description, and failure to use validated outcomes assessments. These deficiencies may introduce bias and lead to statistically under-powered studies.
Back to Annual Meeting Posters