Title
A protocol for a systematic review on the impact of unpublished studies and studies published in the gray literature in meta-analyses

Authors
Christine Schmucker, Annette Bluemle, Matthias Briel, Susan Portalupi, Britta Lang, Edith Motschall, Guido Schwarzer,Dirk Bassler, Katharina F Mueller, Erik von Elm and Joerg J Meerpohl

Background 
Meta-analyses are particularly vulnerable to the effects of publication bias. Despite methodologists' best efforts to locate all evidence for a given topic the most comprehensive searches are likely to miss unpublished studies and studies that are published in the gray literature only. If the results of the missing studies differ systematically from the published ones, a meta-analysis will be biased with an inaccurate assessment of the intervention's effects.

As part of the OPEN project (www.open-project.eu) we will conduct a systematic review with the following objectives:

• To assess the impact of studies that are not published or published in the gray literature on pooled effect estimates in meta-analyses (quantitative measure)
• To assess whether the inclusion of unpublished studies or studies published in the gray literature leads to different conclusions in meta-analyses (qualitative measure) 

Background

The pharmaceutical and biotechnology industries depend on findings from academic investigators prior to initiating programs to develop new diagnostic and therapeutic agents to benefit cancer patients. The success of these programs depends on the validity of published findings. This validity, represented by the reproducibility of published findings, has come into question recently as investigators from companies have raised the issue of poor reproducibility of published results from academic laboratories. Furthermore, retraction rates in high impact journals are climbing.

Methods and Findings

To examine a microcosm of the academic experience with data reproducibility, we surveyed the faculty and trainees at MD Anderson Cancer Center using an anonymous computerized questionnaire; we sought to ascertain the frequency and potential causes of non-reproducible data. We found that ~50% of respondents had experienced at least one episode of the inability to reproduce published data; many who pursued this issue with the original authors were never able to identify the reason for the lack of reproducibility; some were even met with a less than “collegial” interaction.

Conclusions

These results suggest that the problem of data reproducibility is real. Biomedical science needs to establish processes to decrease the problem and adjudicate discrepancies in findings when they are discovered.

[via Retraction Watch, which also offers some commentary on the study]

The video's worth watching, but you can get the gist from these studies:

1. Seeing the fruits of our labor may make us more productive.

The Study: In a study conducted at Harvard University, Ariely asked participants to build characters from Lego’s Bionicles series. In both conditions, participants were paid decreasing amounts for each subsequent Bionicle: $3 for the first one, $2.70 for the next one, and so on. But while one group’s creations were stored under the table, to be disassembled at the end of the experiment, the other group’s Bionicles were disassembled as soon as they’d been built. “This was an endless cycle of them building and we destroying in front of their eyes,” Ariely says.

The Results: The first group made 11 Bionicles, on average, while the second group made only seven before they quit.

The Upshot: Even though there wasn’t huge meaning at stake, and even though the first group knew their work would be destroyed at the end of the experiment, seeing the results of their labor for even a short time was enough to dramatically improve performance.

 

2. Knowing that our work helps others may increase our unconscious motivation

The Study: As described in a recent New York Times Magazine profile, psychologist Adam Grant led a study at a University of Michigan fundraising call center in which  student who had benefited from the center’s scholarship fundraising efforts spoke to the callers for 10 minutes

The Results: A month later, the callers were spending 142 percent more time on the phone than before, and revenues had increased by 171 percent, according to the Times. But the callers denied the scholarship students’ visit had impacted them.

The Upshot: “It was almost as if the good feelings had bypassed the callers’ conscious cognitive processes and gone straight to a more subconscious source of motivation,” the Times reports. “They were more driven to succeed, even if they could not pinpoint the trigger for that drive.”

 

3. The harder a project is, the prouder we feel of it

The Study: In another study, Ariely gave origami novices paper and instructions to build a (pretty ugly) form. Those who did the origami project, as well as bystanders, were asked at the end how much they’d pay for the product. In a second trial, Ariely hid the instructions from some participants, resulting in a harder process — and an uglier product.

The Results: In the first experiment, the builders paid five times as much as those who just evaluated the product. In the second experiment, the lack of instructions exaggerated this difference: builders valued the ugly-but-difficult products even more highly than the easier, prettier ones, while observers valued them even less.

The Upshot: Our valuation of our own work is directly tied to the effort we’ve expended. (Plus, we erroneously think that other people will ascribe the same value to our own work as we do.)

 

4. The less appreciated we feel our work is, the more money we want to do it

The Study: Ariely gave study participants — students at MIT — a piece of paper filled with random letters, and asked them to find pairs of identical letters. Each round, they were offered less money than the previous round. People in the first group wrote their names on their sheets and handed them to the experimenter, who looked it over and said “Uh huh” before putting it in a pile. People in the second group didn’t write down their names, and the experimenter put their sheets in a pile without looking at them. People in the third group had their work shredded immediately upon completion.

The Results: People whose work was shredded needed twice as much money as those whose work was acknowledged in order to keep doing the task. People in the second group, whose work was saved but ignored, needed almost as much money as people whose work was shredded.

The Upshot: “Ignoring the performance of people is almost as bad as shredding their effort before their eyes,” Ariely says. “The good news is that adding motivation doesn’t seem to be so difficult. The bad news is that eliminating motivation seems to be incredibly easy, and if we don’t think about it carefully, we might overdo it.”

There are a few more studies referenced in the story. 

[via Stephen's Lighthouse]

“Dissatisfaction is increasing regardless of practice type or EHR system,” said Michael S. Barr, MD, MBA, FACP, who leads ACP's Medical Practice, Professionalism & Quality division. “These findings highlight the need for the Meaningful Use program and EHR manufacturers to focus on improving EHR features and usability to help reduce inefficient work flows, improve error rates and patient care, and for practices to recognize the importance of ongoing training at all stages of EHR adoption.”

The findings are from 4,279 responses to multiple surveys developed and analyzed by ACP and AmericanEHR Partners between March 2010 and December 2012. Of the clinicians who responded to the surveys, 71 percent were in practices of 10 physicians or less and 82 percent of respondents intend to participate in Meaningful Use incentive programs, up from 65 percent in 2010.

More.

Authors: MacEachern MP, Townsend W, Shannon C, Mani NS, Zeylikovich I, Mangrulkar RS. 

Conference: Central Group on Educational Affairs (CGEA) 2013 Spring Meeting, Cincinnati, OH. 

References

1. MacEachern M, Townsend W, Young K, Rana G. (2012). Librarian Integration in a Four-Year Medical School Curriculum: A Timeline. Medical Reference Services Quarterly, 31(1), 105-14.

2. Aronoff SC, Evans B, Fleece D, Lyons P, Kaplan L, Rojas R. (2010). Integrating Evidence Based Medicine Into Undergraduate Medical Education: Combining Online Instruction with Clinical Clerkships. Teaching and Learning in Medicine, 22(3), 219-223.

3. Bernardo V, Ramos MP, Plapler H, de Figueiredo LF, Nader HB, Ancao MS, von Dietrich CP, Sigulem D. (2004). Web-based Learning in Undergraduate Medical Education: Development and Assessment of an Online Course on Experimental Surgery. International Journal of Medical Informatics, 73, 731-742.

4. Chao SH, Brett B, Wiecha JM, Norton LE, Levine SA. (2012). Use of Online Curriculum to Teach Delirium to Fourth-Year Medical Students: A Comparison with Lecture Format. Journal of the American Geriatrics Society, 60(7). 1328-1332.

5. Masters K, Gibbs T. (2007). The Spiral Curriculum: Implications for Online Learning. BMC Medical Education, 7:52.

6. Stoddard, HA, Pamies RJ, Carver DS, Todd GL. (2008). Developing an Online Prematriculation Orientation Program and its Relation to Student Performance in the First Class Taken in Medical School. Teaching and Learning in Medicine, 20(4). 302-307.

7. Cook DA, Levinson AJ, Garside S. (2010). Time and Learning Efficiency in Internet-Based Learning: A Systematic Review and Meta-Analysis. Advances in Health Sciences Education, 15. 755-770.

8. Van Djiken PC, Thevoz S, Jucker-Kupper P, Feihl F, Bonvin R, Waeber B. (2008). Evaluation of an Online, Case-Based Interactive Approach to Teaching Pathophysiology. Medical Teacher, 30(5). 131-136.

9. Howlett D, Vincent T, Watson G, Owens E, Webb R, Gainsborough N, Fairclough J, Taylor N, Miles K, Cohen J, Vincent R. (2011). Blending Online Techniques with Traditional Face-to-Face Teaching Methods to Deliver Final Year Undergraduate Radiology Learning Content. European Journal of Radiology, 78(3). 334-341.

10. So HJ. (2009). Is Blended Learning a Viable Option in Public Health Education? A Case Study of Student Satisfaction with a Blended Graduate Course. Journal of Public Health Management and Practice, 15(1). 59-66. 

Library and information resources were perceived as valuable, and the information obtained was seen as having an impact on patient care.

Electronic access to information resources from multiple locations has increased the ability of health professionals to use these resources for improved patient care.

The roles of librarians are diversifying to include management of electronic resources, user instruction and support, specialized research and clinical information search services, and involvement in institution-level quality improvement.

It is possible to conduct a large-scale, multisite study on the value and impact of library services on patient care.

Searchers can be assured that the clinical queries for therapy, diagnosis, prognosis, etiology, and clinical prediction guides perform equally well in detecting relevant, methodologically sound studies in the current publishing years as they did when they were derived over a decade ago in the publishing year 2000.

JAMIA study.  From the group responsible for creating the filters.   

[via coworkers]

Wilczynski NL, McKibbon KA, Walter SD, Garg AX, Haynes RB. (2013). MEDLINE clinical queries are robust when searching in recent publishing years. JAMIA, 20(2), 363-8. PMID 23019242.

While I'm not going to be running statistical tests to identify out-of-date reviews, knowing that such tests exist might come in handy at some point when chatting w/ researchers. 

[via Evidence-Based Health]

Pattanittum P, Laopaiboon M, Moher D, Lumbiganon P, Ngamjarus C. (2012). A comparison of statistical methods for identifying out-of-date systematic reviews.  PLoS One, 7(11). PMID 23185281 PMC3502410

used internal company documents released in the course of litigation against the pharmaceutical company Pfizer regarding the drug gabapentin, to compare between the internal and published reports the reporting of the numbers of participants, the description of the types of analyses, and the definitions of each type of analysis.

to find that there were (also from summary):

several important differences between the internal and published documents about the number of patients included in the analysis

From the paper:

In our sample of industry-supported trials in off-label uses of gabapentin, we observed discrepancies between what was reported in trial publications and what was described in internal company research reports. In this regard, we found that the trial publication was not a transparent, or accurate (presuming that the research report truly describes the facts), record for the numbers of participants randomized and analyzed for efficacy. In three of ten trials in our sample, the number of participants randomized in the trial, as specified in the “main publication” [24],[26],[27], was not the same as that described in the research report. The “main publication” was a full-length journal article for two of the three trials with a disagreement in the number of participants randomized in the trial [24],[26], and a conference abstract describing preliminary results for the third trial [27]. In one case, the description in the publication did not include data from 40% of participants actually randomized in the trial (as described in the research report; see Table 1) [26]. There was such wide variation in describing the participant flow in the included trials, even among documents for the same trial, that we were unable to summarize what we found.

& 

We are concerned that, even for commonly used types of analysis such as ITT, a number of different definitions were used across trials included in our sample.

& the authors conclude with recommendations for improved standards and increased transparency.  

(More)

--

Vedula SS, Li T, Dickersin K.  Differences in reporting of analyses in internal company documents versus published trial reports: Comparisons in industry-sponsored trials in off-label uses of gabapentin.  PLoS Med. 2013 Jan;10(1). PMID 2332656 PMC3558476