liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Risk of bias and open science practices in systematic reviews of educational effectiveness: A meta‐review
Department of Psychology, Linnaeus University, Växjö, Sweden.ORCID iD: 0000-0002-3829-4169
Department of Psychology, Linnaeus University, Växjö, Sweden.ORCID iD: 0000-0003-2871-9693
Linköping University, Department of Behavioural Sciences and Learning, Disability Research Division. Linköping University, Faculty of Arts and Sciences.ORCID iD: 0000-0002-1017-0025
2023 (English)In: Review of Education, E-ISSN 2049-6613, Vol. 11, no 3, article id e3443Article, review/survey (Refereed) Published
Abstract [en]

In order to produce the most reliable syntheses of the effectiveness of educational interventions, systematic reviews need to adhere to rigorous methodological standards. This meta-review investigated risk of bias occurring while conducting a systematic review and the presence of open science practices like data sharing and reproducibility of the review procedure, in recently published reviews in education. We included all systematic reviews of educational interventions, instructions and methods for all K-12 student populations in any school form with experimental or quasi-experimental designs (an active manipulation of the intervention) with comparisons and where the outcome variables were academic performance of any kind. We searched the database Education Resources Information Center (ERIC) through the years 2019-2021. In parallel we hand-searched four major educational review journals for systematic reviews: Educational Research Review (Elsevier), Educational Review (Taylor & Francis), Review of Education (Wiley), and Review of Educational Research (AERA). Systematic reviews were assessed with the risk of bias tool ROBIS and whether the studies had pre-registered protocols, shared primary research data, and whether a third party could reproduce search strings and details of where exactly primary research data were extracted. A total of 88 studies that matched our PICOS were included in this review; of these, 10 educational systematic reviews were judged as low risk of bias (approximately 11%) . The rest were classified as high risk of bias during a shortened ROBIS assessment or assessed as high risk or unclear risk of bias following a full ROBIS assessment. Of the 10 low risk of bias reviews, 6 had detailed their search sufficiently enough for a third party to reproduce, 3 reviews shared the data from primary studies, however none had specified how and from where exactly data from primary studies were extracted. The study shows that at least a small part of systematic reviews in education has a low risk of bias, but most systematic reviews in our set of studies have high risk of bias in their methodological procedure. There are still improvements in this field to be expected as even the low risk of bias reviews are not consistent regarding pre-registered protocols, data sharing, reproducibility of primary research data and reproducible search strings.Rationale for this studyRigorous systematic review is the method to use to evaluate and provide synthesis of methods and interventions in educational research.Why the new findings matterAlarmingly few systematic reviews were judged with low risk of bias and few reviews utilise recent open science practices. This might undermine the reliability of study findings.Implications for journals and review researchersThis meta-review highlights the need for review researchers and journals to better adopt best practices in systematic reviews. We therefore urge for better dissemination and awareness of systematic reviews standards and for more transparency in the review process, which may lead to more reliable synthesis of what works best in education. In the long run this is a necessary change and way forward to provide evidence-based practices in the classroom.Context and implications

Place, publisher, year, edition, pages
WILEY , 2023. Vol. 11, no 3, article id e3443
Keywords [en]
education; meta-review; open science; reproducibility; risk of bias; systematic review
National Category
Educational Sciences
Identifiers
URN: urn:nbn:se:liu:diva-199902DOI: 10.1002/rev3.3443ISI: 001135375900005OAI: oai:DiVA.org:liu-199902DiVA, id: diva2:1823752
Funder
Swedish Research Council
Note

Funding: Vetenskapsrdet

Available from: 2024-01-03 Created: 2024-01-03 Last updated: 2024-01-23

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full text

Authority records

Batinovic, Lucija

Search in DiVA

By author/editor
Nordström, ThomasKalmendal, AndréBatinovic, Lucija
By organisation
Disability Research DivisionFaculty of Arts and Sciences
In the same journal
Review of Education
Educational Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 38 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf