This paper provides guidance and tools for conducting open and reproducible systematic reviews in psychology. It emphasizes the importance of systematic reviews for evidence-based decision-making and the growing adoption of open science practices. Open science enhances transparency, reproducibility, and minimizes bias in systematic reviews by sharing data, materials, and code. It also fosters collaborations and enables involvement of non-academic stakeholders. The paper is designed for beginners, offering accessible guidance to navigate the many standards and resources that may not obviously align with specific areas of psychology. It covers systematic review conduct standards, pre-registration, registered reports, reporting standards, and open data, materials and code. The paper is concluded with a glimpse of recent innovations like Community Augmented Meta-Analysis and independent reproducibility checks.

Community-augmented meta-analysis (CAMA) platforms pioneer a new standard for promoting FAIR (findable, accessible, interoperable, reusable) data sharing, and allow dynamic and interactive meta-analysis which ensures reproducibility of results (Tsuji et al., 2014). As the area of (special) education research moves towards open science practices, our CAMA platform sets to facilitate data sharing of meta-analyses and make evidence-based practice accessible to practitioners. Furthermore, we aim to promote high-quality standards in conducting evidence synthesis, which are still not readily implemented in the education research area (Nordstrom et al., 2023).Our platform provides Bayesian and frequentist meta-analytic methods, interactive interface to conduct and visualize the analyses, easy-to-understand results and a large database of extracted effects that can be downloaded and reused by researchers. Users can conduct various moderator analyses, based on demographic information, risk of bias assessment, or study characteristics, and contribute to the database by submitting new extracted effects.

We aimed to identify effect sizes of age discrimination in recruitment based on evidence from correspondence studies and scenario experiments conducted between 2010 and 2019. To differentiate our results, we separated outcomes (i.e., call-back rates and hiring/invitation to interview likelihood) by age groups (40-49, 50-59, 60-65, 66+) and assessed age discrimination by comparing older applicants to a control group (29-35 year-olds). We conducted searches in PsycInfo, Web of Science, ERIC, BASE, and Google Scholar, along with backward reference searching. Study bias was assessed with a tool developed for this review, and publication bias by calculating R-index, p-curve, and funnel plots. We calculated odds ratios for callback rates, pooled the results using a random-effects meta-analysis and calculated 95% confidence intervals. We included 13 studies from 11 articles in our review, and conducted meta-analyses on the eight studies that we were able to extract data from. The majority of studies were correspondence studies (k=10) and came largely from European countries (k=9), with the rest being from the U.S. (k=3) and Australia (k=1). Seven studies had a between-participants design, and the remaining six studies had a within-participants design. We conducted six random-effects meta-analyses, one for each age category and type of study design and found an average effect of age discrimination against all age groups in both study designs, with varying effect sizes (ranging from OR = 0.38, CI [0.25, 0.59] to OR = 0.89, CI [0.81, 0.97]). There was moderate to high risk of bias on certain factors, e.g., age randomization, problems with application heterogeneity. Generally, there’s an effect of age discrimination and it tends to increase with age. This has important implications regarding the future of the world’s workforce, given the increase in the older workforce and later retirement.

Background:Community-augmented meta-analysis (CAMA) platforms have begun to set a new standard for promoting FAIR (findable, accessible, interoperable, reusable) data sharing. They allow dynamic and interactive meta-analysis of data and ensure reproducibility of results (Tsuji et al., 2014). As the area of disability research and education moves towards open science practices, the newly created CAMA platform sets to facilitate data sharing of meta-analyses and make evidence-based practice accessible to practitioners. Furthermore, we aim to promote high-quality standards in conducting evidence synthesis, which are still not readily implemented in the education research area (Nordström et al., 2022).

Objectives of the CAMA platform:The Evidence in Learning and Didactics, and Disability research CAMA platform will provide meta-analytic tools to conduct both frequentist (http://194.47.110.50:3838/visualization/), and Bayesian meta-analyses (http://194.47.110.51:3838/) of educational interventions for typically developing and students with intellectual disability. Studies will be subdivided into categories of typically developing students and students with intellectual disability, with further subdivision of educational domains: writing, reading, math, science, and other.

First objective: Create a platform that facilitates sharing of high-quality meta-analyses. The platform will allow publication bias assessment, effect size aggregation and moderator analysis. One important feature of this CAMA platforms is the ability to do analyses based on risk of bias assessments. Quality assessment and risk of bias estimation will be mandatory for dataset inclusion and it will be possible to conduct analyses on studies with different risks of bias estimation.

Second objective: Create a platform that is valuable both as a pedagogical and research tool. Apps will allow high flexibility in model building and provide an interface that provides plain and technical explanations/summaries of statistical outputs. The goal is to have the apps become an easy-to-use tool for students and researchers aiming to conduct meta-analyses and serve as a guide on evidence based practices for practitioners.

Introduction

Listening and communication difficulties can limit people’s participation in activity and adversely affect their quality of life. Hearing, as well as listening and communication difficulties, can be measured either by using behavioural tests or self-report measures, and the outcomes are not always closely linked. The association between behaviourally measured and self-reported hearing is strong, whereas the association between behavioural and self-reported measures of listening and communication difficulties is much weaker, suggesting they assess different aspects of listening. While behavioural measures of listening and communication difficulties have been associated with poorer cognitive performance including executive functions, the same association has not always been shown for self-report measures. The objective of this systematic review and meta-analysis is to understand the relationship between executive function and self-reported listening and communication difficulties in adults with hearing loss, and where possible, potential covariates of age and pure-tone audiometric thresholds.

Methods and analysis

Studies will be eligible for inclusion if they report data from both a self-report measure of listening difficulties and a behavioural measure of executive function. Eight databases are to be searched: MEDLINE (via Ovid SP), EMBASE (via Ovid SP), PsycINFO (via Ovid SP), ASSIA (via ProQuest), Cumulative Index to Nursing and Allied Health Literature or CINAHL (via EBSCO Host), Scopus, PubMed and Web of Science (Science and Social Science Citation Index). The JBI critical appraisal tool will be used to assess risk of bias for included studies. Results will be synthesised primarily using a meta-analysis, and where sufficient quantitative data are not available, a narrative synthesis will be carried out to describe key results.

Ethics and dissemination

No ethical issues are foreseen. Data will be disseminated via academic publication and conference presentations. Findings may also be published in scientific newsletters and magazines.

In order to produce the most reliable syntheses of the effectiveness of educational interventions, systematic reviews need to adhere to rigorous methodological standards. This meta-review investigated risk of bias occurring while conducting a systematic review and the presence of open science practices like data sharing and reproducibility of the review procedure, in recently published reviews in education. We included all systematic reviews of educational interventions, instructions and methods for all K-12 student populations in any school form with experimental or quasi-experimental designs (an active manipulation of the intervention) with comparisons and where the outcome variables were academic performance of any kind. We searched the database Education Resources Information Center (ERIC) through the years 2019-2021. In parallel we hand-searched four major educational review journals for systematic reviews: Educational Research Review (Elsevier), Educational Review (Taylor & Francis), Review of Education (Wiley), and Review of Educational Research (AERA). Systematic reviews were assessed with the risk of bias tool ROBIS and whether the studies had pre-registered protocols, shared primary research data, and whether a third party could reproduce search strings and details of where exactly primary research data were extracted. A total of 88 studies that matched our PICOS were included in this review; of these, 10 educational systematic reviews were judged as low risk of bias (approximately 11%) . The rest were classified as high risk of bias during a shortened ROBIS assessment or assessed as high risk or unclear risk of bias following a full ROBIS assessment. Of the 10 low risk of bias reviews, 6 had detailed their search sufficiently enough for a third party to reproduce, 3 reviews shared the data from primary studies, however none had specified how and from where exactly data from primary studies were extracted. The study shows that at least a small part of systematic reviews in education has a low risk of bias, but most systematic reviews in our set of studies have high risk of bias in their methodological procedure. There are still improvements in this field to be expected as even the low risk of bias reviews are not consistent regarding pre-registered protocols, data sharing, reproducibility of primary research data and reproducible search strings.Rationale for this studyRigorous systematic review is the method to use to evaluate and provide synthesis of methods and interventions in educational research.Why the new findings matterAlarmingly few systematic reviews were judged with low risk of bias and few reviews utilise recent open science practices. This might undermine the reliability of study findings.Implications for journals and review researchersThis meta-review highlights the need for review researchers and journals to better adopt best practices in systematic reviews. We therefore urge for better dissemination and awareness of systematic reviews standards and for more transparency in the review process, which may lead to more reliable synthesis of what works best in education. In the long run this is a necessary change and way forward to provide evidence-based practices in the classroom.Context and implications