The MACH project is funded by the Medical Research Council (Methodology Research Programme). MACH will have four components:
- Trans-disciplinary review of methods and practice in highly cited systematic reviews;
- Trans-disciplinary review of systematic review methods including meta-analysis, complexity and heterogeneity;
- Case studies including consultation and workshops with experts across disciplines; and
- Development of guidance
Trans-disciplinary review of methods and practice in highly cited systematic reviews
This componentwill explore current methodology and practice adopted by systematic reviews across disciplines. Its purpose is to inform the development of new guidance on the appropriate conduct of meta-analysis versus narrative synthesis in both clinical and non-clinical interventions which impact on human health and wellbeing. The disciplinary scope of the review will be broad to ensure that all relevant information is captured. We will use a combination of targeted searches and citation indexes to locate the top 20 cited systematic reviews in each discipline. The disciplines that are likely to include: medicine, psychology, public health, social policy research including criminology and education, international development, and environmental sciences including ecology and climate science.
An initial report summarising methods and practice in meta-analysis across disciplines, with particular attention given to any conflicts or lack of clarity in available methodological guidance, and draft guidance for use in the following project Phases.
Trans-disciplinary review of systematic review methods including meta-analysis, complexity and heterogeneity
The objective of this component will be to conduct a wide-ranging pragmatic review of current methodology and practice in different disciplines in order to inform the development of new guidance.
Consultation with experts and Case studies
Firstly, this component will aim to develop an initial consensus on the factors that should influence the decision to conduct meta-analysis, and will produce definitions and thresholds for these factors. Secondly, it will determine whether meta-analysis was feasible in some cases but not applied and if so, what it would have added to the review and whether the conclusions would have changed. It will also determine whether meta-analysis inappropriately applied and in what situations; and to explore researchers’ reasons for the adoption of different synthesis approaches, extending Ioannidis’ table of reasons beyond clinical and healthcare reviews.
We will convene three workshops in 2015, to be held either in London or attached to major relevant conferences such as the Cochrane Colloquium or the Society for Research Synthesis Methodology annual meeting, or the Methods in Meta-analysis /Meta-analysis in Medicine group. The workshops themselves will use a mixture of presentations and discussion, with an iterative process of placing draft guidance before attendees for criticism and to develop consensus. We have previously used this to develop an equity extension for the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidance and to develop guidance on systematic review of complex interventions as part of the Cochrane Health Equity Methods Group.
We will conduct 6-8 case studies, looking in depth at specific systematic reviews. Unlike the data for Phase 2, the case studies will come from explicitly health-related fields, with a focus on complex social interventions: public health, health system delivery and organisation, and/or international development interventions with health outcomes. These reviews will be identified using two methods. First, specific reviews known to the research team and existing networks, for which there is known to have been debate about the appropriateness of meta-analysis, will be included (e.g., there are examples in public health in the fields of housing and transport are known to the team). Second, we will use citation indexes to identify the most-cited recent systematic reviews in each field, and use this – by analysing the list of reviews in terms of the draft guidance – to systematically identify cases in which either (a) meta-analysis was conducted despite the review not meeting the draft guidance, or (b) meta-analysis was not conducted despite the review meeting the guidance, and where review authors are available for consultation.
Each case study will be investigated using two methods. First, we will re-analyse the data reported in the review; for the reviews where meta-analysis was not conducted, we will carry out the meta-analysis based on the data available from the review, and re-extracting data from primary studies where necessary. Second, we will contact review authors and carry out semi-structured interviews with them, exploring the reasoning behind the decisions regarding meta-analysis, in order to gain a more informed sense of what factors were in play within the review team. We will then draw together these two aspects of each case study in a mixed-methods synthesis, exploring where review authors’ perceptions may illuminate specific issues encountered in the meta-analysis.
- Draft guidance for conducting meta-analyses of complex social and public health interventions. Aside from issues of statistical heterogeneity, this draft guidance may include factors such as: primary study methods and/or quality; factors to do with the specific topic area (e.g., the level of complexity involved in the intervention); the breadth of the review question; and the intended audience and purpose of the review itself (relevant to “lumping” or “splitting”). Ioannidis’ table of reasons for not showing summary estimates  will also be used as an input into the draft guidance (e.g. these reasons include “Different interventions compared”, “synthesis considered inappropriate, no reason given,” etc).
- Summary report of the case studies, drawing together the analyses described above, and a set of annotations to the draft guidance, summarising where the case-study evidence may indicate issues with the appropriateness or feasibility of the draft guidance. Where we have conducted a reanalysis of an existing review, we will also work with the original review authors to publish the reanalysis.
Development of guidance
The objective of this component will be to draw together the elements of the project and produce usable guidance for reviewers. The final phase will involve re-convening the Advisory Group and running two workshops to refine the final guidelines (using consensus-building methods we have used in previous studies).