how to create a systematic review question

University of Tasmania, Australia

Systematic reviews for health: 1. formulate the research question.

Handbooks / Guidelines for Systematic Reviews
Standards for Reporting
Registering a Protocol
Tools for Systematic Review
Online Tutorials & Courses
Books and Articles about Systematic Reviews
Finding Systematic Reviews
Critical Appraisal
Library Help
Bibliographic Databases
Grey Literature
Handsearching
Citation Tracking
1. Formulate the Research Question
2. Identify the Key Concepts
3. Develop Search Terms - Free-Text
4. Develop Search Terms - Controlled Vocabulary
5. Search Fields
6. Phrase Searching, Wildcards and Proximity Operators
7. Boolean Operators
8. Search Limits
9. Pilot Search Strategy & Monitor Its Development
10. Final Search Strategy
11. Adapt Search Syntax
Documenting Search Strategies
Handling Results & Storing Papers

how to create a systematic review question

Step 1. Formulate the Research Question

A systematic review is based on a pre-defined specific research question ( Cochrane Handbook, 1.1 ). The first step in a systematic review is to determine its focus - you should clearly frame the question(s) the review seeks to answer ( Cochrane Handbook, 2.1 ). It may take you a while to develop a good review question - it is an important step in your review. Well-formulated questions will guide many aspects of the review process, including determining eligibility criteria, searching for studies, collecting data from included studies, and presenting findings ( Cochrane Handbook, 2.1 ).

The research question should be clear and focused - not too vague, too specific or too broad.

You may like to consider some of the techniques mentioned below to help you with this process. They can be useful but are not necessary for a good search strategy.

PICO - to search for quantitative review questions

Richardson, WS, Wilson, MC, Nishikawa, J & Hayward, RS 1995, 'The well-built clinical question: A key to evidence-based decisions', ACP Journal Club , vol. 123, no. 3, pp. A12-A12 .

We do not have access to this article at UTAS.

A variant of PICO is PICOS . S stands for Study designs . It establishes which study designs are appropriate for answering the question, e.g. randomised controlled trial (RCT). There is also PICO C (C for context) and PICO T (T for timeframe).

You may find this document on PICO / PIO / PEO useful:

Framing a PICO / PIO / PEO question Developed by Teesside University

SPIDER - to search for qualitative and mixed methods research studies

Cooke, A, Smith, D & Booth, A 2012, 'Beyond pico the spider tool for qualitative evidence synthesis', Qualitative Health Research , vol. 22, no. 10, pp. 1435-1443.

This article is only accessible for UTAS staff and students.

SPICE - to search for qualitative evidence

Cleyle, S & Booth, A 2006, 'Clear and present questions: Formulating questions for evidence based practice', Library hi tech , vol. 24, no. 3, pp. 355-368.

ECLIPSE - to search for health policy/management information

Wildridge, V & Bell, L 2002, 'How clip became eclipse: A mnemonic to assist in searching for health policy/management information', Health Information & Libraries Journal , vol. 19, no. 2, pp. 113-115.

There are many more techniques available. See the below guide from the CQUniversity Library for an extensive list:

Question frameworks overview from Framing your research question guide, developed by CQUniversity Library

This is the specific research question used in the example:

"Is animal-assisted therapy more effective than music therapy in managing aggressive behaviour in elderly people with dementia?"

Within this question are the four PICO concepts :

S - Study design

This is a therapy question. The best study design to answer a therapy question is a randomised controlled trial (RCT). You may decide to only include studies in the systematic review that were using a RCT, see Step 8 .

See source of example

Need More Help? Book a consultation with a Learning and Research Librarian or contact [email protected] .

<< Previous: Building Search Strategies
Next: 2. Identify the Key Concepts >>
Last Updated: Apr 4, 2024 10:17 AM
URL: https://utas.libguides.com/SystematicReviews

Library Services

UCL LIBRARY SERVICES

Guides and databases
Library skills
Systematic reviews

Formulating a research question

What are systematic reviews?
Types of systematic reviews
Identifying studies
Searching databases
Describing and appraising studies
Synthesis and systematic maps
Software for systematic reviews
Online training and support
Live and face to face training
Individual support
Further help

Clarifying the review question leads to specifying what type of studies can best address that question and setting out criteria for including such studies in the review. This is often called inclusion criteria or eligibility criteria. The criteria could relate to the review topic, the research methods of the studies, specific populations, settings, date limits, geographical areas, types of interventions, or something else.

Systematic reviews address clear and answerable research questions, rather than a general topic or problem of interest. They also have clear criteria about the studies that are being used to address the research questions. This is often called inclusion criteria or eligibility criteria.

Six examples of types of question are listed below, and the examples show different questions that a review might address based on the topic of influenza vaccination. Structuring questions in this way aids thinking about the different types of research that could address each type of question. Mneumonics can help in thinking about criteria that research must fulfil to address the question. The criteria could relate to the context, research methods of the studies, specific populations, settings, date limits, geographical areas, types of interventions, or something else.

Examples of review questions

Needs - What do people want? Example: What are the information needs of healthcare workers regarding vaccination for seasonal influenza?
Impact or effectiveness - What is the balance of benefit and harm of a given intervention? Example: What is the effectiveness of strategies to increase vaccination coverage among healthcare workers. What is the cost effectiveness of interventions that increase immunisation coverage?
Process or explanation - Why does it work (or not work)? How does it work (or not work)? Example: What factors are associated with uptake of vaccinations by healthcare workers? What factors are associated with inequities in vaccination among healthcare workers?
Correlation - What relationships are seen between phenomena? Example: How does influenza vaccination of healthcare workers vary with morbidity and mortality among patients? (Note: correlation does not in itself indicate causation).
Views / perspectives - What are people's experiences? Example: What are the views and experiences of healthcare workers regarding vaccination for seasonal influenza?
Service implementation - What is happening? Example: What is known about the implementation and context of interventions to promote vaccination for seasonal influenza among healthcare workers?

Examples in practice : Seasonal influenza vaccination of health care workers: evidence synthesis / Loreno et al. 2017

Example of eligibility criteria

Research question: What are the views and experiences of UK healthcare workers regarding vaccination for seasonal influenza?

Population: healthcare workers, any type, including those without direct contact with patients.
Context: seasonal influenza vaccination for healthcare workers.
Study design: qualitative data including interviews, focus groups, ethnographic data.
Date of publication: all.
Country: all UK regions.
Studies focused on influenza vaccination for general population and pandemic influenza vaccination.
Studies using survey data with only closed questions, studies that only report quantitative data.

Consider the research boundaries

It is important to consider the reasons that the research question is being asked. Any research question has ideological and theoretical assumptions around the meanings and processes it is focused on. A systematic review should either specify definitions and boundaries around these elements at the outset, or be clear about which elements are undefined.

For example if we are interested in the topic of homework, there are likely to be pre-conceived ideas about what is meant by 'homework'. If we want to know the impact of homework on educational attainment, we need to set boundaries on the age range of children, or how educational attainment is measured. There may also be a particular setting or contexts: type of school, country, gender, the timeframe of the literature, or the study designs of the research.

Research question: What is the impact of homework on children's educational attainment?

Scope : Homework - Tasks set by school teachers for students to complete out of school time, in any format or setting.
Population: children aged 5-11 years.
Outcomes: measures of literacy or numeracy from tests administered by researchers, school or other authorities.
Study design: Studies with a comparison control group.
Context: OECD countries, all settings within mainstream education.
Date Limit: 2007 onwards.
Any context not in mainstream primary schools.
Non-English language studies.

Mnemonics for structuring questions

Some mnemonics that sometimes help to formulate research questions, set the boundaries of question and inform a search strategy.

Intervention effects

PICO Population – Intervention– Outcome– Comparison

Variations: add T on for time, or ‘C’ for context, or S’ for study type,

Policy and management issues

ECLIPSE : Expectation – Client group – Location – Impact ‐ Professionals involved – Service

Expectation encourages reflection on what the information is needed for i.e. improvement, innovation or information. Impact looks at what you would like to achieve e.g. improve team communication .

How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information / Wildridge & Bell, 2002

Analysis tool for management and organisational strategy

PESTLE: Political – Economic – Social – Technological – Environmental ‐ Legal

An analysis tool that can be used by organizations for identifying external factors which may influence their strategic development, marketing strategies, new technologies or organisational change.

PESTLE analysis / CIPD, 2010

Service evaluations with qualitative study designs

SPICE: Setting (context) – Perspective– Intervention – Comparison – Evaluation

Perspective relates to users or potential users. Evaluation is how you plan to measure the success of the intervention.

Clear and present questions: formulating questions for evidence based practice / Booth, 2006

Read more about some of the frameworks for constructing review questions:

Formulating the Evidence Based Practice Question: A Review of the Frameworks / Davis, 2011
<< Previous: Stages in a systematic review
Next: Identifying studies >>
Last Updated: Apr 4, 2024 10:09 AM
URL: https://library-guides.ucl.ac.uk/systematic-reviews

Systematic reviews: Formulate your question

Introduction
Formulate your question
Write a protocol
Search the literature
Manage references
Select studies
Assess the evidence
Write your review
Further resources

Defining the question

Defining the research question and developing a protocol are the essential first steps in your systematic review. The success of your systematic review depends on a clear and focused question, so take the time to get it right.

A framework may help you to identify the key concepts in your research question and to organise your search terms in one of the Library's databases.
Several frameworks or models exist to help researchers structure a research question and three of these are outlined on this page: PICO, SPICE and SPIDER.
It is advisable to conduct some scoping searches in a database to look for any reviews on your research topic and establish whether your topic is an original one .
Y ou will need to identify the relevant database(s) to search and your choice will depend on your topic and the research question you need to answer.
By scanning the titles, abstracts and references retrieved in a scoping search, you will reveal the terms used by authors to describe the concepts in your research question, including the synonyms or abbreviations that you may wish to add to a database search.
The Library can help you to search for existing reviews: make an appointment with your Subject Librarian to learn more.

The PICO framework

PICO may be the most well-known model framework: it has its origins in epidemiology and now is widely-used for evidence-based practice and systematic reviews.

PICO normally stands for Population (or Patient or Problem) - Intervention - Comparator - Outcome.

The SPICE framework

SPICE is used mostly in social science and healthcare research. It stands for Setting - Population (or Perspective) - Intervention - Comparator - Evaluation. It is similar to PICO and was devised by Booth (2004).

The examples in the SPICE table are based on the following research question: Can mortality rates for older people be reduced if a greater proportion are examined initially by allied health staff in A&E? Source: Booth, A (2004) Formulating answerable questions. In Booth, A & Brice, A (Eds) Evidence Based Practice for Information Professionals: A handbook. (pp. 61-70) London: Facet Publishing.

The SPIDER framework

SPIDER was adapted from the PIC O framework in order to include searches for qualitative and mixed-methods research. SPIDER was developed by Cooke, Smith and Booth (2012).

Source : Cooke, A., Smith, D. & Booth, A. (2012). Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qualitative Health Research (10), 1435-1443. http://doi.org/10.1177/1049732312452938 .

Systematic Reviews & Evidence Synthesis Methods

Formulate Question
Types of Reviews
Find Existing Reviews & Protocols
Register a Protocol
Searching Systematically
Supplementary Searching
Managing Results
Deduplication
Critical Appraisal
Glossary of terms
Librarian Support
Video tutorials This link opens in a new window
Systematic Review & Evidence Synthesis Boot Camp

Formulate your Research Question

Formulating a strong research question for a systematic review can be a lengthy process. While you may have an idea about the topic you want to explore, your specific research question is what will drive your review and requires some consideration.

You will want to conduct preliminary or exploratory searches of the literature as you refine your question. In these searches you will want to:

Determine if a systematic review has already been conducted on your topic and if so, how yours might be different, or how you might shift or narrow your anticipated focus.
Scope the literature to determine if there is enough literature on your topic to conduct a systematic review.
Identify key concepts and terminology.
Identify seminal or landmark studies.
Identify key studies that you can test your search strategy against (more on that later).
Begin to identify databases that might be useful to your search question.

Types of Research Questions for Systematic Reviews

A narrow and specific research question is required in order to conduct a systematic review. The goal of a systematic review is to provide an evidence synthesis of ALL research performed on one particular topic. Your research question should be clearly answerable from the studies included in your review.

Another consideration is whether the question has been answered enough to warrant a systematic review. If there have been very few studies, there won't be enough qualitative and/or quantitative data to synthesize. You then have to adjust your question... widen the population, broaden the topic, reconsider your inclusion and exclusion criteria, etc.

When developing your question, it can be helpful to consider the FINER criteria (Feasible, Interesting, Novel, Ethics, and Relevant). Read more about the FINER criteria on the Elsevier blog .

If you have a broader question or aren't certain that your question has been answered enough in the literature, you may be better served by pursuing a systematic map, also know as a scoping review . Scoping reviews are conducted to give a broad overview of a topic, to review the scope and themes of the prior research, and to identify the gaps and areas for future research.

CEE Example Questions Collaboration for Environmental Evidence Guidelines contains Table 2.2 outlining answers sought and example questions in environmental management.

Learn More . . .

Cochrane Handbook Chapter 2 - Determining the scope of the review and the questions it will address

Frameworks for Developing your Research Question

PICO : P atient/ P opulation, I ntervention, C omparison, O utcome.

PEO: P opulation, E xposure, O utcomes

SPIDER : S ample, P henomenon of I nterest, D esign, E valuation, R esearch Type

For more frameworks and guidance on developing the research question, check out:

1. Advanced Literature Search and Systematic Reviews: Selecting a Framework. City University of London Library

2. Select the Appropriate Framework for your Question. Tab "1-1" from PIECES: A guide to developing, conducting, & reporting reviews [Excel workbook ]. Margaret J. Foster, Texas A&M University. CC-BY-3.0 license .

3. Formulating a Research Question. University College London Library. Systematic Reviews .

4. Question Guidance. UC Merced Library. Systematic Reviews

Sort Link Group

Add / Reorder

Video - Formulating a Research Question (4:43 minutes)

Last Updated: Feb 27, 2024 12:53 PM
URL: https://guides.lib.utexas.edu/systematicreviews

Memorial Sloan Kettering Library

The physical space of the MSK Library is scheduled to close to visitors on Friday, May 17, 2024. Please visit this guide for more information.

Systematic Review Service

Review Types
How Long Do Reviews Take
Which Review Type is Right for Me
Policies for Partnering with an MSK Librarian
Request to Work with an MSK Librarian
Your First Meeting with an MSK Librarian
Covidence Review Software
Step 1: Form Your Team
Step 2: Define Your Research Question
Step 3: Write and Register Your Protocol
Step 4: Search for Evidence
Step 5: Screen Your Results
Step 6: Assess the Quality
Step 7: Collect the Data
Step 8: Write and Publish the Review
Additional Resources

Define Your Research Question

Example of a focused and a broad research question

A well-developed research question will inform the entirety of your review process, including:

The development of your inclusion and exclusion criteria.
The terms used in your search strategies.
The tool(s) used to assess the quality of included studies.
The data pulled from the included studies.
The analysis completed in your review.
The target journal(s) for your review's publication.

If your question is too broad, you may have trouble completing the review. If your topic is too narrow, there may not be sufficient literature to warrant a review.

How the MSK Library Can Help

One of the first conversations you will have with your MSK librarian will be about your topic.

Your MSK librarian will:

Work with you to determine whether a systematic review on your topic has been published or planned by searching databases like PubMed , Embase , and Epistemonikos and registries like PROSPERO , Protocols.io , and Open Science Framework (OSF) Registries .
Ask you for a sample set of relevant publications (also known as seed articles) that you know you want your review to capture. This helps provide a better sense of the scope of your research question. If your topic is too broad or narrow, your MSK librarian can help improve the focus. This sample set will later inform the construction of the search strategy.

Using a Question Framework

What if my topic does not fit a framework?

PICO is a model commonly used for clinical and healthcare related questions, and is often, although not exclusively, used for searching for quantitively designed studies.

Example question: In elderly patients, does patient handwashing compared to no handwashing impact rates of hospital-acquired infections?

Richardson, W.S., Wilson, M.C, Nishikawa, J. and Hayward, R.S.A. (1995). "The well-built clinical question: a key to evidence-based decisions." ACP Journal Club , 123(3), A12.

Question framework content adapted from The University of Plymouth Library .

PEO is useful for qualitative research questions.

Example question: In homeless populations, do addiction services impact housing rates?

Moola S, Munn Z, Sears K, Sfetcu R, Currie M, Lisy K, Tufanaru C, Qureshi R, Mattis P & Mu P. (2015). "Conducting systematic reviews of association (etiology): The Joanna Briggs Institute's approach." International Journal of Evidence - Based Healthcare, 13(3), 163-9. Available at: 10.1097/XEB.0000000000000064.

PCC is useful for both qualitative and quantitative (mixed methods) topics, and is commonly used in scoping reviews.

Example question: What patient-led models of care are used to manage chronic disease in high income countries?

Chronic disease

Patient-led care models

Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil, H. "Chapter 11: Scoping Reviews" (2020 version). In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis, JBI, 2020. Available from https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-12

Question framework content adapted from The University of Plymouth Library .

SPIDER is a model useful for qualitative and mixed method type research questions.

Example question: What are young parents’ experiences of attending antenatal education?

Cooke, A., Smith, D. and Booth, A. (2012)."Beyond PICO: the SPIDER tool for qualitative evidence synthesis." Qualitative Health Research , 22(10), 1435-1443.

SPICE is a model useful for qualitative and mixed method type research questions.

Example question: Does mindfulness therapy in a counseling service impact the attitudes of patients diagnosed with cancer?

Example question adapted from: Tate, KJ., Newbury-Birch, D., and McGeechan, GJ. (2018). "A systematic review of qualitative evidence of cancer patients’ attitudes to mindfulness." European Journal of Cancer Care , 27(2), 1-10.

ECLIPSE is a model useful for qualitative and mixed method type research questions, especially for questions examining particular services or professions.

Example question: Can cross-service communication impact the support of adults with learning difficulties?

You might find that your topic does not always fall into one of the models listed on this page. You can always modify a model to make it work for your topic, and either remove or incorporate additional elements.

The important thing is to ensure that you have a high quality question that can be separated into its component parts.

<< Previous: Step 1: Form Your Team
Next: Step 3: Write and Register Your Protocol >>
Last Updated: Apr 5, 2024 4:54 PM
URL: https://libguides.mskcc.org/systematic-review-service

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base

Methodology

Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

Formulate a research question
Develop a protocol
Search for all relevant studies
Apply the selection criteria
Extract the data
Synthesize the data
Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Prevent plagiarism. Run a free check.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
Access to databases and journal archives. Often, your educational institution provides you with access.
Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

They minimize research bias by considering all available evidence and evaluating each study for bias.
Their methods are transparent , so they can be scrutinized by others.
They’re thorough : they summarize all available evidence.
They can be replicated and updated by others.

Systematic reviews also have a few cons .

They’re time-consuming .
They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

Allow you to more effectively communicate your research to other researchers and practitioners
Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

Population(s) or problem(s)
Intervention(s)
Comparison(s)

You can rearrange these four components to write your research question:

What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

Type of study design(s)
The population of patients with eczema
The intervention of probiotics
In comparison to no treatment, placebo , or non-probiotic treatment
The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
Randomized control trials, a type of study design

Their research question was:

What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

Background information : Provide the context of the research question, including why it’s important.
Research objective (s) : Rephrase your research question as an objective.
Selection criteria: State how you’ll decide which studies to include or exclude from your review.
Search strategy: Discuss your plan for finding studies.
Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
Handsearch: Conference proceedings and reference lists of articles
Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

Abstract : A summary of the review
Introduction : Including the rationale and objectives
Methods : Including the selection criteria, search method, data extraction method, and synthesis method
Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
Discussion : Including interpretation of the results and limitations of the review
Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

Student’s t -distribution
Normal distribution
Null and Alternative Hypotheses
Chi square tests
Confidence interval
Quartiles & Quantiles
Cluster sampling
Stratified sampling
Data cleansing
Reproducibility vs Replicability
Peer review
Prospective cohort study

Research bias

Implicit bias
Cognitive bias
Placebo effect
Hawthorne effect
Hindsight bias
Affect heuristic
Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other academic texts , with an introduction , a main body, and a conclusion .

An annotated bibliography is a list of source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a paper .

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved April 3, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, unlimited academic ai-proofreading.

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Mayo Clinic Libraries
Systematic Reviews

Develop & Refine Your Research Question

Systematic reviews: develop & refine your research question.

Knowledge Synthesis Comparison
Knowledge Synthesis Decision Tree
Standards & Reporting Results
Materials in the Mayo Clinic Libraries
Training Resources
Review Teams
Develop a Timeline
Project Management
Communication
PRISMA-P Checklist
Eligibility Criteria
Register your Protocol
Other Resources
Other Screening Tools
Grey Literature Searching
Citation Searching
Data Extraction Tools
Minimize Bias
Critical Appraisal by Study Design
Synthesis & Meta-Analysis
Publishing your Systematic Review

A clear, well-defined, and answerable research question is essential for any systematic review, meta-analysis, or other form of evidence synthesis. The question must be answerable. Spend time refining your research question.

PICO Worksheet

PICO Framework

Focused question frameworks.

The PICO mnemonic is frequently used for framing quantitative clinical research questions. 1

The PEO acronym is appropriate for studies of diagnostic accuracy 2

The SPICE framework is effective “for formulating questions about qualitative or improvement research.” 3

The SPIDER search strategy was designed for framing questions best answered by qualitative and mixed-methods research. 4

References & Recommended Reading

1. Anastasiadis E, Rajan P, Winchester CL. Framing a research question: The first and most vital step in planning research. Journal of Clinical Urology. 2015;8(6):409-411.

2. Speckman RA, Friedly JL. Asking Structured, Answerable Clinical Questions Using the Population, Intervention/Comparator, Outcome (PICO) Framework. PM&R. 2019;11(5):548-553.

3. Knowledge Into Action Toolkit. NHS Scotland. http://www.knowledge.scot.nhs.uk/k2atoolkit/source/identify-what-you-need-to-know/spice.aspx . Accessed April 23, 2021.

4. Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qualitative health research. 2012;22(10):1435-1443.

<< Previous: Review Teams
Next: Develop a Timeline >>
Last Updated: Apr 5, 2024 12:20 PM
URL: https://libraryguides.mayo.edu/systematicreviewprocess

Duke NetID Login
919.660.1100
Duke Health Badge: 24-hour access
Accounts & Access
Databases, Journals & Books
Request & Reserve
Training & Consulting
Request Articles & Books
Renew Online
Reserve Spaces
Reserve a Locker
Study & Meeting Rooms
Course Reserves
Digital Health Device Collection
Pay Fines/Fees
Recommend a Purchase
Access From Off Campus
Building Access
Computers & Equipment
Wifi Access
My Accounts
Mobile Apps
Known Access Issues
Report an Access Issue
All Databases
Article Databases
Basic Sciences
Clinical Sciences
Dissertations & Theses
Drugs, Chemicals & Toxicology
Grants & Funding
Interprofessional Education
Non-Medical Databases
Search for E-Journals
Search for Print & E-Journals
Search for E-Books
Search for Print & E-Books
E-Book Collections
Biostatistics
Global Health
MBS Program
Medical Students
MMCi Program
Occupational Therapy
Path Asst Program
Physical Therapy
Researchers
Community Partners

Conducting Research

Archival & Historical Research
Black History at Duke Health
Data Analytics & Viz Software
Data: Find and Share
Evidence-Based Practice
NIH Public Access Policy Compliance
Publication Metrics
Qualitative Research
Searching Animal Alternatives

Systematic Reviews

Test Instruments

Using Databases

JCR Impact Factors
Web of Science

Finding & Accessing

COVID-19: Core Clinical Resources
Health Literacy
Health Statistics & Data
Library Orientation

Writing & Citing

Creating Links
Getting Published
Reference Mgmt
Scientific Writing

Meet a Librarian

Request a Consultation
Find Your Liaisons
Register for a Class
Request a Class
Self-Paced Learning

Search Services

Literature Search
Systematic Review
Animal Alternatives (IACUC)
Research Impact

Citation Mgmt

Other Software

Scholarly Communications

About Scholarly Communications
Publish Your Work
Measure Your Research Impact
Engage in Open Science
Libraries and Publishers
Directions & Maps
Floor Plans

Library Updates

Annual Snapshot
Conference Presentations
Contact Information
Gifts & Donations
What is a Systematic Review?
Types of Reviews
Manuals and Reporting Guidelines
Our Service
1. Assemble Your Team

2. Develop a Research Question

3. Write and Register a Protocol
4. Search the Evidence
5. Screen Results
6. Assess for Quality and Bias
7. Extract the Data
8. Write the Review
Additional Resources
Finding Full-Text Articles

A well-developed and answerable question is the foundation for any systematic review. This process involves:

Systematic review questions typically follow a PICO-format (patient or population, intervention, comparison, and outcome)
Using the PICO framework can help team members clarify and refine the scope of their question. For example, if the population is breast cancer patients, is it all breast cancer patients or just a segment of them?
When formulating your research question, you should also consider how it could be answered. If it is not possible to answer your question (the research would be unethical, for example), you'll need to reconsider what you're asking
Typically, systematic review protocols include a list of studies that will be included in the review. These studies, known as exemplars, guide the search development but also serve as proof of concept that your question is answerable. If you are unable to find studies to include, you may need to reconsider your question

Ohio State nav bar

The Ohio State University

BuckeyeLink
Find People
Search Ohio State

Systematic Reviews

What is a Systematic Review?
1. Choose the Right Kind of Review
2. Formulate Your Question

Formulate Your Question

Inclusion and exclusion criteria, scoping the question.

3. Establish a Team
4. Develop a Protocol
5. Conduct the Search
6. Select Studies
7. Extract Data
8. Synthesize Your Results
9. Disseminate Your Report
Request a Librarian Consultation

Consult With a Librarian

To make an appointment to consult with an HSL librarian on your systematic review, please read our Systematic Review Policy and submit a Systematic Review Consultation Request .

To ask a question or make an appointment for assistance with a narrative review, please complete the Ask a Librarian Form .

A clear, specific, and answerable question is essential to a successful systematic review. A well formulated question will help determine your protocol and search strategy, and help you to find relevant and valid information quickly.

As you develop your question, PICO is a useful tool that can help you define the question's core concepts. There are four components that typically make up a clinical question:

P atient, problem, or population
I ntervention
C omparison intervention (if applicable)

You may also see this framework referred to as PICO(T), with T referring to time.

Here's an example of a clinical question broken down using PICO:

https://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0079444/#ddd00064

The Cochrane Handbook suggests the following factors to consider when using PICO to develop your question:

Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011.

Once you have a question, you can use those PICO components to help define your inclusion and exclusion criteria.

Inclusion Criteria are everything that a study must have to be included in the review.

Exclusion Criteria are anything that would make a study ineligible to be included in the review.

It is important to have these criteria clearly defined before beginning your search to ensure that your selection process is thorough, consistent, and reproducible, and focuses on studies applicable to the research question..

In addition to the factors associated with P opulations, I nterventions and C omparitors, and O utcomes, other parameters along which inclusion criteria could be set include:

Study design (Randomized controlled trials? Cohort studies? Case-Control Studies? etc.)
Date (Was it published sufficiently recently? Is the technology used outdated? etc.)

It is not generally possible to formulate an answerable question and determine appropriate inclusion criteria for a review without some knowledge of the existing research relevant to the question. You may need to conduct some preliminary research to help you develop a question that is viable for a systematic review.

Performing preliminary research on a topic can help you:

Explore the extent and nature of existing literature.
Identify gaps and uncertainties that might be addressed by a systematic review.
Understand the terms and concepts used in relevant literature.
Help identify appropriate parameters for the review (PICO).
Identify the potential scope of a systematic review.

Your research question needs to be specific enough to get a meaningful conclusion, but broad enough to have enough literature to analyze.

Prior investigation can help you assess whether a systematic review on a given question is justified. It is necessary to check whether there are already existing or ongoing reviews on your topic. Doing so may help you in choosing or refining your question.

Find current/on-going systematic reviews with:

PROSPERO The International Prospective Register of Systematic Reviews

Find published systematic reviews with:

PubMed Clinical Queries Systematic reviews appear in the middle column of search results.
Cochrane Database of Systematic Reviews
Systematic Review Data Repository The Agency for Healthcare Research and Quality's repository of systematic reviews.

If you choose to update an existing systematic review , this article can provide some guidance:

Article: When and how to update an existing systematic review BMJ 2016;354:i3507
<< Previous: 1. Choose the Right Kind of Review
Next: 3. Establish a Team >>
Last Updated: Sep 12, 2023 11:57 AM
URL: https://hslguides.osu.edu/systematic_reviews

Systematic reviews

Introduction to systematic reviews
Steps in a systematic review

Formulating a clear and concise question

Pico framework, other search frameworks.

Create a protocol (plan)
Sources to search
Conduct a thorough search
Post search phase
Select studies (screening)
Appraise the quality of the studies
Extract data, synthesise and analyse
Interpret results and write
Guides and manuals
Training and support

General principles

"A good systematic review is based on a well formulated, answerable question. The question guides the review by defining which studies will be included, what the search strategy to identify the relevant primary studies should be, and which data need to be extracted from each study."

A systematic review question needs to be

You may find it helpful to use a search framework, such as those listed below, to help you to refine your research question, but it is not mandatory. Similarly, you may not always need to use every aspect of the framework in order to build a workable research question.

Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews . Ann Intern Med. 1997;127(5):380–387.

To help formulate a focussed research question the PICO tool has been created. PICO is a mnemonic for Population, Intervention, Comparison, and Outcome. These elements have been highlighted to help define the core elements of the question which will be used in the literature search.

The elements of PICO

Population:.

Who or what is the topic of interest, in the health sciences this may be a disease or a condition, in the social sciences this may be a social group with a particular need.

Intervention:

The intervention is the effect or the change upon the population in question. In the health sciences, this could be a treatment, such as a drug, a procedure, or a preventative activity. Depending on the discipline the intervention could be a social policy, education, ban, or legislation.

Comparison:

The comparison is a comparison to the intervention, so if it were a drug it may be a similar drug in which effectiveness is compared. Sometimes the comparator is a placebo or no comparison.

The outcomes in PICO represent the outcomes of interest for the research question. The outcome measures will vary according to the question but will provide the data against which the effectiveness of the intervention is measured.

Examples of using the PICO framework (PDF, 173KB) This document contains worked examples of how to use the PICO search framework as well as other frameworks based on PICO.

Not all systematic review questions are well served by the PICO mnemonic and a number of other models have been created, these include: ECLIPSE (Wildridge & Bell, 2002), SPICE (Booth, 2004), and SPIDER (Cooke, Smith, & Booth, 2012).

Wildridge V, Bell L. How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/ management information . Health Info Libr J. 2002;19(2):113–115.

Booth A. Clear and present questions: formulating questions for evidence based practice . Library Hi Tech. 2006;24(3):355-368.

Cooke A, Smith D, Booth, A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis . Qual Health Res. 2012;22(10):1435–1443.

Remember: you do not have to use a search framework but it can help you to focus your research question and identify the key concepts and terms that you can use in your search. Similarly, you may not need to use all of the elements in your chosen framework, only the ones that are useful for your individual research question.

Using the SPIDER search framework (PDF, 134 KB) This document shows how you can use the SPIDER framework to guide your search.
Using the SPICE search framework (PDF, 134 KB) This document shows how you can use the SPICE framework to guide your search.
Using the ECLIPSE search framework (PDF, 145 KB) This document shows how you can use the ECLIPSE framework to guide your search.
<< Previous: Steps in a systematic review
Next: Create a protocol (plan) >>
Last Updated: Mar 26, 2024 2:54 PM
URL: https://guides.library.uq.edu.au/research-techniques/systematic-reviews

Systematic Reviews: Formulate your question and protocol

Formulate your question and protocol
Developing the review protocol
Searching for evidence
Search strategy
Managing search results
Evaluating results (critical appraisal)
Synthesising and reporting
Further resources

This video illustrates how to use the PICO framework to formulate an effective research question, and it also shows how to search a database using the search terms identified. The database used in this video is CINAHL but the process is very similar in databases from other companies as well.

Formulate Your Question

Having a focused and specific research question is especially important when undertaking a systematic review. If your search question is too broad you will retrieve too many search results and you will be unable to work with them all. If your question is too narrow, you may miss relevant papers. Taking the time to break down your question into separate, focused concepts will also help you search the databases effectively.

Deciding on your inclusion and exclusion criteria early on in the research process can also help you when it comes to focusing your research question and your search strategy.

A literature searching planning template can help to break your search question down into concepts and to record alternative search terms. Frameworks such as PICO and PEO can also help guide your search. A planning template is available to download below, and there is also information on PICO and other frameworks ( Adapted from: https://libguides.kcl.ac.uk/systematicreview/define).

Looking at published systematic reviews can give you ideas of how to construct a focused research question and an effective search strategy.

Example of an unfocused research question: How can deep vein thrombosis be prevented?

Example of a focused research question: What are the effects of wearing compression stockings versus not wearing them for preventing DVT in people travelling on flights lasting at least four hours.

In this Cochrane systematic review by Clarke et al. (2021), publications on randomised trials of compression stockings versus no stockings in passengers on flights lasting at least four hours were gathered. The appendix of the published review contains the comprehensive search strategy used. This research question has focused on a particular method (wearing compression stockings) in a particular setting (flights of at least 4 hrs) and included only specific studies (randomised trails). An additional way of focusing a question could be to look at a particular section of the population.

Clarke M. J., Broderick C., Hopewell S., Juszczak E., and Eisinga A., 20121. Compression stockings for preventing deep vein thrombosis in airline passengers. Cochrane Database of Systematic Reviews 2021, Issue 4. Art. No.: CD004002 [Accessed 30th April 2021]. Available from: 10.1002/14651858.CD004002.pub4

There are many different frameworks that you can use to structure your research question with clear parameters. The most commonly used framework is PICO:

Population This could be the general population, or a specific group defined by: age, socioeconomic status, location and so on.
Intervention This is the therapy/test/strategy to be investigated and can include medication, exercise, environmental factors, and counselling for example. It may help to think of this as 'the thing that will make a difference'.
Comparator This is a measure that you will use to compare results against. This can be patients who received no treatment or a placebo, or people who received alternative treatment/exposure, for instance.
Outcome What outcome is significant to your population or issue? This may be different from the outcome measures used in the studies.

Adapted from: https://libguides.reading.ac.uk/systematic-review/protocol

Developing an efficient search strategy using PICO A tool created by Health Evidence to help construct a search strategy using PICO

Other Frameworks: alternatives to PICO

As well as PICO, there are other frameworks available, for instance:

PICOT : Population, Intervention, Comparison, Outcome, Time.
PEO: Population and/or Problem, Exposures, Outcome
SPICE: Setting, Population or Perspective, Intervention, Comparison, Evaluation
ECLIPS: Expectations, Client Group, Location, Impact, Professionals Involved, Service
SPIDER: Sample, Phenomenon of interest, Design, Evaluation, Research type

This page from City, University of London, contains useful information on several frameworks, including the ones listed above.

Develop Your Protocol

Atfer you have created your research question, the next step is to develop a protocol which outlines the study methodology. You need to include the following:

Research question and aims
Criteria for inclusion and exclusion
search strategy
selecting studies for inclusion
quality assessment
data extraction & analysis
synthesis of results
dissemination

To find out how much has been published on a particular topic, you can perform scoping searches in relevant databases. This can help you decide on the time limits of your study.

Systematic review protocol template This template from the University of Reading can help you plan your protocol.
Protocol Guidance This document from the University of York describes what each element of your protocol should cover.

Register Your Protocol

It is good practice to register your protocol and often this is a requirement for future publication of the review.

You can register your protocol here:

PROSPERO: international prospective register of systematic review
Cochrane Collaboration, Getting Involved
Campbell Collaboration, Co-ordinating Groups

Adapted from: https://libguides.bodleian.ox.ac.uk/systematic-reviews/methodology

<< Previous: Home
Next: Developing the review protocol >>
Last Updated: Sep 12, 2023 5:29 PM
URL: https://libguides.qmu.ac.uk/systematic-reviews

Systematic Reviews

Introduction
Review Process: Step by Step
1. Planning a Review

Question Guidance

Research question frameworks, specifying your criteria.

3. Standards & Protocols
4. Search Terms & Strategies
5. Locating Published Research
6. Locating Grey Literature
7. Managing & Documenting Results
8. Selecting & Appraising Studies
9. Extracting Data
10. Writing a Systematic Review
Tools & Software
Guides & Tutorials
Accessing Resources
Research Assistance

The first and most important decision in preparing a systematic review is to determine its focus. This is best done by clearly framing the questions the review seeks to answer.

Systematic reviews should address answerable questions and fill important gaps in knowledge.
Developing good review questions takes time, expertise and engagement with intended users of the review.
Cochrane Reviews can focus on broad questions, or be more narrowly defined. There are advantages and disadvantages of each.
Logic models are a way of documenting how interventions, particularly complex interventions, are intended to ‘work’, and can be used to refine review questions and the broader scope of the review.
Using priority-setting exercises, involving relevant stakeholders, and ensuring that the review takes account of issues relating to equity can be strategies for ensuring that the scope and focus of reviews address the right questions.

From Chapter 2 of the Cochrane Handbook for Systematic Reviews of Interventions . This chapter provides detailed guidance for developing a research question.

PICO Concepts
PICO Example
PECO Example
Other Search Frameworks
Selecting a Framework
Review Typology & Frameworks

As you consider the scope of your research, think about how you will define these concepts:

Population / Problem: who are you screening? Why?
Intervention: what are you evaluating? e.g., a treatment, an intervention, etc.
Comparison: are you comparing this group to another group, e.g. a placebo group?
Outcome: what are the outcomes? Is there a specific one you are looking at?

Qualitative PICo

Population / Problem
Phenomenon of Interest

PICO variations

PEO: Exposure
PICOT: Timeframe
PICOTS: Timeframe, Setting
PICOS: Study Design , e.g. cohorts or randomized controlled trials.

(From Lackey, M. (2013). Systematic reviews: Searching the literature [PowerPoint slides].

In adults , is screening for depression and feedback of results to providers more effective than no screening and feedback in improving outcomes of major depression in primary care settings ?

Population / Problem : adults / depression (major)
Intervention : screening, feedback
Comparison : none
Outcome: no particular outcomes specified

(From Lackey, M. (2013). Systematic reviews: Searching the literature [PowerPoint slides]. Retrieved from http://guides.lib.unc.edu/ld.php?content_id=258919 )

Table 1. Five paradigmatic approaches and examples for identifying the exposure and comparator in systematic review and decision-making questions.

dB: decibel ; PECO: population, exposure, comparator, outcome(s).
a. Cut-off is a broad term referring to thresholds, levels, durations, means, medians, or ranges of exposure.

Morgan, R. L., Whaley, P., Thayer, K. A., & Schünemann, H. J. (2018). Identifying the PECO: a framework for formulating good questions to explore the association of environmental and other exposures with health outcomes . Environment international , 121 (Pt 1), 1027.

As well as PICO there are many other frameworks for conceptualizing your question:

BeHEMoTh- Behavior of interest, Health context, Exclusions, Models or Theories

Booth, A., & Carroll, C. (2015). Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable?. Health Information & Libraries Journal, 32(3), 220-235.

ECLIPSE- Expectation/Client group/Location/Impact/Professionals/Service (Evaluating services)

Wildridge, V., & Bell, L. (2002). How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information. Health Information and Libraries Journal, 19(2), 113-115. doi: 10.1046/j.1471-1842.2002.00378.x

FINER- Feasibility, Interesting, Novel, Ethical, Relevant

Cummings, S. R., Browner, W. S., & Hulley, S. B. (1988) Conceiving the research question. In S. B. Hulley, & S. R. Cummings SR (Eds), Designing Clinical Research. (pp. 12 - 17). Baltimore, MD: Williams & Wilkins

SPIDER- Sample/Phenomenon of Interest/Design/Evaluation/Research type (Qualitative studies, especially with samples rather than populations)

Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative Health Research, 22(10), 1435-1443. doi: 10.1177/1049732312452938

SPICE- Setting/Perspective (or Population)/Intervention/Comparison/Evaluation (Evaluating outcomes of a specific intervention)

Maryse C. Kok, Hermen Ormel, Jacqueline E. W. Broerse, Sumit Kane, Ireen Namakhoma, Lilian Otiso, Moshin Sidat, Aschenaki Z. Kea, Miriam Taegtmeyer, Sally Theobald, Marjolein Dieleman. (2017) Optimising the benefits of community health workers’ unique position between communities and the health sector: A comparative analysis of factors shaping relationships in four countries. Global Public Health 12:11, pages 1404-1432.

From Monash University Library

From James Cook University Library

From: Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences . BMC medical research methodology , 18 (1), 1-9.

Review typology chart with framework suggestions and examples

What framework can be applied to our example article?

Dhillon, J., Jacobs, A. G., Ortiz, S., & Rios, L. (2022). A Systematic Review of Literature On the Representation of Racial and Ethnic Minority Groups in Clinical Nutrition Interventions. Advances in nutrition (Bethesda, Md.) . Advance online publication. https://doi.org/10.1093/advances/nmac002

Chapter 3 of the Cochrane Handbook for Systematic Reviews of Interventions , Defining the criteria for including studies and how they will be grouped for the synthesis provides detailed guidance for developing a research question.

Predefined, unambiguous eligibility criteria are a fundamental prerequisite for a systematic review.
The criteria for considering types of people included in studies in a review should be sufficiently broad to encompass the likely diversity of studies, but sufficiently narrow to ensure that a meaningful answer can be obtained when studies are considered in aggregate.
Considerations when specifying participants include setting, diagnosis or definition of condition and demographic factors.

Criteria Considerations:

The population, intervention and comparison components of the question, with the additional specification of types of study that will be included, form the basis of the pre-specified eligibility criteria for the review. It is rare to use outcomes as eligibility criteria...
Cochrane Reviews should include all outcomes that are likely to be meaningful and not include trivial outcomes. Critical and important outcomes should be limited in number and include adverse as well as beneficial outcomes.
Review authors should plan at the protocol stage how the different populations, interventions, outcomes and study designs within the scope of the review will be grouped for analysis.

Criteria to Specify:

Language of publication (ideally this would not be limited)
Year or year range of publications
How is the disease/condition defined?
What are the most important characteristics that describe these people (participants)?
Are there any relevant demographic factors (e.g. age, sex, ethnicity)?
What is the setting (e.g. hospital, community, etc)?
Who should make the diagnosis?
Are there other types of people who should be excluded from the review (because they are likely to react to the intervention in a different way)?
How will studies involving only a subset of relevant participants be handled?
Types of intervention
Data type/study type
Study design
<< Previous: 1. Planning a Review
Next: 3. Standards & Protocols >>
Last Updated: Feb 26, 2024 2:04 PM
URL: https://libguides.ucmerced.edu/systematic-reviews

Help and Support
Research Guides

Systematic Reviews - Research Guide

Defining your review question
Starting a Systematic Review
Developing your search strategy
Where to search
Appraising Your Results
Documenting Your Review
Find Systematic Reviews
Software and Tools for Systematic Reviews
Guidance for conducting systematic reviews by discipline
Library Support

Review question

A systematic review aims to answer a clear and focused clinical question. The question guides the rest of the systematic review process. This includes determining inclusion and exclusion criteria, developing the search strategy, collecting data and presenting findings. Therefore, developing a clear, focused and well-formulated question is critical to successfully undertaking a systematic review.

A good review question:

allows you to find information quickly
allows you to find relevant information (applicable to the patient) and valid (accurately measures stated objectives)
provides a checklist for the main concepts to be included in your search strategy.

How to define your systematic review question and create your protocol

Starting the process
Defining the question
Creating a protocol

Types of clinical questions

PICO/PICo framework
Other frameworks

Research topic vs review question

A research topic is the area of study you are researching, and the review question is the straightforward, focused question that your systematic review will attempt to answer.

Developing a suitable review question from a research topic can take some time. You should:

perform some scoping searches
use a framework such as PICO
consider the FINER criteria; review questions should be F easible, I nteresting, N ovel, E thical and R elevant
check for existing or prospective systematic reviews.

When considering the feasibility of a potential review question, there should be enough evidence to answer the question whilst ensuring that the quantity of information retrieved remains manageable. A scoping search will aid in defining the boundaries of the question and determining feasibility.

For more information on FINER criteria in systematic review questions, read Section 2.1 of the Cochrane Handbook .

Check for existing or prospective systematic reviews

Before finalising your review question, you should determine if any other systematic review is in progress or has been completed on your intended question (i.e. consider if the review is N ovel).

To find systematic reviews you might search specialist resources such as the Cochrane Library , Joanna Briggs Institute EBP Database or the Campbell Collaboration . "Systematic review" can also be used as a search term or limit when searching the recommended databases .

You should appraise any systematic reviews you find to assess their quality. An article may include ‘systematic review’ in its title without correctly following the systematic review methodology. Checklists, including those developed by AMSTAR and JBI , are useful tools for appraisal.

You may undertake a review on a similar question if that posed by a previously published review had issues with its methodology such as not having a comprehensive search strategy, for example. You may choose to narrow the parameters of a previously conducted search or to update the review if it was published some years ago.

Searching a register of prospective systematic reviews such as PROSPERO will allow you to check that you are not duplicating research already underway.

Once you have performed scoping searches and checked for other systematic reviews on your topic, you can focus and refine your review question. Any PICO elements identified during the initial development of the review question from the research topic should now be further refined.

The review question should always be:

unambiguous
structured.

Review questions may be broad or narrow in focus; however, you should consider the FINER criteria when determining the breadth of the PICO elements of your review question.

A question that is too broad may present difficulty with searching, data collection, analysis, and writing, as the number of studies retrieved would be unwieldy. A broad review question could be more suited to another type of review .

A question that is too narrow may not have enough evidence to allow you to answer your review question. Table 2.3.a in the Cochrane Handbook summarises the advantages and disadvantages of broad versus narrow reviews and provides examples of how you could broaden or narrow different PICO elements.

It is essential to formulate your research question with care to avoid missing relevant studies or collecting a potentially biased result set.

A systematic review protocol is a document that describes the rationale, question, and planned methods of a systematic review. Creating a protocol is an essential part of the systematic review process, ensuring careful planning and detailed documentation of what is planned before undertaking the review.

The Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P) checklist outlines recommended items to address in a systematic review protocol, including:

review question, with PICO elements defined
eligibility criteria
information sources (e.g. planned databases, trial registers, grey literature sources, etc.)
draft search strategy.

Systematic reviews must have pre-specified criteria for including and excluding studies in the review. The Cochrane Handbook states that "predefined, unambiguous eligibility criteria are a fundamental prerequisite for a systematic review."

The first step in developing a protocol is determining the PICO elements of the review question and how the intervention produces the expected outcomes in the specified population. You should then specify the types of studies that will provide the evidence to answer your review question. Then outline the inclusion and exclusion criteria based on these PICO elements.

For more information on defining eligibility criteria, see Chapter 3 of the Cochrane Handbook .

A key purpose of a protocol is to make plans to minimise bias in the findings of the review; where possible, changes should not be made to the eligibility criteria of a published protocol. Where such changes are made, they must be justified and documented in the review. Appropriate time and consideration should be given to creating the protocol.

You may wish to register your protocol in a publicly accessible way. This will help prevent other people from completing a review on your topic.

If you intend to publish a systematic review in the health sciences, it should conform to the IOM Standards for Reporting Systematic Reviews .

If you intend to publish a systematic review in the Cochrane Database of Systematic Reviews , it should conform to the Methodological Expectations in Cochrane Intervention Review s (MECIR).

A clinical question needs to be directly relevant to the patient or problem and phrased to facilitate the search for an answer. A clear and focused question is more likely to lead to a credible and useful answer, whereas a poorly formulated question can lead to an uncertain answer and create confusion.

The population and intervention should be specific, but if any or both are described too narrowly, it may not be easy to find relevant studies or sufficient data to demonstrate a reliable answer.

PICO is a framework for developing a focused clinical question.

Slightly different versions of this concept are used to search for quantitative and qualitative reviews, examples are given below:

PICO for quantitative studies

Here is an example of a clinical question that outlines the PICO components:

PICo for qualitative studies

Here is an example of a clinical question that outlines the PICo components:

Two other mnemonics may be used to frame questions for qualitative and quantitative studies - SPIDER and SPICE .

SPIDER for qualitative or quantitative studies

SPIDER can be used for both qualitative and quantitative studies:

Within social sciences research, SPICE may be more appropriate for formulating research questions:

Systematic Reviews: Develop a Research Question

Getting Started
Develop a Research Question
Create a Protocol
Search for Literature
Conduct Screening
Appraise & Synthesize
Report Results
Types of Reviews Chart Page
Data Management Page

Step 1: Develop a Research Question

The first step in the systematic review process is to create a focused, well-defined research question. The question needs to be structured using a framework such as PICO which stands for Patient, Intervention, Comparison, and Outcome. There are many other frameworks available, use the best one to fit your topic. Please review all steps before starting your review.

Develop Questions Based on Review Type

Systematic Reviews
Scoping Reviews
Other Review Types

Developing systematic review questions

Systematic review questions usually follow a framework (e.g. PICO, SPIDER, Eclipse, COPES). These questions are specific and usually are comparative. Always evaluating quality. Usually going for a practice change. Your question will clearly define your search variables.

Example Health Sciences Question:

Are intranasal steroids or oral antihistamines better at controlling allergic rhinitis in adults?

Example Social Sciences Question:

Is meditation with peers or meditation with adults more effective in reducing incidence of bullying within K-12 students?

Developing scoping review questions:

The questions are much more broad in context. They tend to answer questions of gaps in or mapping the existing literature. When developing a scoping review question you must clearly defined your variables.

How does hearing loss affect physical fitness or sport engagement?

What are the ethical considerations in online ethnographic research with military populations?

Learn about other review types:

Types of Reviews Chart outlining characteristics of different types of reviews.
What Type of Review Is Right for You? Decision tree for selecting types of reviews and descriptions of each. Provided by Cornell University Library

Foundational Literature on Methodologies

Hilary Arksey & Lisa O'Malley Scoping studies: towards a methodological framework . International Journal of Social Research Methodology, 8:1, 19-32, DOI: 10.1080/1364557032000119616
Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology . Implement Sci. 2010 Sep 20;5:69. doi: 10.1186/1748-5908-5-69. PMID: 20854677; PMCID: PMC2954944.
Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach . BMC Med Res Methodol . 2018;18(1):143. Published 2018 Nov 19. doi:10.1186/s12874-018-0611-x
Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences . BMC Med Res Methodol . 2018;18(1):5. Published 2018 Jan 10. doi:10.1186/s12874-017-0468-4
Siddaway AP, Wood AM, Hedges LV. How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses . Annu Rev Psychol . 2019;70:747-770. doi:10.1146/annurev-psych-010418-102803
Sutton A, Clowes M, Preston L, Booth A. Meeting the review family: exploring review types and associated information retrieval requirements. Health Info Libr J. 2019 Sep;36(3):202-222. doi: 10.1111/hir.12276. PMID: 31541534.

Standards & Guidelines

PICO & Question Types Learn about PICO and the various question types. In addition, view some PICO question templates and examples.
Cochrane Handbook of Systematic Reviews Part 2: Core Methods,Chapter 2: Determining the scope and questions
Finding What Works in Health Care: Standards for Systematic Reviews Recommended standards for formulating the topic.

Searching the Grey Literature: a Handbook for Searching Reports, Working Papers, and Other Unpublished Research

Umbrella Reviews: Evidence Synthesis with Overviews of Reviews and Meta-Epidemiologic Studies

Cochrane Handbook for Systematic Reviews of Interventions

Comprehensive Systematic Review for Advanced Practice Nursing, Second Edition

<< Previous: Getting Started
Next: Create a Protocol >>
Last Updated: Jan 12, 2024 3:52 PM
URL: https://libguides.uky.edu/systematicreview

Systematic Review

Library Help
What is a Systematic Review (SR)?

Steps of a Systematic Review

Framing a Research Question
Developing a Search Strategy
Searching the Literature
Managing the Process
Meta-analysis
Publishing your Systematic Review

Forms and templates

Image: David Parmenter's Shop

PICO Template
Inclusion/Exclusion Criteria
Database Search Log
Review Matrix
Cochrane Tool for Assessing Risk of Bias in Included Studies

• PRISMA Flow Diagram - Record the numbers of retrieved references and included/excluded studies. You can use the Create Flow Diagram tool to automate the process.

• PRISMA Checklist - Checklist of items to include when reporting a systematic review or meta-analysis

PRISMA 2020 and PRISMA-S: Common Questions on Tracking Records and the Flow Diagram

PROSPERO Template
Manuscript Template
Steps of SR (text)
Steps of SR (visual)
Steps of SR (PIECES)

Adapted from A Guide to Conducting Systematic Reviews: Steps in a Systematic Review by Cornell University Library

Source: Cochrane Consumers and Communications (infographics are free to use and licensed under Creative Commons )

Check the following visual resources titled " What Are Systematic Reviews?"

Video with closed captions available
Animated Storyboard
<< Previous: What is a Systematic Review (SR)?
Next: Framing a Research Question >>
Last Updated: Mar 4, 2024 12:09 PM
URL: https://lib.guides.umd.edu/SR

Systematic Reviews: Formulating Your Research Question

What Type of Review is Right for You?
What is in a Systematic Review
Finding and Appraising Systematic Reviews
Formulating Your Research Question
Inclusion and Exclusion Criteria
Creating a Protocol
Results and PRISMA Flow Diagram
Searching the Published Literature
Searching the Gray Literature
Methodology and Documentation
Managing the Process
Scoping Reviews

Types of Questions

Research questions should be answerable and also fill important gaps in the knowledge. Developing a good question takes time and may not fit in the traditional framework. Questions can be broad or narrow and there are advantages and disadvantages to each type.

Questions can be about interventions, diagnosis, screening, measuring, patients/student/customer experiences, or even management strategies. They can also be about policies. As the field of systematic reviews grow, more and more people in humanities and social sciences are embracing systematic reviews and creating questions that fit within their fields of practice.

More information can be found here:

Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S. Chapter 2: Determining the scope of the review and the questions it will address. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook .

Frameworks are used to develop the question being asked. The type of framework doesn't matter as much as the question being selected.

Think of these frameworks as you would for a house or building. A framework is there to provide support and to be a scaffold for the rest of the structure. In the same way, a research question framework can also help structure your evidence synthesis question.

Organizing Your Question

Formulating non-PICO questions Although the PICO formulation should apply easily to the majority of effectiveness questions and a great number besides you may encounter questions that are not easily accommodated within this particular framework. Below you will find a number of acceptable alternatives:
Using The PICOS Model To Design And Conduct A Systematic Search: A Speech Pathology Case Study
7 STEPS TO THE PERFECT PICO SEARCH Searching for high-quality clinical research evidence can be a daunting task, yet it is an integral part of the evidence-based practice process. One way to streamline and improve the research process for nurses and researchers of all backgrounds is to utilize the PICO search strategy. PICO is a format for developing a good clinical research question prior to starting one’s research. It is a mnemonic used to describe the four elements of a sound clinical foreground question (Yale University’s Cushing/Whitney Medical Library)

to search for quantitative review questions

P: Patient or Population

I: Intervention (or Exposure)

C: Comparison (or Control)

Variations Include:

S: Study Design

T: Timeframe

to search for qualitative evidence

S: Setting (where?)

P: Perspecitve (for whom?)

I: Intervention (what?)

C: Comparison (compared with what?)

E: Evaluation (with what result?)

to search for qualitative and mixed methods research studies

S: Sample

PI: Phenomenon of Interest

E: Evaluation

R: Research type

to search for health policy/management information

E: Expectation (improvement or information or innovation)

C: Client group (at whom the service is aimed)

L: Location (where is the service located?)

I: Impact (outcomes)

P: Professionals (who is involved in providing/improving the service)

Se: Service (for which service are you looking for information)

PICO Template Questions

Try words from your topic in these templates. Your PICO should fit only one type of question in the list.

For an intervention/therapy:

In _______(P), what is the effect of _______(I) on ______(O) compared with _______(C) within ________ (T)?

For etiology:

Are ____ (P) who have _______ (I) at ___ (Increased/decreased) risk for/of_______ (O) compared with ______ (P) with/without ______ (C) over _____ (T)?

Diagnosis or diagnostic test:

Are (is) _________ (I) more accurate in diagnosing ________ (P) compared with ______ (C) for _______ (O)?

Prevention:

For ________ (P) does the use of ______ (I) reduce the future risk of ________ (O) compared with _________ (C)?

Prognosis/Predictions

In__________ (P) how does ________ (I) compared to _______(C) influence _______ (O) over ______ (T)?

How do ________ (P) diagnosed with _______ (I) perceive ______ (O) during _____ (T)?

Template taken from Southern Illinois University- Edwardsville

Example PICO Questions

Intervention/Therapy:

In school-age children (P), what is the effect of a school-based physical activity program (I) on a reduction in the incidence of childhood obesity (O) compared with no intervention (C) within a 1 year period (T)?

In high school children (P), what is the effect of a nurse-led presentation on bullying (I) on a reduction in reported incidences of bullying (O) compared with no intervention (C) within a 6 month time frame (T)?

Are males 50 years of age and older (P) who have a history of 1 year of smoking or less (I) at an increased risk of developing esophageal cancer (O) compared with males age 50 and older (P) who have no smoking history (C)?

Are women ages 25-40 (P) who take oral contraceptives (I) at greater risk for developing blood clots (O) compared with women ages 25-40 (P) who use IUDs for contraception (C) over a 5 year time frame (T)?

Diagnosis/Diagnostic Test:

Is a yearly mammogram (I) more effective in detecting breast cancer (O) compared with a mammogram every 3 years (C) in women under age 50 (P)?

Is a colonoscopy combined with fecal occult blood testing (I) more accurate in detecting colon cancer (O) compared with a colonoscopy alone (C) in adults over age 50 (P)?

For women under age 60 (P), does the daily use of 81mg low-dose Aspirin (I) reduce the future risk of stroke (O) compared with no usage of low-dose Aspirin (C)?

For adults over age 65 (P) does a daily 30 minute exercise regimen (I) reduce the future risk of heart attack (O) compared with no exercise regimen (C)?

Prognosis/Predictions:

Does daily home blood pressure monitoring (I) influence compliance with medication regimens for hypertension (O) in adults over age 60 who have hypertension (P) during the first year after being diagnosed with the condition (T)?

Does monitoring blood glucose 4 times a day (I) improve blood glucose control (O) in people with Type 1 diabetes (P) during the first six months after being diagnosed with the condition (T)?

How do teenagers (P) diagnosed with cancer (I) perceive chemotherapy and radiation treatments (O) during the first 6 months after diagnosis (T)?

How do first-time mothers (P) of premature babies in the NICU (I) perceive bonding with their infant (O) during the first month after birth (T)?

<< Previous: Finding and Appraising Systematic Reviews
Next: Inclusion and Exclusion Criteria >>
Last Updated: Mar 5, 2024 3:43 PM
URL: https://guides.lib.lsu.edu/Systematic_Reviews

Provide Website Feedback Accessibility Statement

Covidence website will be inaccessible as we upgrading our platform on Monday 23rd August at 10am AEST, / 2am CEST/1am BST (Sunday, 15th August 8pm EDT/5pm PDT)

How to formulate the review question using PICO. 5 steps to get you started.

Home | Blog | How To | How to formulate the review question using PICO. 5 steps to get you started.

Covidence covers five key steps to formulate your review question using PICO

You’ve decided to go ahead. You have identified a gap in the evidence and you know that conducting a systematic review, with its explicit methods and replicable search, is the best way to fill it – great choice 🙌.

The review will produce useful information to enable informed decision-making and to improve patient care. Your review team’s first job is to capture exactly what you need to know in a well-formulated review question.

At this stage there is a lot to plan. You might be recruiting people to your review team, thinking about the time-frame for completion and considering what software to use. It’s tempting to get straight on to the search for studies 🏃.

Take it slowly: it’s vital to get the review question right. A clear and precise question will ensure that you gather the appropriate data to answer your question. Time invested up-front to consider every aspect of the question will pay off once the review is underway. The review question will shape all the subsequent stages in the review, particularly setting the criteria for including and excluding studies, the search strategy, and the way you choose to present the results. So it’s worth taking the time to get this right!

Let’s take a look at five key steps in formulating the question for a standard systematic review of interventions. It’s a process that requires careful thought from a range of stakeholders and meticulous planning. But what if, once you have started the review, you find that you need to tweak the question anyway? Don’t worry, we’ll cover that too ✅.

📌 Consider the audience of the review

Who will use this review? What do they want to know? How do they measure effectiveness? Good review teams partner with the people who will use the evidence and make sure that their research plan (or protocol) asks a question that is relevant and important for patients.

📌 Think about what you already know

How much do you need to know about the topic area at this stage? Ideally, enough to come up with a relevant, useful question but not so much that your knowledge influences the way in which you phrase it. Why? Because setting a review question when you are already familiar with the data can introduce bias by allowing you to direct the question in favour of achieving a particular result. In practice, the review team is very likely to have some knowledge of relevant studies and some preconceived ideas about how the treatments work. That’s fine – and it’s useful – but it’s also good practice to recognise the influence this knowledge and these ideas might have on the choice of question. Issues of bias will come up again as we work through the rest of these steps.

If not enough is known about the subject area to ask a useful question, you might undertake a scoping review . This is a separate exercise from a systematic review and is sometimes used by researchers to map the literature and highlight gaps in the evidence before they start work on a systematic review.

📌 Use a framework

Faced with a heady mixture of concepts, ideas, aims and outcomes, researchers in every field have come up with question frameworks (and some great backronyms ) to help them. Question frameworks impose order on a complex thought process by breaking down a question into its component parts. A commonly used framework in clinical medicine is PICO:

👦 P opulation (or patients) refers to the characteristics of the people that you want to study. For example, the review might look at children with nocturnal enuresis.

💊 I ntervention is the treatment you are investigating. For example, the review might look at the effectiveness of enuresis alarms.

💊 C omparison, if you decide to use one, is the treatment you want to compare the intervention with. For example, the review might look at the effectiveness of enuresis alarms versus the effectiveness of drug therapy.

📏 O utcomes are the measures used to assess the effectiveness of the treatment. It’s particularly important to select outcomes that matter to the end users of the review. In this example, a useful outcome might be bedwetting. (Helpfully, some clinical areas use standardised sets of outcomes in their clinical trials to facilitate the comparison of data between studies 👏.)

But back to bedwetting. In our example, a PICO review question would look something like this:

“In children with nocturnal enuresis (population), how effective are alarms (intervention) versus drug treatments (comparison) for the prevention of bedwetting (outcome)?”

PICO is suitable for reviews of interventions. If you plan to review prognostic or qualitative data, or diagnostic test accuracy, PICO is unlikely to be a suitable framework for your question. In Covidence you can save your PICO for easy reference throughout the screening, extraction and quality assessment phases of your review.

📌 Set the scope

The scope of a review question requires careful thought. To answer the example PICO question above, the review would compare one treatment (alarms) with another (drug therapy). A broader question might consider all the available treatments for nocturnal enuresis in children. The broad scope of this question would still allow the review team to drill down and separate the data into groups of specific treatments later in the review process. And to minimise bias, the intended grouping of data would be pre-specified and justified in the protocol or research plan.

Broader systematic reviews are great because they summarise all the evidence on a given topic in one place. A potential disadvantage is that they can produce a large volume of data that is difficult to manage.

If the size of the review has started to escalate beyond your comfort zone, you might consider narrowing the scope. This can make the size of the review more manageable, both for the review team and for the reader. But it’s worth examining the motivations for narrowing the scope more closely. Suppose we wanted to define a smaller population in the example PICO question. Is there a good reason (other than to reduce the review team’s workload) to restrict the population to boys with nocturnal enuresis? Or to children under 10 years old? On the basis of what is already known, could the treatment effect be expected to differ by sex or age of the study participants? 🤔 Be prepared to explain your choices and to demonstrate that they are legitimate.

Some reviews with a narrow scope retrieve only a small number of studies. If this happens, there is a risk that the data collected from these studies might not be enough to produce a useful synthesis or to guide decision-making. It can be frustrating for review teams who have spent time defining the question, planning the methods, and conducting an extensive search to find that their question is unanswerable. This is another reason why it is useful for the review team to have prior knowledge of the subject area and some familiarity with the existing evidence. The Cochrane Handbook contains some useful contingencies for dealing with sparse data .

Covidence can help review teams to save time whatever the scope and size of the review. In Covidence, data can be grouped to the review team’s exact specification for seamless export into data analysis software. The intuitive workflow makes collaboration simple so if one reviewer spots a problem, they can alert the rest of the team quickly and easily.

📌 Adjust if necessary

Systematic reviews follow explicit, pre-specified methods. So it’s no surprise to learn that the review question needs to be considered carefully and explained in detail before the review gets underway. But what about the unknown unknowns – those issues that the review teams will have to deal with later in the process but that they cannot foresee at the outset, no matter how much time they spend on due diligence?

Clearly, reviews need the agility to control for issues that the project plan did not anticipate – strict adherence to the pre-specified process when a good reason to deviate has come to light would carry its own risks for the quality of the review. So if an initial scan of, for example, the search results indicates that it would be sensible to modify the question, this can be done. The research plan might make explicit the process for dealing with these types of changes. It might also contain plans for sensitivity analysis , to examine whether these choices have any effect on the findings of the review. As mentioned above with regard to scope, it might be difficult to defend a data-driven change to the question. And as before, the issue is the risk of bias and the danger of producing a spurious result.

(Figure 4. Image from Eshun‐Wilson I, Siegfried N, Akena DH, Stein DJ, Obuku EA, Joska JA. Antidepressants for depression in adults with HIV infection. Cochrane Database of Systematic Reviews 2018, Issue 1. Art. No.: CD008525. DOI: 10.1002/14651858.CD008525.pub3. Accessed 27 May 2021.)

This blog post is part of the Covidence series on how to write a systematic review.

Laura Mellor. Portsmouth, UK

Perhaps you'd also like....

Top 5 Tips for High-Quality Systematic Review Data Extraction

Data extraction can be a complex step in the systematic review process. Here are 5 top tips from our experts to help prepare and achieve high quality data extraction.

How to get through study quality assessment Systematic Review

Find out 5 tops tips to conducting quality assessment and why it’s an important step in the systematic review process.

How to extract study data for your systematic review

Learn the basic process and some tips to build data extraction forms for your systematic review with Covidence.

Better systematic review management

Head office, working for an institution or organisation.

Find out why over 350 of the world’s leading institutions are seeing a surge in publications since using Covidence!

Request a consultation with one of our team members and start empowering your researchers:

By using our site you consent to our use of cookies to measure and improve our site’s performance. Please see our Privacy Policy for more information.

Jump to navigation

Cochrane Training

Chapter 1: starting a review.

Toby J Lasserson, James Thomas, Julian PT Higgins

Key Points:

Systematic reviews address a need for health decision makers to be able to access high quality, relevant, accessible and up-to-date information.
Systematic reviews aim to minimize bias through the use of pre-specified research questions and methods that are documented in protocols, and by basing their findings on reliable research.
Systematic reviews should be conducted by a team that includes domain expertise and methodological expertise, who are free of potential conflicts of interest.
People who might make – or be affected by – decisions around the use of interventions should be involved in important decisions about the review.
Good data management, project management and quality assurance mechanisms are essential for the completion of a successful systematic review.

Cite this chapter as: Lasserson TJ, Thomas J, Higgins JPT. Chapter 1: Starting a review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook .

1.1 Why do a systematic review?

Systematic reviews were developed out of a need to ensure that decisions affecting people’s lives can be informed by an up-to-date and complete understanding of the relevant research evidence. With the volume of research literature growing at an ever-increasing rate, it is impossible for individual decision makers to assess this vast quantity of primary research to enable them to make the most appropriate healthcare decisions that do more good than harm. By systematically assessing this primary research, systematic reviews aim to provide an up-to-date summary of the state of research knowledge on an intervention, diagnostic test, prognostic factor or other health or healthcare topic. Systematic reviews address the main problem with ad hoc searching and selection of research, namely that of bias. Just as primary research studies use methods to avoid bias, so should summaries and syntheses of that research.

A systematic review attempts to collate all the empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman et al 1992, Oxman and Guyatt 1993). Systematic review methodology, pioneered and developed by Cochrane, sets out a highly structured, transparent and reproducible methodology (Chandler and Hopewell 2013). This involves: the a priori specification of a research question; clarity on the scope of the review and which studies are eligible for inclusion; making every effort to find all relevant research and to ensure that issues of bias in included studies are accounted for; and analysing the included studies in order to draw conclusions based on all the identified research in an impartial and objective way.

This Handbook is about systematic reviews on the effects of interventions, and specifically about methods used by Cochrane to undertake them. Cochrane Reviews use primary research to generate new knowledge about the effects of an intervention (or interventions) used in clinical, public health or policy settings. They aim to provide users with a balanced summary of the potential benefits and harms of interventions and give an indication of how certain they can be of the findings. They can also compare the effectiveness of different interventions with one another and so help users to choose the most appropriate intervention in particular situations. The primary purpose of Cochrane Reviews is therefore to inform people making decisions about health or health care.

Systematic reviews are important for other reasons. New research should be designed or commissioned only if it does not unnecessarily duplicate existing research (Chalmers et al 2014). Therefore, a systematic review should typically be undertaken before embarking on new primary research. Such a review will identify current and ongoing studies, as well as indicate where specific gaps in knowledge exist, or evidence is lacking; for example, where existing studies have not used outcomes that are important to users of research (Macleod et al 2014). A systematic review may also reveal limitations in the conduct of previous studies that might be addressed in the new study or studies.

Systematic reviews are important, often rewarding and, at times, exciting research projects. They offer the opportunity for authors to make authoritative statements about the extent of human knowledge in important areas and to identify priorities for further research. They sometimes cover issues high on the political agenda and receive attention from the media. Conducting research with these impacts is not without its challenges, however, and completing a high-quality systematic review is often demanding and time-consuming. In this chapter we introduce some of the key considerations for potential review authors who are about to start a systematic review.

1.2 What is the review question?

Getting the research question right is critical for the success of a systematic review. Review authors should ensure that the review addresses an important question to those who are expected to use and act upon its conclusions.

We discuss the formulation of questions in detail in Chapter 2 . For a question about the effects of an intervention, the PICO approach is usually used, which is an acronym for Population, Intervention, Comparison(s) and Outcome. Reviews may have additional questions, for example about how interventions were implemented, economic issues, equity issues or patient experience.

To ensure that the review addresses a relevant question in a way that benefits users, it is important to ensure wide input. In most cases, question formulation should therefore be informed by people with various relevant – but potentially different – perspectives (see Chapter 2, Section 2.4 ).

1.3 Who should do a systematic review?

Systematic reviews should be undertaken by a team. Indeed, Cochrane will not publish a review that is proposed to be undertaken by a single person. Working as a team not only spreads the effort, but ensures that tasks such as the selection of studies for eligibility, data extraction and rating the certainty of the evidence will be performed by at least two people independently, minimizing the likelihood of errors. First-time review authors are encouraged to work with others who are experienced in the process of systematic reviews and to attend relevant training.

Review teams must include expertise in the topic area under review. Topic expertise should not be overly narrow, to ensure that all relevant perspectives are considered. Perspectives from different disciplines can help to avoid assumptions or terminology stemming from an over-reliance on a single discipline. Review teams should also include expertise in systematic review methodology, including statistical expertise.

Arguments have been made that methodological expertise is sufficient to perform a review, and that content expertise should be avoided because of the risk of preconceptions about the effects of interventions (Gøtzsche and Ioannidis 2012). However, it is important that both topic and methodological expertise is present to ensure a good mix of skills, knowledge and objectivity, because topic expertise provides important insight into the implementation of the intervention(s), the nature of the condition being treated or prevented, the relationships between outcomes measured, and other factors that may have an impact on decision making.

A Cochrane Review should represent an independent assessment of the evidence and avoiding financial and non-financial conflicts of interest often requires careful management. It will be important to consider if there are any relevant interests that may constitute a conflict of interest. There are situations where employment, holding of patents and other financial support should prevent people joining an author team. Funding of Cochrane Reviews by commercial organizations with an interest in the outcome of the review is not permitted. To ensure that any issues are identified early in the process, authors planning Cochrane Reviews should consult the Conflict of Interest Policy . Authors should make complete declarations of interest before registration of the review, and refresh these annually thereafter until publication and just prior to publication of the protocol and the review. For authors of review updates, this must be done at the time of the decision to update the review, annually thereafter until publication, and just prior to publication. Authors should also update declarations of interest at any point when their circumstances change.

1.3.1 Involving consumers and other stakeholders

Because the priorities of decision makers and consumers may be different from those of researchers, it is important that review authors consider carefully what questions are important to these different stakeholders. Systematic reviews are more likely to be relevant to a broad range of end users if they are informed by the involvement of people with a range of experiences, in terms of both the topic and the methodology (Thomas et al 2004, Rees and Oliver 2017). Engaging consumers and other stakeholders, such as policy makers, research funders and healthcare professionals, increases relevance, promotes mutual learning, improved uptake and decreases research waste.

Mapping out all potential stakeholders specific to the review question is a helpful first step to considering who might be invited to be involved in a review. Stakeholders typically include: patients and consumers; consumer advocates; policy makers and other public officials; guideline developers; professional organizations; researchers; funders of health services and research; healthcare practitioners, and, on occasion, journalists and other media professionals. Balancing seniority, credibility within the given field, and diversity should be considered. Review authors should also take account of the needs of resource-poor countries and regions in the review process (see Chapter 16 ) and invite appropriate input on the scope of the review and the questions it will address.

It is established good practice to ensure that consumers are involved and engaged in health research, including systematic reviews. Cochrane uses the term ‘consumers’ to refer to a wide range of people, including patients or people with personal experience of a healthcare condition, carers and family members, representatives of patients and carers, service users and members of the public. In 2017, a Statement of Principles for consumer involvement in Cochrane was agreed. This seeks to change the culture of research practice to one where both consumers and other stakeholders are joint partners in research from planning, conduct, and reporting to dissemination. Systematic reviews that have had consumer involvement should be more directly applicable to decision makers than those that have not (see online Chapter II ).

1.3.2 Working with consumers and other stakeholders

Methods for working with consumers and other stakeholders include surveys, workshops, focus groups and involvement in advisory groups. Decisions about what methods to use will typically be based on resource availability, but review teams should be aware of the merits and limitations of such methods. Authors will need to decide who to involve and how to provide adequate support for their involvement. This can include financial reimbursement, the provision of training, and stating clearly expectations of involvement, possibly in the form of terms of reference.

While a small number of consumers or other stakeholders may be part of the review team and become co-authors of the subsequent review, it is sometimes important to bring in a wider range of perspectives and to recognize that not everyone has the capacity or interest in becoming an author. Advisory groups offer a convenient approach to involving consumers and other relevant stakeholders, especially for topics in which opinions differ. Important points to ensure successful involvement include the following.

The review team should co-ordinate the input of the advisory group to inform key review decisions.
The advisory group’s input should continue throughout the systematic review process to ensure relevance of the review to end users is maintained.
Advisory group membership should reflect the breadth of the review question, and consideration should be given to involving vulnerable and marginalized people (Steel 2004) to ensure that conclusions on the value of the interventions are well-informed and applicable to all groups in society (see Chapter 16 ).

Templates such as terms of reference, job descriptions, or person specifications for an advisory group help to ensure clarity about the task(s) required and are available from INVOLVE . The website also gives further information on setting and organizing advisory groups. See also the Cochrane training website for further resources to support consumer involvement.

1.4 The importance of reliability

Systematic reviews aim to be an accurate representation of the current state of knowledge about a given issue. As understanding improves, the review can be updated. Nevertheless, it is important that the review itself is accurate at the time of publication. There are two main reasons for this imperative for accuracy. First, health decisions that affect people’s lives are increasingly taken based on systematic review findings. Current knowledge may be imperfect, but decisions will be better informed when taken in the light of the best of current knowledge. Second, systematic reviews form a critical component of legal and regulatory frameworks; for example, drug licensing or insurance coverage. Here, systematic reviews also need to hold up as auditable processes for legal examination. As systematic reviews need to be both correct, and be seen to be correct, detailed evidence-based methods have been developed to guide review authors as to the most appropriate procedures to follow, and what information to include in their reports to aid auditability.

1.4.1 Expectations for the conduct and reporting of Cochrane Reviews

Cochrane has developed methodological expectations for the conduct, reporting and updating of systematic reviews of interventions (MECIR) and their plain language summaries ( Plain Language Expectations for Authors of Cochrane Summaries ; PLEACS). Developed collaboratively by methodologists and Cochrane editors, they are intended to describe the desirable attributes of a Cochrane Review. The expectations are not all relevant at the same stage of review conduct, so care should be taken to identify those that are relevant at specific points during the review. Different methods should be used at different stages of the review in terms of the planning, conduct, reporting and updating of the review.

Each expectation has a title, a rationale and an elaboration. For the purposes of publication of a review with Cochrane, each has the status of either ‘mandatory’ or ‘highly desirable’. Items described as mandatory are expected to be applied, and if they are not then an appropriate justification should be provided; failure to implement such items may be used as a basis for deciding not to publish a review in the Cochrane Database of Systematic Reviews (CDSR). Items described as highly desirable should generally be implemented, but there are reasonable exceptions and justifications are not required.

All MECIR expectations for the conduct of a review are presented in the relevant chapters of this Handbook . Expectations for reporting of completed reviews (including PLEACS) are described in online Chapter III . The recommendations provided in the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement have been incorporated into the Cochrane reporting expectations, ensuring compliance with the PRISMA recommendations and summarizing attributes of reporting that should allow a full assessment of the methods and findings of the review (Moher et al 2009).

1.5 Protocol development

Preparing a systematic review is complex and involves many judgements. To minimize the potential for bias in the review process, these judgements should be made as far as possible in ways that do not depend on the findings of the studies included in the review. Review authors’ prior knowledge of the evidence may, for example, influence the definition of a systematic review question, the choice of criteria for study eligibility, or the pre-specification of intervention comparisons and outcomes to analyse. It is important that the methods to be used should be established and documented in advance (see MECIR Box 1.5.a , MECIR Box 1.5.b and MECIR Box 1.5.c ).

Publication of a protocol for a review that is written without knowledge of the available studies reduces the impact of review authors’ biases, promotes transparency of methods and processes, reduces the potential for duplication, allows peer review of the planned methods before they have been completed, and offers an opportunity for the review team to plan resources and logistics for undertaking the review itself. All chapters in the Handbook should be consulted when drafting the protocol. Since systematic reviews are by their nature retrospective, an element of knowledge of the evidence is often inevitable. This is one reason why non-content experts such as methodologists should be part of the review team (see Section 1.3 ). Two exceptions to the retrospective nature of a systematic review are a meta-analysis of a prospectively planned series of trials and some living systematic reviews, as described in Chapter 22 .

The review question should determine the methods used in the review, and not vice versa. The question may concern a relatively straightforward comparison of one treatment with another; or it may necessitate plans to compare different treatments as part of a network meta-analysis, or assess differential effects of an intervention in different populations or delivered in different ways.

The protocol sets out the context in which the review is being conducted. It presents an opportunity to develop ideas that are foundational for the review. This concerns, most explicitly, definition of the eligibility criteria such as the study participants and the choice of comparators and outcomes. The eligibility criteria may also be defined following the development of a logic model (or an articulation of the aspects of an extent logic model that the review is addressing) to explain how the intervention might work (see Chapter 2, Section 2.5.1 ).

MECIR Box 1.5.a Relevant expectations for conduct of intervention reviews

A key purpose of the protocol is to make plans to minimize bias in the eventual findings of the review. Reliable synthesis of available evidence requires a planned, systematic approach. Threats to the validity of systematic reviews can come from the studies they include or the process by which reviews are conducted. Biases within the studies can arise from the method by which participants are allocated to the intervention groups, awareness of intervention group assignment, and the collection, analysis and reporting of data. Methods for examining these issues should be specified in the protocol. Review processes can generate bias through a failure to identify an unbiased (and preferably complete) set of studies, and poor quality assurance throughout the review. The availability of research may be influenced by the nature of the results (i.e. reporting bias). To reduce the impact of this form of bias, searching may need to include unpublished sources of evidence (Dwan et al 2013) ( MECIR Box 1.5.b ).

MECIR Box 1.5.b Relevant expectations for the conduct of intervention reviews

Developing a protocol for a systematic review has benefits beyond reducing bias. Investing effort in designing a systematic review will make the process more manageable and help to inform key priorities for the review. Defining the question, referring to it throughout, and using appropriate methods to address the question focuses the analysis and reporting, ensuring the review is most likely to inform treatment decisions for funders, policy makers, healthcare professionals and consumers. Details of the planned analyses, including investigations of variability across studies, should be specified in the protocol, along with methods for interpreting the results through the systematic consideration of factors that affect confidence in estimates of intervention effect ( MECIR Box 1.5.c ).

MECIR Box 1.5.c Relevant expectations for conduct of intervention reviews

While the intention should be that a review will adhere to the published protocol, changes in a review protocol are sometimes necessary. This is also the case for a protocol for a randomized trial, which must sometimes be changed to adapt to unanticipated circumstances such as problems with participant recruitment, data collection or event rates. While every effort should be made to adhere to a predetermined protocol, this is not always possible or appropriate. It is important, however, that changes in the protocol should not be made based on how they affect the outcome of the research study, whether it is a randomized trial or a systematic review. Post hoc decisions made when the impact on the results of the research is known, such as excluding selected studies from a systematic review, or changing the statistical analysis, are highly susceptible to bias and should therefore be avoided unless there are reasonable grounds for doing this.

Enabling access to a protocol through publication (all Cochrane Protocols are published in the CDSR ) and registration on the PROSPERO register of systematic reviews reduces duplication of effort, research waste, and promotes accountability. Changes to the methods outlined in the protocol should be transparently declared.

This Handbook provides details of the systematic review methods developed or selected by Cochrane. They are intended to address the need for rigour, comprehensiveness and transparency in preparing a Cochrane systematic review. All relevant chapters – including those describing procedures to be followed in the later stages of the review – should be consulted during the preparation of the protocol. A more specific description of the structure of Cochrane Protocols is provide in online Chapter II .

1.6 Data management and quality assurance

Systematic reviews should be replicable, and retaining a record of the inclusion decisions, data collection, transformations or adjustment of data will help to establish a secure and retrievable audit trail. They can be operationally complex projects, often involving large research teams operating in different sites across the world. Good data management processes are essential to ensure that data are not inadvertently lost, facilitating the identification and correction of errors and supporting future efforts to update and maintain the review. Transparent reporting of review decisions enables readers to assess the reliability of the review for themselves.

Review management software, such as Covidence and EPPI-Reviewer , can be used to assist data management and maintain consistent and standardized records of decisions made throughout the review. These tools offer a central repository for review data that can be accessed remotely throughout the world by members of the review team. They record independent assessment of studies for inclusion, risk of bias and extraction of data, enabling checks to be made later in the process if needed. Research has shown that even experienced reviewers make mistakes and disagree with one another on risk-of-bias assessments, so it is particularly important to maintain quality assurance here, despite its cost in terms of author time. As more sophisticated information technology tools begin to be deployed in reviews (see Chapter 4, Section 4.6.6.2 and Chapter 22, Section 22.2.4 ), it is increasingly apparent that all review data – including the initial decisions about study eligibility – have value beyond the scope of the individual review. For example, review updates can be made more efficient through (semi-) automation when data from the original review are available for machine learning.

1.7 Chapter information

Authors: Toby J Lasserson, James Thomas, Julian PT Higgins

Acknowledgements: This chapter builds on earlier versions of the Handbook . We would like to thank Ruth Foxlee, Richard Morley, Soumyadeep Bhaumik, Mona Nasser, Dan Fox and Sally Crowe for their contributions to Section 1.3 .

Funding: JT is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North Thames at Barts Health NHS Trust. JPTH is a member of the NIHR Biomedical Research Centre at University Hospitals Bristol NHS Foundation Trust and the University of Bristol. JPTH received funding from National Institute for Health Research Senior Investigator award NF-SI-0617-10145. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

1.8 References

Antman E, Lau J, Kupelnick B, Mosteller F, Chalmers T. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: treatment for myocardial infarction. JAMA 1992; 268 : 240–248.

Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, Howells DW, Ioannidis JP, Oliver S. How to increase value and reduce waste when research priorities are set. Lancet 2014; 383 : 156–165.

Chandler J, Hopewell S. Cochrane methods – twenty years experience in developing systematic review methods. Systematic Reviews 2013; 2 : 76.

Dwan K, Gamble C, Williamson PR, Kirkham JJ, Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias: an updated review. PloS One 2013; 8 : e66844.

Gøtzsche PC, Ioannidis JPA. Content area experts as authors: helpful or harmful for systematic reviews and meta-analyses? BMJ 2012; 345 .

Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, Al-Shahi Salman R, Chan AW, Glasziou P. Biomedical research: increasing value, reducing waste. Lancet 2014; 383 : 101–104.

Moher D, Liberati A, Tetzlaff J, Altman D, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Medicine 2009; 6 : e1000097.

Oxman A, Guyatt G. The science of reviewing research. Annals of the New York Academy of Sciences 1993; 703 : 125–133.

Rees R, Oliver S. Stakeholder perspectives and participation in reviews. In: Gough D, Oliver S, Thomas J, editors. An Introduction to Systematic Reviews . 2nd ed. London: Sage; 2017. p. 17–34.

Steel R. Involving marginalised and vulnerable people in research: a consultation document (2nd revision). INVOLVE; 2004.

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J. Integrating qualitative research with trials in systematic reviews. BMJ 2004; 328 : 1010–1012.

For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Affiliations.

1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected].
2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom.
3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected].
PMID: 30089228
DOI: 10.1146/annurev-psych-010418-102803

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Keywords: evidence; guide; meta-analysis; meta-synthesis; narrative; systematic review; theory.

Guidelines as Topic
Meta-Analysis as Topic*
Publication Bias
Review Literature as Topic
Systematic Reviews as Topic*

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
J R Soc Med
v.96(3); 2003 Mar

Five steps to conducting a systematic review

Regina kunz.

1 German Cochrane Centre, Freiburg and Department of Nephrology, Charité, Berlin, Germany

Jos Kleijnen

2 Centre for Reviews and Dissemination, York, UK

3 German Cochrane Centre, Freiburg, Germany

Systematic reviews and meta-analyses are a key element of evidence-based healthcare, yet they remain in some ways mysterious. Why did the authors select certain studies and reject others? What did they do to pool results? How did a bunch of insignificant findings suddenly become significant? This paper, along with a book 1 that goes into more detail, demystifies these and other related intrigues.

A review earns the adjective systematic if it is based on a clearly formulated question, identifies relevant studies, appraises their quality and summarizes the evidence by use of explicit methodology. It is the explicit and systematic approach that distinguishes systematic reviews from traditional reviews and commentaries. Whenever we use the term review in this paper it will mean a systematic review . Reviews should never be done in any other way.

In this paper we provide a step-by-step explanation—there are just five steps—of the methods behind reviewing, and the quality elements inherent in each step (Box 1). For purposes of illustration we use a published review concerning the safety of public water fluoridation, but we must emphasize that our subject is review methodology, not fluoridation.

EXAMPLE: SAFETY OF PUBLIC WATER FLUORIDATION

You are a public health professional in a locality that has public water fluoridation. For many years, your colleagues and you have believed that it improves dental health. Recently there has been pressure from various interest groups to consider the safety of this public health intervention because they fear that it is causing cancer. Public health decisions have been based on professional judgment and practical feasibility without explicit consideration of the scientific evidence. (This was yesterday; today the evidence is available in a York review 2 , 3 , identifiable on MEDLINE through the freely accessible PubMed clinical queries interface [ http://www.ncbi.nlm.nib.gov/entrez/query/static/clinical.html ], under ‘systematic reviews’.)

STEP 1: FRAMING THE QUESTION

The research question may initially be stated as a query in free form but reviewers prefer to pose it in a structured and explicit way. The relations between various components of the question and the structure of the research design are shown in Figure 1 . This paper focuses only on the question of safety related to the outcomes described below.

An external file that holds a picture, illustration, etc.
Object name is 119f1l.jpg

Structured questions for systematic reviews and relations between question components in a comparative study

Box 1 The steps in a systematic review

The problems to be addressed by the review should be specified in the form of clear, unambiguous and structured questions before beginning the review work. Once the review questions have been set, modifications to the protocol should be allowed only if alternative ways of defining the populations, interventions, outcomes or study designs become apparent

The search for studies should be extensive. Multiple resources (both computerized and printed) should be searched without language restrictions. The study selection criteria should flow directly from the review questions and be specified a priori . Reasons for inclusion and exclusion should be recorded

Study quality assessment is relevant to every step of a review. Question formulation (Step 1) and study selection criteria (Step 2) should describe the minimum acceptable level of design. Selected studies should be subjected to a more refined quality assessment by use of general critical appraisal guides and design-based quality checklists (Step 3). These detailed quality assessments will be used for exploring heterogeneity and informing decisions regarding suitability of meta-analysis (Step 4). In addition they help in assessing the strength of inferences and making recommendations for future research (Step 5)

Data synthesis consists of tabulation of study characteristics, quality and effects as well as use of statistical methods for exploring differences between studies and combining their effects (meta-analysis). Exploration of heterogeneity and its sources should be planned in advance (Step 3). If an overall meta-analysis cannot be done, subgroup meta-analysis may be feasible

The issues highlighted in each of the four steps above should be met. The risk of publication bias and related biases should be explored. Exploration for heterogeneity should help determine whether the overall summary can be trusted, and, if not, the effects observed in high-quality studies should be used for generating inferences. Any recommendations should be graded by reference to the strengths and weaknesses of the evidence

Free-form question

Is it safe to provide population-wide drinking water fluoridation to prevent caries?

Structured question

The populations —Populations receiving drinking water sourced through a public water supply
The interventions or exposures —Fluoridation of drinking water (natural or artificial) compared with non-fluoridated water
The outcomes —Cancer is the main outcome of interest for the debate in your health authority
The study designs —Comparative studies of any design examining the harmful outcomes in at least two population groups, one with fluoridated drinking water and the other without. Harmful outcomes can be rare and they may develop over a long time. There are considerable difficulties in designing and conducting safety studies to capture these outcomes, since a large number of people need to be observed over a long period. These circumstances demand observational, not randomized studies. With this background, systematic reviews on safety have to include evidence from studies with a range of designs.

STEP 2: IDENTIFYING RELEVANT PUBLICATIONS

To capture as many relevant citations as possible, a wide range of medical, environmental and scientific databases were searched to identify primary studies of the effects of water fluoridation. The electronic searches were supplemented by hand searching of Index Medicus and Excerpta Medica back to 1945. Furthermore, various internet engines were searched for web pages that might provide references. This effort resulted in 3246 citations from which relevant studies were selected for the review. Their potential relevance was examined, and 2511 citations were excluded as irrelevant. The full papers of the remaining 735 citations were assessed to select those primary studies in man that directly related to fluoride in drinking water supplies, comparing at least two groups. These criteria excluded 481 studies and left 254 in the review. They came from thirty countries, published in fourteen languages between 1939 and 2000. Of these studies 175 were relevant to the question of safety, of which 26 used cancer as an outcome.

STEP 3: ASSESSING STUDY QUALITY

Design threshold for study selection.

Adequate study design as a marker of quality, is listed as an inclusion criterion in Box 1. This approach is most applicable when the main source of evidence is randomized studies. However, randomized studies are almost impossible to conduct at community level for a public health intervention such as water fluoridation. Thus, systematic reviews assessing the safety of such interventions have to include evidence from a broader range of study designs. Consideration of the type and amount of research likely to be available led to inclusion of comparative studies of any design. In this way, selected studies provided information about the harmful effects of exposure to fluoridated water compared with non-exposure.

Quality assessment of safety studies

After studies of an acceptable design have been selected, their in-depth assessment for the risk of various biases allows us to gauge the quality of the evidence in a more refined way. Biases either exaggerate or underestimate the ‘true’ effect of an exposure. The objective of the included studies was to compare groups exposed to fluoridated drinking water and those without such exposure for rates of undesirable outcomes, without bias. Safety studies should ascertain exposures and outcomes in such a way that the risk of misclassification is minimized. The exposure is likely to be more accurately ascertained if the study was prospective rather than retrospective and if it was started soon after water fluoridation rather than later. The outcomes of those developing cancer (and remaining free of cancer) are likely to be more accurately ascertained if the follow-up was long and if the assessment was blind to exposure status.

When examining how the effect of exposure on outcome was established, reviewers assessed whether the comparison groups were similar in all respects other than their exposure to fluoridated water. This is because the other differences may be related to the outcomes of interest independent of the drinking-water fluoridation, and this would bias the comparison. For example, if the people exposed to fluoridated water had other risk factors that made them more prone to have cancer, the apparent association between exposure and outcome might be explained by the more frequent occurrence of these factors among the exposed group. The technical word for such defects is confounding. In a randomized study, confounding factors are expected to be roughly equally distributed between groups. In observational studies their distribution may be unequal. Primary researchers can statistically adjust for these differences, when estimating the effect of exposure on outcomes, by use of multivariable modelling.

Put simply, use of a prospective design, robust ascertainment of exposure and outcomes, and control for confounding are the generic issues one would look for in quality assessment of studies on safety. Consequently, studies may range from satisfactorily meeting quality criteria, to having some deficiencies, to not meeting the criteria at all, and they can be assigned to one of three prespecified quality categories as shown in Table 1 . A quality hierarchy can then be developed, based on the degree to which studies comply with the criteria. None of the studies on cancer were in the high-quality category, but this was because randomized studies were non-existent and control for confounding was not always ideal in the observational studies. There were 8 studies of moderate quality and 18 of low quality.

Description of quality assessment of studies on safety of public water fluoridation

STEP 4: SUMMARIZING THE EVIDENCE

To summarize the evidence from studies of variable design and quality is not easy. The original review 3 provides details of how the differences between study results were investigated and how they were summarized (with or without meta-analysis). This paper restricts itself to summarizing the findings narratively. The association between exposure to fluoridated water and cancer in general was examined in 26 studies. Of these, 10 examined all-cause cancer incidence or mortality, in 22 analyses. Of these, 11 analyses found a negative association (fewer cancers due to exposure), 9 found a positive one and 2 found no association. Only 2 studies reported statistically significant differences. Thus no clear association between water fluoridation and increased cancer incidence or mortality was apparent. Bone/joint and thyroid cancers were of particular concern because of fluoride uptake by these organs. Neither the 6 studies of osteosarcoma nor the 2 studies of thyroid cancer and water fluoridation revealed significant differences. Overall no association was detected between water fluoridation and mortality from any cancer. These findings were also borne out in the moderate-quality subgroup of studies.

STEP 5: INTERPRETING THE FINDINGS

In the fluoridation example, the focus was on the safety of a community-based public health intervention. The generally low quality of available studies means that the results must be interpreted with caution. However, the elaborate efforts in searching an unusually large number of databases provide some safeguard against missing relevant studies. Thus the evidence summarized in this review is likely to be as good as it will get in the foreseeable future. Cancer was the harmful outcome of most interest in this instance. No association was found between exposure to fluoridated water and specific cancers or all cancers. The interpretation of the results may be generally limited because of the low quality of studies, but the findings for the cancer outcomes are supported by the moderate-quality studies.

After having spent some time reading and understanding the review, you are impressed by the sheer amount of published work relevant to the question of safety. However, you are somewhat disappointed by the poor quality of the primary studies. Of course, examination of safety only makes sense in a context where the intervention has some beneficial effect. Benefit and harm have to be compared to provide the basis for decision making. On the issue of the beneficial effect of public water fluoridation, the review 3 reassures you that the health authority was correct in judging that fluoridation of drinking water prevents caries. From the review you also discovered that dental fluorosis (mottled teeth) was related to concentration of fluoride. When the interest groups raise the issue of safety again, you will be able to declare that there is no evidence to link cancer with drinking-water fluoridation; however, you will have to come clean about the risk of dental fluorosis, which appears to be dose dependent, and you may want to measure the fluoride concentration in the water supply and share this information with the interest groups.

The ability to quantify the safety concerns of your population through a review, albeit from studies of moderate to low quality, allows your health authority, the politicians and the public to consider the balance between beneficial and harmful effects of water fluoridation. Those who see the prevention of caries as of primary importance will favour fluoridation. Others, worried about the disfigurement of mottled teeth, may prefer other means of fluoride administration or even occasional treatment for dental caries. Whatever the opinions on this matter, you are able to reassure all parties that there is no evidence that fluoridation of drinking water increases the risk of cancer.

With increasing focus on generating guidance and recommendations for practice through systematic reviews, healthcare professionals need to understand the principles of preparing such reviews. Here we have provided a brief step-by-step explanation of the principles. Our book 1 describes them in detail.

Log in using your username and password

Search More Search for this keyword Advanced search
Latest content
Current issue
Write for Us
BMJ Journals More You are viewing from: Google Indexer

http://orcid.org/0000-0003-0157-5319 Ahtisham Younas 1 , 2 ,
http://orcid.org/0000-0002-7839-8130 Parveen Ali 3 , 4
1 Memorial University of Newfoundland , St John's , Newfoundland , Canada
2 Swat College of Nursing , Pakistan
3 School of Nursing and Midwifery , University of Sheffield , Sheffield , South Yorkshire , UK
4 Sheffield University Interpersonal Violence Research Group , Sheffield University , Sheffield , UK
Correspondence to Ahtisham Younas, Memorial University of Newfoundland, St John's, NL A1C 5C4, Canada; ay6133{at}mun.ca

https://doi.org/10.1136/ebnurs-2021-103417

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Literature reviews offer a critical synthesis of empirical and theoretical literature to assess the strength of evidence, develop guidelines for practice and policymaking, and identify areas for future research. 1 It is often essential and usually the first task in any research endeavour, particularly in masters or doctoral level education. For effective data extraction and rigorous synthesis in reviews, the use of literature summary tables is of utmost importance. A literature summary table provides a synopsis of an included article. It succinctly presents its purpose, methods, findings and other relevant information pertinent to the review. The aim of developing these literature summary tables is to provide the reader with the information at one glance. Since there are multiple types of reviews (eg, systematic, integrative, scoping, critical and mixed methods) with distinct purposes and techniques, 2 there could be various approaches for developing literature summary tables making it a complex task specialty for the novice researchers or reviewers. Here, we offer five tips for authors of the review articles, relevant to all types of reviews, for creating useful and relevant literature summary tables. We also provide examples from our published reviews to illustrate how useful literature summary tables can be developed and what sort of information should be provided.

Tip 1: provide detailed information about frameworks and methods

Download figure
Open in new tab
Download powerpoint

Tabular literature summaries from a scoping review. Source: Rasheed et al . 3

The provision of information about conceptual and theoretical frameworks and methods is useful for several reasons. First, in quantitative (reviews synthesising the results of quantitative studies) and mixed reviews (reviews synthesising the results of both qualitative and quantitative studies to address a mixed review question), it allows the readers to assess the congruence of the core findings and methods with the adapted framework and tested assumptions. In qualitative reviews (reviews synthesising results of qualitative studies), this information is beneficial for readers to recognise the underlying philosophical and paradigmatic stance of the authors of the included articles. For example, imagine the authors of an article, included in a review, used phenomenological inquiry for their research. In that case, the review authors and the readers of the review need to know what kind of (transcendental or hermeneutic) philosophical stance guided the inquiry. Review authors should, therefore, include the philosophical stance in their literature summary for the particular article. Second, information about frameworks and methods enables review authors and readers to judge the quality of the research, which allows for discerning the strengths and limitations of the article. For example, if authors of an included article intended to develop a new scale and test its psychometric properties. To achieve this aim, they used a convenience sample of 150 participants and performed exploratory (EFA) and confirmatory factor analysis (CFA) on the same sample. Such an approach would indicate a flawed methodology because EFA and CFA should not be conducted on the same sample. The review authors must include this information in their summary table. Omitting this information from a summary could lead to the inclusion of a flawed article in the review, thereby jeopardising the review’s rigour.

Tip 2: include strengths and limitations for each article

Critical appraisal of individual articles included in a review is crucial for increasing the rigour of the review. Despite using various templates for critical appraisal, authors often do not provide detailed information about each reviewed article’s strengths and limitations. Merely noting the quality score based on standardised critical appraisal templates is not adequate because the readers should be able to identify the reasons for assigning a weak or moderate rating. Many recent critical appraisal checklists (eg, Mixed Methods Appraisal Tool) discourage review authors from assigning a quality score and recommend noting the main strengths and limitations of included studies. It is also vital that methodological and conceptual limitations and strengths of the articles included in the review are provided because not all review articles include empirical research papers. Rather some review synthesises the theoretical aspects of articles. Providing information about conceptual limitations is also important for readers to judge the quality of foundations of the research. For example, if you included a mixed-methods study in the review, reporting the methodological and conceptual limitations about ‘integration’ is critical for evaluating the study’s strength. Suppose the authors only collected qualitative and quantitative data and did not state the intent and timing of integration. In that case, the strength of the study is weak. Integration only occurred at the levels of data collection. However, integration may not have occurred at the analysis, interpretation and reporting levels.

Tip 3: write conceptual contribution of each reviewed article

While reading and evaluating review papers, we have observed that many review authors only provide core results of the article included in a review and do not explain the conceptual contribution offered by the included article. We refer to conceptual contribution as a description of how the article’s key results contribute towards the development of potential codes, themes or subthemes, or emerging patterns that are reported as the review findings. For example, the authors of a review article noted that one of the research articles included in their review demonstrated the usefulness of case studies and reflective logs as strategies for fostering compassion in nursing students. The conceptual contribution of this research article could be that experiential learning is one way to teach compassion to nursing students, as supported by case studies and reflective logs. This conceptual contribution of the article should be mentioned in the literature summary table. Delineating each reviewed article’s conceptual contribution is particularly beneficial in qualitative reviews, mixed-methods reviews, and critical reviews that often focus on developing models and describing or explaining various phenomena. Figure 2 offers an example of a literature summary table. 4

Tabular literature summaries from a critical review. Source: Younas and Maddigan. 4

Tip 4: compose potential themes from each article during summary writing

While developing literature summary tables, many authors use themes or subthemes reported in the given articles as the key results of their own review. Such an approach prevents the review authors from understanding the article’s conceptual contribution, developing rigorous synthesis and drawing reasonable interpretations of results from an individual article. Ultimately, it affects the generation of novel review findings. For example, one of the articles about women’s healthcare-seeking behaviours in developing countries reported a theme ‘social-cultural determinants of health as precursors of delays’. Instead of using this theme as one of the review findings, the reviewers should read and interpret beyond the given description in an article, compare and contrast themes, findings from one article with findings and themes from another article to find similarities and differences and to understand and explain bigger picture for their readers. Therefore, while developing literature summary tables, think twice before using the predeveloped themes. Including your themes in the summary tables (see figure 1 ) demonstrates to the readers that a robust method of data extraction and synthesis has been followed.

Tip 5: create your personalised template for literature summaries

Often templates are available for data extraction and development of literature summary tables. The available templates may be in the form of a table, chart or a structured framework that extracts some essential information about every article. The commonly used information may include authors, purpose, methods, key results and quality scores. While extracting all relevant information is important, such templates should be tailored to meet the needs of the individuals’ review. For example, for a review about the effectiveness of healthcare interventions, a literature summary table must include information about the intervention, its type, content timing, duration, setting, effectiveness, negative consequences, and receivers and implementers’ experiences of its usage. Similarly, literature summary tables for articles included in a meta-synthesis must include information about the participants’ characteristics, research context and conceptual contribution of each reviewed article so as to help the reader make an informed decision about the usefulness or lack of usefulness of the individual article in the review and the whole review.

In conclusion, narrative or systematic reviews are almost always conducted as a part of any educational project (thesis or dissertation) or academic or clinical research. Literature reviews are the foundation of research on a given topic. Robust and high-quality reviews play an instrumental role in guiding research, practice and policymaking. However, the quality of reviews is also contingent on rigorous data extraction and synthesis, which require developing literature summaries. We have outlined five tips that could enhance the quality of the data extraction and synthesis process by developing useful literature summaries.

Aromataris E ,
Rasheed SP ,

Twitter @Ahtisham04, @parveenazamali

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Patient consent for publication Not required.

Provenance and peer review Not commissioned; externally peer reviewed.

Read the full text or download the PDF:

The use and impact of surveillance-based technology initiatives in inpatient and acute mental health settings: A systematic review

Find this author on Google Scholar
Find this author on PubMed
Search for this author on this site
ORCID record for Jessica L. Griffiths
For correspondence: [email protected]
ORCID record for Katherine R. K. Saunders
ORCID record for Una Foye
ORCID record for Anna Greenburgh
ORCID record for Antonio Rojas-Garcia
ORCID record for Brynmor Lloyd-Evans
ORCID record for Sonia Johnson
ORCID record for Alan Simpson
Info/History
Supplementary material
Preview PDF

Background: The use of surveillance technologies is becoming increasingly common in inpatient mental health settings, commonly justified as efforts to improve safety and cost-effectiveness. However, the use of these technologies has been questioned in light of limited research conducted and the sensitivities, ethical concerns and potential harms of surveillance. This systematic review aims to: 1) map how surveillance technologies have been employed in inpatient mental health settings, 2) identify any best practice guidance, 3) explore how they are experienced by patients, staff and carers, and 4) examine evidence regarding their impact. Methods: We searched five academic databases (Embase, MEDLINE, PsycInfo, PubMed and Scopus), one grey literature database (HMIC) and two pre-print servers (medRxiv and PsyArXiv) to identify relevant papers published up to 18/09/2023. We also conducted backwards and forwards citation tracking and contacted experts to identify relevant literature. Quality was assessed using the Mixed Methods Appraisal Tool. Data were synthesised using a narrative approach. Results: A total of 27 studies were identified as meeting the inclusion criteria. Included studies reported on CCTV/video monitoring (n = 13), Vision-Based Patient Monitoring and Management (VBPMM) (n = 6), Body Worn Cameras (BWCs) (n = 4), GPS electronic monitoring (n = 2) and wearable sensors (n = 2). Twelve papers (44.4%) were rated as low quality, five (18.5%) medium quality, and ten (37.0%) high quality. Five studies (18.5%) declared a conflict of interest. We identified minimal best practice guidance. Qualitative findings indicate that patient, staff and carer perceptions and experiences of surveillance technologies are mixed and complex. Quantitative findings regarding the impact of surveillance on outcomes such as self-harm, violence, aggression, care quality and cost-effectiveness were inconsistent or weak. Discussion: There is currently insufficient evidence to suggest that surveillance technologies in inpatient mental health settings are achieving the outcomes they are employed to achieve, such as improving safety and reducing costs. The studies were generally of low methodological quality, lacked lived experience involvement, and a substantial proportion (18.5%) declared conflicts of interest. Further independent coproduced research is needed to more comprehensively evaluate the impact of surveillance technologies in inpatient settings, including harms and benefits. If surveillance technologies are to be implemented, it will be important to engage all key stakeholders in the development of policies, procedures and best practice guidance to regulate their use, with a particular emphasis on prioritising the perspectives of patients.

Competing Interest Statement

AS and UF have undertaken and published research on BWCs. We have received no financial support from BWC or any other surveillance technology companies. All other authors declare no competing interests.

Clinical Protocols

https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=463993

Funding Statement

This study is funded by the National Institute for Health and Care Research (NIHR) Policy Research Programme (grant no. PR-PRU-0916-22003). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. ARG was supported by the Ramon y Cajal programme (RYC2022-038556-I), funded by the Spanish Ministry of Science, Innovation and Universities.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Data Availability

The template data extraction form is available in Supplementary 1. MMAT quality appraisal ratings for each included study are available in Supplementary 2. All data used is publicly available in the published papers included in this review.

View the discussion thread.

Supplementary Material

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Citation Manager Formats

EndNote (tagged)
EndNote 8 (xml)
RefWorks Tagged
Ref Manager
Tweet Widget
Facebook Like
Google Plus One
Addiction Medicine (316)
Allergy and Immunology (618)
Anesthesia (160)
Cardiovascular Medicine (2277)
Dentistry and Oral Medicine (279)
Dermatology (201)
Emergency Medicine (370)
Endocrinology (including Diabetes Mellitus and Metabolic Disease) (798)
Epidemiology (11579)
Forensic Medicine (10)
Gastroenterology (678)
Genetic and Genomic Medicine (3582)
Geriatric Medicine (337)
Health Economics (617)
Health Informatics (2305)
Health Policy (913)
Health Systems and Quality Improvement (864)
Hematology (335)
HIV/AIDS (753)
Infectious Diseases (except HIV/AIDS) (13155)
Intensive Care and Critical Care Medicine (757)
Medical Education (359)
Medical Ethics (100)
Nephrology (389)
Neurology (3351)
Nursing (191)
Nutrition (507)
Obstetrics and Gynecology (651)
Occupational and Environmental Health (646)
Oncology (1758)
Ophthalmology (525)
Orthopedics (209)
Otolaryngology (284)
Pain Medicine (223)
Palliative Medicine (66)
Pathology (438)
Pediatrics (1004)
Pharmacology and Therapeutics (422)
Primary Care Research (406)
Psychiatry and Clinical Psychology (3062)
Public and Global Health (5992)
Radiology and Imaging (1223)
Rehabilitation Medicine and Physical Therapy (715)
Respiratory Medicine (811)
Rheumatology (367)
Sexual and Reproductive Health (353)
Sports Medicine (316)
Surgery (386)
Toxicology (50)
Transplantation (170)
Urology (142)

In one of L.A.’s largest cash heists ever, burglars steal as much as $30 million from vault

Los Angeles police headquarters downtown.

Show more sharing options
Copy Link URL Copied!

In one of the largest cash heists in Los Angeles history, thieves made off with as much as $30 million in an Easter Sunday burglary at a San Fernando Valley money storage facility, an L.A. police official said.

The burglary occurred Sunday night at a facility in Sylmar where cash from businesses across the region is handled and stored, said L.A. Police Department Cmdr. Elaine Morales.

The thieves were able to breach the building as well as the safe where the money was stored, Morales said. Law enforcement sources said the burglary was among the largest in city history when it comes to cash, and the total also surpassed any armored-car heist in the city.

Mystery surrounds the break-in.

Sources familiar with the investigation told The Times that a burglary crew broke through the roof of the Gardaworld building on Roxford Street to gain access to the vault. But it is unclear how they avoided the alarm system.

LOS ANGELES, CA- APRIL 04: Thieves made off with as much as $30 million in an Easter Sunday burglary from the Gardaworld building on Rexford Street in Sylmar. Published news reports said they entered through the roof. Photographed on Thursday, April 4, 2024. (Myung J. Chun / Los Angeles Times)

Neighbor heard odd noises amid heist of up to $30 million from Sylmar vault

Days after thieves stole as much as $30 million from a security company vault in Sylmar, residents and workers piece together details of the crime.

April 5, 2024

The Canada-based security company has not responded to requests for comment.

The operators of the business did not discover the massive theft until they opened the vault Monday. An ABC-7 TV news helicopter video showed a large cut on the side of the building covered by a piece of plywood.

Authorities were alerted, and detectives from the LAPD’s Mission Division station responded to the crime scene to gather evidence.

A law enforcement source confirmed to The Times there was an effort to breach the side of the cash-holding building in addition to the roof. At least one alarm was triggered during the crime, but it was not connected to local law enforcement, according to a source familiar with the investigation who was not authorized to discuss it publicly.

Mystery shrouds the multimillion-dollar Brink's big rig heist

Brink’s heist mystery: Questions about a timeline that ‘doesn’t make any sense’

A 298-mile drive in 2 hours and 4 minutes? The chronology described in a legal filing and law enforcement documents has added to the mystery surrounding the multimillion-dollar jewelry heist from a Brink’s truck.

Sept. 16, 2022

Further adding to the intrigue is that very few individuals would have known of the huge sums of cash being kept in that safe, according to law enforcement sources.

The break-in was described as elaborate and suggested an experienced crew who knew how to gain entry to a secure facility and go unnoticed.

Scott Andrew Selby, co-author of “Flawless: Inside the Largest Diamond Heist in History,” said that the theft has “all the markings of a really well-thought-out job” that was done by a “professional crew,” adding that based on other major heists of this nature, it is likely that the thieves had some inside intelligence.

He said investigators will be “looking around the globe for crimes with a similar M.O.”

As to whether the money is traceable, Selby said it depends on whether there are records of serial numbers or the cash that was collected is already in circulation. It is hard to hide ill-gotten gains and launder traceable bills, he said.

“As technology progresses and the world gets small, there are a lot of ways you can mess up and get caught,” Selby said. “With touch DNA, the slightest mistake can expose the identity of a member of the crew, leading authorities to eventually identify their associates.”

An FBI spokeswoman confirmed Wednesday night that the agency and the LAPD are investigating the theft.

A federal source said investigators were trying to complete a full accounting of the missing cash, but said it could be the largest cash heist in L.A.’s history.

The prior largest cash robbery in Los Angeles was on Sept. 12, 1997, with the theft of $18.9 million from the former site of the Dunbar Armored facility on Mateo Street. Those behind the incident were eventually caught .

Sunday’s theft comes nearly two years after the multimillion-dollar heist of jewelry from a Brink’s big rig at a Grapevine truck stop.

As much as $100 million in jewels and valuables was taken from the truck.

Ryan Gosling in DRIVE. Courtesy of FilmDistrict and Bold Films and OddLot Entertainment

These are the 5 best L.A. heist movies

Our film editor picks a handful of gems shot in Los Angeles that define the crime film, including classics by Michael Mann, Sofia Coppola and Quentin Tarantino.

March 14, 2024

In that case, thieves made off with the goods at 3 a.m. on July 11, 2022, taking more than 20 large bags stuffed with jewelry, gems and other items that the Brink’s tractor-trailer had been transporting to the L.A. area from the International Gem and Jewelry Show in San Mateo.

The heist occurred during a 27-minute window in which one driver slept in the vehicle’s sleeper berth and another ate a meal at the Flying J, a sprawling truck stop just off Interstate 5’s sinuous Grapevine in Lebec, Calif.

That crime remains unsolved.

Since a burglary crew broke through the reinforced roof of a Laguna Niguel bank and blew a hole in the safe more than four decades ago, it’s been extremely rare in Southern California for thieves to break in from above.

But a decade ago, rooftop bandits breached a series of banks in strip malls across the San Gabriel Valley. They stole at least $16 million before five were caught by the major crimes unit of the Los Angeles County Sheriff’s Department.

Last July, a burglary crew broke in through the roof of a wine specialist in Venice, making off with about 800 bottles worth about $600,000 — one of the biggest crimes in California wine history .

The perfect heist? Inside the seamless, sophisticated, stealthy L.A. theft that netted up to $30 million

April 6, 2024

The Peninsula Hotel in Beverly Hills on September 05, 2014.

Thief who finagled luxury hotel room keys admits to brazen diamond necklace heist

April 2, 2024

Paul Pierce speaks while wearing a black hoodie in front of a black backgroud.

Burglars reportedly hit Paul Pierce’s L.A. home, make off with $100,000, luxury watches

March 19, 2024

Start your day right

Sign up for Essential California for news, features and recommendations from the L.A. Times and beyond in your inbox six days a week.

You may occasionally receive promotional content from the Los Angeles Times.

Richard Winton is an investigative crime writer for the Los Angeles Times and part of the team that won the Pulitzer Prize for public service in 2011. Known as @lacrimes on Twitter, during almost 30 years at The Times he also has been part of the breaking news staff that won Pulitzers in 1998, 2004 and 2016.

More From the Los Angeles Times

Students walk outside building at Pomona College

19 Pomona College protesters arrested after storming, occupying president’s office

Pasadena CA., January 1, 2020: Pasadena Police Department clear the roads at the start 2020 Rose Bowl Parade route on Wednesday, January 1, 2020 in Pasadena, California. (Jason Armond / Los Angeles Times)

Pasadena police consider body cameras that activate when officers draw handguns

Norby Williamson holds an Emmy award during 26th Annual Sports Emmy Awards in 2005.

ESPN executive criticized on-air by Pat McAfee leaves network after nearly 40 years

LaVar Ball smiles with his tongue out during a game between the Charlotte Hornets and Indiana Pacers.

LaVar Ball knows why LaMelo, Lonzo are injured: ‘raggedy shoes’ and ‘rooty-toot workouts’

IMAGES

Systematic reviews
Research Systematic Review
What is a systematic review?
systematic literature review mixed methods
Research question by type of systematic review.
What is a Systematic Review? Ultimate Guide to Systematic Reviews

VIDEO

Systematic Review for Beginners: Understanding What, Why and How
Introduction to Systematic Review of Research
Introduction to Systematic Review
Conducting a Systematic Literature
Systematic review_01
Systematic Review for Beginners

COMMENTS

1. Formulate the Research Question
Step 1. Formulate the Research Question. A systematic review is based on a pre-defined specific research question (Cochrane Handbook, 1.1).The first step in a systematic review is to determine its focus - you should clearly frame the question(s) the review seeks to answer (Cochrane Handbook, 2.1).It may take you a while to develop a good review question - it is an important step in your review.
Formulating a research question
They also have clear criteria about the studies that are being used to address the research questions. This is often called inclusion criteria or eligibility criteria. Six examples of types of question are listed below, and the examples show different questions that a review might address based on the topic of influenza vaccination.
Systematic reviews: Formulate your question
Defining the question. Defining the research question and developing a protocol are the essential first steps in your systematic review. The success of your systematic review depends on a clear and focused question, so take the time to get it right. A framework may help you to identify the key concepts in your research question and to organise ...
Formulate Question
A narrow and specific research question is required in order to conduct a systematic review. The goal of a systematic review is to provide an evidence synthesis of ALL research performed on one particular topic. Your research question should be clearly answerable from the studies included in your review. Another consideration is whether the ...
Step 2: Define Your Research Question
Systematic reviews require a focused research question, often developed using one of the frameworks in the box below. A well-developed research question will inform the entirety of your review process, including: The development of your inclusion and exclusion criteria. The terms used in your search strategies.
Systematic Review
A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer. Example: Systematic review. In 2008, Dr. Robert Boyle and his colleagues published a systematic review in ...
Systematic Reviews: Develop & Refine Your Research Question
Develop & Refine Your Research Question. A clear, well-defined, and answerable research question is essential for any systematic review, meta-analysis, or other form of evidence synthesis. The question must be answerable. Spend time refining your research question. PICO Worksheet.
LibGuides: Systematic Reviews: 2. Develop a Research Question
Systematic Reviews. 2. Develop a Research Question. A well-developed and answerable question is the foundation for any systematic review. This process involves: Using the PICO framework can help team members clarify and refine the scope of their question. For example, if the population is breast cancer patients, is it all breast cancer patients ...
LibGuides: Systematic Reviews: 2. Formulate Your Question
Identify the potential scope of a systematic review. Your research question needs to be specific enough to get a meaningful conclusion, but broad enough to have enough literature to analyze. Prior investigation can help you assess whether a systematic review on a given question is justified. It is necessary to check whether there are already ...
Library Guides: Systematic reviews: Formulate the question
General principles. "A good systematic review is based on a well formulated, answerable question. The question guides the review by defining which studies will be included, what the search strategy to identify the relevant primary studies should be, and which data need to be extracted from each study." A systematic review question needs to be.
Systematic Reviews: Formulate your question and protocol
Having a focused and specific research question is especially important when undertaking a systematic review. If your search question is too broad you will retrieve too many search results and you will be unable to work with them all. If your question is too narrow, you may miss relevant papers. ... In this Cochrane systematic review by Clarke ...
2. Defining Your Question & Criteria
Chapter 3 of the Cochrane Handbook for Systematic Reviews of Interventions, Defining the criteria for including studies and how they will be grouped for the synthesis provides detailed guidance for developing a research question.. Predefined, unambiguous eligibility criteria are a fundamental prerequisite for a systematic review. The criteria for considering types of people included in studies ...
Defining your review question
Systematic reviews must have pre-specified criteria for including and excluding studies in the review. The Cochrane Handbook states that "predefined, unambiguous eligibility criteria are a fundamental prerequisite for a systematic review.". The first step in developing a protocol is determining the PICO elements of the review question and how the intervention produces the expected outcomes in ...
Systematic Reviews: Develop a Research Question
The first step in the systematic review process is to create a focused, well-defined research question. The question needs to be structured using a framework such as PICO which stands for Patient, Intervention, Comparison, and Outcome. There are many other frameworks available, use the best one to fit your topic.
Steps of a Systematic Review
Tools: Steps: PICO template. 1. Id entify your research question. Formulate a clear, well-defined research question of appropriate scope. Define your terminology. Find existing reviews on your topic to inform the development of your research question, identify gaps, and confirm that you are not duplicating the efforts of previous reviews.
Systematic Reviews: Formulating Your Research Question
The type of framework doesn't matter as much as the question being selected. Think of these frameworks as you would for a house or building. A framework is there to provide support and to be a scaffold for the rest of the structure. In the same way, a research question framework can also help structure your evidence synthesis question.
How to formulate the review question using PICO. 5 steps to get you
Set the scope. The scope of a review question requires careful thought. To answer the example PICO question above, the review would compare one treatment (alarms) with another (drug therapy). A broader question might consider all the available treatments for nocturnal enuresis in children.
How to Write a Systematic Review: A Narrative Review
Background. A systematic review, as its name suggests, is a systematic way of collecting, evaluating, integrating, and presenting findings from several studies on a specific question or topic.[] A systematic review is a research that, by identifying and combining evidence, is tailored to and answers the research question, based on an assessment of all relevant studies.[2,3] To identify assess ...
Chapter 1: Starting a review
Investing effort in designing a systematic review will make the process more manageable and help to inform key priorities for the review. Defining the question, referring to it throughout, and using appropriate methods to address the question focuses the analysis and reporting, ensuring the review is most likely to inform treatment decisions ...
How to do a systematic review
A systematic review aims to bring evidence together to answer a pre-defined research question. This involves the identification of all primary research relevant to the defined review question, the critical appraisal of this research, and the synthesis of the findings.13 Systematic reviews may combine data from different.
How to Do a Systematic Review: A Best Practice Guide for Conducting and
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question.
Introduction to systematic review and meta-analysis
It is easy to confuse systematic reviews and meta-analyses. A systematic review is an objective, reproducible method to find answers to a certain research question, by collecting all available studies related to that question and reviewing and analyzing their results. A meta-analysis differs from a systematic review in that it uses statistical ...
PDF Systematic Reviews
systematic review ~ 6+ months Scoping Review Aims to identify the nature and extent of research evidence ~ 6+ months Systematized Review Includes elements of a systematic review; good for post graduate assignments ~ 3+ months Narrative Useful for obtaining a broad perspective on a topic ~ 2+ months Types of Literature Reviews
Five steps to conducting a systematic review
Reasons for inclusion and exclusion should be recorded. Step 3: Assessing the quality of studies. Study quality assessment is relevant to every step of a review. Question formulation (Step 1) and study selection criteria (Step 2) should describe the minimum acceptable level of design.
Five tips for developing useful literature summary tables for writing
The provision of information about conceptual and theoretical frameworks and methods is useful for several reasons. First, in quantitative (reviews synthesising the results of quantitative studies) and mixed reviews (reviews synthesising the results of both qualitative and quantitative studies to address a mixed review question), it allows the readers to assess the congruence of the core ...
Retraction and republication—Pharmacotherapy for adults ...
A reader contacted The Lancet regarding five trials included in the Article Pharmacotherapy for adults with overweight and obesity: a systematic review and network meta-analysis of randomised controlled trials1 by Qingyang Shi and colleagues, expressing concern that these trials were not "unique" and that there was potential overlap in the populations studied. In response, the authors ...
Information
Secondly, the research method adopted is described, following a systematic series of steps that analyze the quality of the primary studies. Thirdly, the results are demonstrated, which provide answers to the research questions of the systematic review, and finally, these results are discussed.
The use and impact of surveillance-based technology initiatives in
Background: The use of surveillance technologies is becoming increasingly common in inpatient mental health settings, commonly justified as efforts to improve safety and cost-effectiveness. However, the use of these technologies has been questioned in light of limited research conducted and the sensitivities, ethical concerns and potential harms of surveillance. This systematic review aims to ...
Burglars steal $30 million on Easter in one of L.A.'s biggest heists
Published April 3, 2024 Updated April 5, 2024 5:55 PM PT. In one of the largest cash heists in Los Angeles history, thieves made off with as much as $30 million in an Easter Sunday burglary at a ...

University of Tasmania, Australia

Step 1. Formulate the Research Question

PICO - to search for quantitative review questions

SPIDER - to search for qualitative and mixed methods research studies

SPICE - to search for qualitative evidence

ECLIPSE - to search for health policy/management information

Need More Help? Book a consultation with a Learning and Research Librarian or contact [email protected] .

UCL LIBRARY SERVICES

Formulating a research question

Examples of review questions

Example of eligibility criteria

Consider the research boundaries

Mnemonics for structuring questions

Systematic reviews: Formulate your question

Defining the question

The PICO framework

The SPICE framework

The SPIDER framework

More advice about formulating a research question

Systematic Reviews & Evidence Synthesis Methods

Formulate your Research Question

Types of Research Questions for Systematic Reviews

Learn More . . .

Frameworks for Developing your Research Question

For more frameworks and guidance on developing the research question, check out:

Video - Formulating a Research Question (4:43 minutes)

Memorial Sloan Kettering Library

Systematic Review Service

Define Your Research Question

How the MSK Library Can Help

Using a Question Framework

Have a language expert improve your writing

Systematic Review | Definition, Example & Guide

Table of contents

Here's why students love Scribbr's proofreading services

Prevent plagiarism. Run a free check.

Step 1: Formulate a research question

Step 2: Develop a protocol

Step 3: Search for all relevant studies

Step 4: Apply the selection criteria

Step 5: Extract the data

Step 6: Synthesize the data

Step 7: Write and publish a report

Cite this Scribbr article

Is this article helpful?

Shaun Turney

Develop & Refine Your Research Question

PICO Framework

References & Recommended Reading

Conducting Research

Systematic Reviews

Using Databases

Finding & Accessing

Writing & Citing

Meet a Librarian

Search Services

Citation Mgmt

Scholarly Communications

Library Updates

2. Develop a Research Question

Other Question Frameworks

Ohio State nav bar

Systematic Reviews

Formulate Your Question

Consult With a Librarian

Systematic reviews

Formulating a clear and concise question

General principles

The elements of PICO

Intervention:

Comparison:

Systematic Reviews: Formulate your question and protocol

Recommended Reading

Formulate Your Question

Other Frameworks: alternatives to PICO

Develop Your Protocol

Register Your Protocol

Systematic Reviews

Question Guidance

Systematic Reviews - Research Guide