literature research scholarly journal

Explore millions of high-quality primary sources and images from around the world, including artworks, maps, photographs, and more.

Explore migration issues through a variety of media types

• Part of The Streets are Talking: Public Forms of Creative Expression from Around the World
• Part of The Journal of Economic Perspectives, Vol. 34, No. 1 (Winter 2020)
• Part of Cato Institute (Aug. 3, 2021)
• Part of University of California Press
• Part of Open: Smithsonian National Museum of African American History & Culture
• Part of Indiana Journal of Global Legal Studies, Vol. 19, No. 1 (Winter 2012)
• Part of R Street Institute (Nov. 1, 2020)
• Part of Leuven University Press
• Part of UN Secretary-General Papers: Ban Ki-moon (2007-2016)
• Part of Perspectives on Terrorism, Vol. 12, No. 4 (August 2018)
• Part of Leveraging Lives: Serbia and Illegal Tunisian Migration to Europe, Carnegie Endowment for International Peace (Mar. 1, 2023)
• Part of UCL Press

Harness the power of visual materials—explore more than 3 million images now on JSTOR.

Enhance your scholarly research with underground newspapers, magazines, and journals.

Explore collections in the arts, sciences, and literature from the world’s leading museums, archives, and scholars.

Methodological Approaches to Literature Review

• Living reference work entry
• First Online: 09 May 2023
• Cite this living reference work entry

• Dennis Thomas 2 ,
• Elida Zairina 3 &
• Johnson George 4  

435 Accesses

The literature review can serve various functions in the contexts of education and research. It aids in identifying knowledge gaps, informing research methodology, and developing a theoretical framework during the planning stages of a research study or project, as well as reporting of review findings in the context of the existing literature. This chapter discusses the methodological approaches to conducting a literature review and offers an overview of different types of reviews. There are various types of reviews, including narrative reviews, scoping reviews, and systematic reviews with reporting strategies such as meta-analysis and meta-synthesis. Review authors should consider the scope of the literature review when selecting a type and method. Being focused is essential for a successful review; however, this must be balanced against the relevance of the review to a broad audience.

• Literature review
• Systematic review
• Meta-analysis
• Scoping review
• Research methodology

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Akobeng AK. Principles of evidence based medicine. Arch Dis Child. 2005;90(8):837–40.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Alharbi A, Stevenson M. Refining Boolean queries to identify relevant studies for systematic review updates. J Am Med Inform Assoc. 2020;27(11):1658–66.

Article   PubMed   PubMed Central   Google Scholar  

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Article   Google Scholar  

Aromataris E MZE. JBI manual for evidence synthesis. 2020.

Google Scholar  

Aromataris E, Pearson A. The systematic review: an overview. Am J Nurs. 2014;114(3):53–8.

Article   PubMed   Google Scholar  

Aromataris E, Riitano D. Constructing a search strategy and searching for evidence. A guide to the literature search for a systematic review. Am J Nurs. 2014;114(5):49–56.

Babineau J. Product review: covidence (systematic review software). J Canad Health Libr Assoc Canada. 2014;35(2):68–71.

Baker JD. The purpose, process, and methods of writing a literature review. AORN J. 2016;103(3):265–9.

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.

Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6(1):1–12.

Brown D. A review of the PubMed PICO tool: using evidence-based practice in health education. Health Promot Pract. 2020;21(4):496–8.

Cargo M, Harris J, Pantoja T, et al. Cochrane qualitative and implementation methods group guidance series – paper 4: methods for assessing evidence on intervention implementation. J Clin Epidemiol. 2018;97:59–69.

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

Article   CAS   PubMed   Google Scholar  

Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med. 1997;127(5):380–7.

Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Cummings SR, Browner WS, Hulley SB, editors. Designing Clinical Research: An Epidemiological Approach. 4th ed. Philadelphia (PA): P Lippincott Williams & Wilkins; 2007. p. 14–22.

Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. JMLA. 2018;106(4):420.

Ferrari R. Writing narrative style literature reviews. Medical Writing. 2015;24(4):230–5.

Flemming K, Booth A, Hannes K, Cargo M, Noyes J. Cochrane qualitative and implementation methods group guidance series – paper 6: reporting guidelines for qualitative, implementation, and process evaluation evidence syntheses. J Clin Epidemiol. 2018;97:79–85.

Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26(2):91–108.

Green BN, Johnson CD, Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med. 2006;5(3):101–17.

Gregory AT, Denniss AR. An introduction to writing narrative and systematic reviews; tasks, tips and traps for aspiring authors. Heart Lung Circ. 2018;27(7):893–8.

Harden A, Thomas J, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. J Clin Epidemiol. 2018;97:70–8.

Harris JL, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis. J Clin Epidemiol. 2018;97:39–48.

Higgins J, Thomas J. In: Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3, updated February 2022). Available from www.training.cochrane.org/handbook.: Cochrane; 2022.

International prospective register of systematic reviews (PROSPERO). Available from https://www.crd.york.ac.uk/prospero/ .

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Landhuis E. Scientific literature: information overload. Nature. 2016;535(7612):457–8.

Lockwood C, Porritt K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, Loveday H, Carrier J, Stannard D. Chapter 2: Systematic reviews of qualitative evidence. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. Available from https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-03 .

Chapter   Google Scholar  

Lorenzetti DL, Topfer L-A, Dennett L, Clement F. Value of databases other than medline for rapid health technology assessments. Int J Technol Assess Health Care. 2014;30(2):173–8.

Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group. Preferred reporting items for (SR) and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;6:264–9.

Mulrow CD. Systematic reviews: rationale for systematic reviews. BMJ. 1994;309(6954):597–9.

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143.

Munthe-Kaas HM, Glenton C, Booth A, Noyes J, Lewin S. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol. 2019;19(1):1–13.

Murphy CM. Writing an effective review article. J Med Toxicol. 2012;8(2):89–90.

NHMRC. Guidelines for guidelines: assessing risk of bias. Available at https://nhmrc.gov.au/guidelinesforguidelines/develop/assessing-risk-bias . Last published 29 August 2019. Accessed 29 Aug 2022.

Noyes J, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 1: introduction. J Clin Epidemiol. 2018b;97:35–8.

Noyes J, Booth A, Flemming K, et al. Cochrane qualitative and implementation methods group guidance series – paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. J Clin Epidemiol. 2018a;97:49–58.

Noyes J, Booth A, Moore G, Flemming K, Tunçalp Ö, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods. BMJ Glob Health. 2019;4(Suppl 1):e000893.

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Healthcare. 2015;13(3):141–6.

Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Methods. 2019;10(3):330–42.

Article   PubMed Central   Google Scholar  

Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):1–7.

Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Brit Med J. 2017;358

Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Br Med J. 2016;355

Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008–12.

Tawfik GM, Dila KAS, Mohamed MYF, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health. 2019;47(1):1–9.

The Critical Appraisal Program. Critical appraisal skills program. Available at https://casp-uk.net/ . 2022. Accessed 29 Aug 2022.

The University of Melbourne. Writing a literature review in Research Techniques 2022. Available at https://students.unimelb.edu.au/academic-skills/explore-our-resources/research-techniques/reviewing-the-literature . Accessed 29 Aug 2022.

The Writing Center University of Winconsin-Madison. Learn how to write a literature review in The Writer’s Handbook – Academic Professional Writing. 2022. Available at https://writing.wisc.edu/handbook/assignments/reviewofliterature/ . Accessed 29 Aug 2022.

Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999;18(20):2693–708.

Tricco AC, Lillie E, Zarin W, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16(1):15.

Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Yoneoka D, Henmi M. Clinical heterogeneity in random-effect meta-analysis: between-study boundary estimate problem. Stat Med. 2019;38(21):4131–45.

Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. Am J Gastroenterol. 2009;104(5):1086–92.

Download references

Author information

Authors and affiliations.

Centre of Excellence in Treatable Traits, College of Health, Medicine and Wellbeing, University of Newcastle, Hunter Medical Research Institute Asthma and Breathing Programme, Newcastle, NSW, Australia

Dennis Thomas

Department of Pharmacy Practice, Faculty of Pharmacy, Universitas Airlangga, Surabaya, Indonesia

Elida Zairina

Centre for Medicine Use and Safety, Monash Institute of Pharmaceutical Sciences, Faculty of Pharmacy and Pharmaceutical Sciences, Monash University, Parkville, VIC, Australia

Johnson George

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Johnson George .

Section Editor information

College of Pharmacy, Qatar University, Doha, Qatar

Derek Charles Stewart

Department of Pharmacy, University of Huddersfield, Huddersfield, United Kingdom

Zaheer-Ud-Din Babar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this entry

Cite this entry.

Thomas, D., Zairina, E., George, J. (2023). Methodological Approaches to Literature Review. In: Encyclopedia of Evidence in Pharmaceutical Public Health and Health Services Research in Pharmacy. Springer, Cham. https://doi.org/10.1007/978-3-030-50247-8_57-1

Download citation

DOI : https://doi.org/10.1007/978-3-030-50247-8_57-1

Received : 22 February 2023

Accepted : 22 February 2023

Published : 09 May 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-50247-8

Online ISBN : 978-3-030-50247-8

eBook Packages : Springer Reference Biomedicine and Life Sciences Reference Module Biomedical and Life Sciences

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved April 1, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core Collection This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• Next: Planning the Review >>
• Last Updated: Jan 17, 2024 10:05 AM
• URL: https://libguides.northwestern.edu/literaturereviews

Research Methods

• Getting Started
• Literature Review Research
• Research Design
• Research Design By Discipline
• SAGE Research Methods
• Teaching with SAGE Research Methods

Literature Review

• What is a Literature Review?
• What is NOT a Literature Review?
• Purposes of a Literature Review
• Types of Literature Reviews
• Literature Reviews vs. Systematic Reviews
• Systematic vs. Meta-Analysis

Literature Review  is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

• Summarizes and analyzes previous research relevant to a topic
• Includes scholarly books and articles published in academic journals
• Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

• Help gather ideas or information
• Keep up to date in current trends and findings
• Help develop new questions

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas.
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic.
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
• Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University 

Keep in mind the following, a literature review is NOT:

Not an essay 

Not an annotated bibliography  in which you summarize each article that you have reviewed.  A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper   where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

• provides thorough knowledge of previous studies; introduces seminal works.
• helps focus one’s own research topic.
• identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
• suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
• identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
• helps the researcher avoid repetition of earlier research.
• suggests unexplored populations.
• determines whether past studies agree or disagree; identifies controversy in the literature.
• tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature."  Educational Researcher  36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015),  Literature reviews vs systematic reviews.  Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Systematic review or meta-analysis?

A  systematic review  answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A  meta-analysis  is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

• clearly stated objectives with pre-defined eligibility criteria for studies
• explicit, reproducible methodology
• a systematic search that attempts to identify all studies
• assessment of the validity of the findings of the included studies (e.g. risk of bias)
• systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis. 

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review.  More information on meta-analyses can be found in  Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies.  It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings.  Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted.  In that case, an integrative review is an appropriate strategy. 

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

• << Previous: Getting Started
• Next: Research Design >>
• Last Updated: Aug 21, 2023 4:07 PM
• URL: https://guides.lib.udel.edu/researchmethods

• All subject areas
• Agricultural and Biological Sciences
• Arts and Humanities
• Biochemistry, Genetics and Molecular Biology
• Business, Management and Accounting
• Chemical Engineering
• Computer Science
• Decision Sciences
• Earth and Planetary Sciences
• Economics, Econometrics and Finance
• Engineering
• Environmental Science
• Health Professions
• Immunology and Microbiology
• Materials Science
• Mathematics
• Multidisciplinary
• Neuroscience
• Pharmacology, Toxicology and Pharmaceutics
• Physics and Astronomy
• Social Sciences
• All subject categories
• Acoustics and Ultrasonics
• Advanced and Specialized Nursing
• Aerospace Engineering
• Agricultural and Biological Sciences (miscellaneous)
• Agronomy and Crop Science
• Algebra and Number Theory
• Analytical Chemistry
• Anesthesiology and Pain Medicine
• Animal Science and Zoology
• Anthropology
• Applied Mathematics
• Applied Microbiology and Biotechnology
• Applied Psychology
• Aquatic Science
• Archeology (arts and humanities)
• Architecture
• Artificial Intelligence
• Arts and Humanities (miscellaneous)
• Assessment and Diagnosis
• Astronomy and Astrophysics
• Atmospheric Science
• Atomic and Molecular Physics, and Optics
• Automotive Engineering
• Behavioral Neuroscience
• Biochemistry
• Biochemistry, Genetics and Molecular Biology (miscellaneous)
• Biochemistry (medical)
• Bioengineering
• Biological Psychiatry
• Biomaterials
• Biomedical Engineering
• Biotechnology
• Building and Construction
• Business and International Management
• Business, Management and Accounting (miscellaneous)
• Cancer Research
• Cardiology and Cardiovascular Medicine
• Care Planning
• Cell Biology
• Cellular and Molecular Neuroscience
• Ceramics and Composites
• Chemical Engineering (miscellaneous)
• Chemical Health and Safety
• Chemistry (miscellaneous)
• Chiropractics
• Civil and Structural Engineering
• Clinical Biochemistry
• Clinical Psychology
• Cognitive Neuroscience
• Colloid and Surface Chemistry
• Communication
• Community and Home Care
• Complementary and Alternative Medicine
• Complementary and Manual Therapy
• Computational Mathematics
• Computational Mechanics
• Computational Theory and Mathematics
• Computer Graphics and Computer-Aided Design
• Computer Networks and Communications
• Computer Science Applications
• Computer Science (miscellaneous)
• Computers in Earth Sciences
• Computer Vision and Pattern Recognition
• Condensed Matter Physics
• Conservation
• Control and Optimization
• Control and Systems Engineering
• Critical Care and Intensive Care Medicine
• Critical Care Nursing
• Cultural Studies
• Decision Sciences (miscellaneous)
• Dental Assisting
• Dental Hygiene
• Dentistry (miscellaneous)
• Dermatology
• Development
• Developmental and Educational Psychology
• Developmental Biology
• Developmental Neuroscience
• Discrete Mathematics and Combinatorics
• Drug Discovery
• Drug Guides
• Earth and Planetary Sciences (miscellaneous)
• Earth-Surface Processes
• Ecological Modeling
• Ecology, Evolution, Behavior and Systematics
• Economic Geology
• Economics and Econometrics
• Economics, Econometrics and Finance (miscellaneous)
• Electrical and Electronic Engineering
• Electrochemistry
• Electronic, Optical and Magnetic Materials
• Emergency Medical Services
• Emergency Medicine
• Emergency Nursing
• Endocrine and Autonomic Systems
• Endocrinology
• Endocrinology, Diabetes and Metabolism
• Energy Engineering and Power Technology
• Energy (miscellaneous)
• Engineering (miscellaneous)
• Environmental Chemistry
• Environmental Engineering
• Environmental Science (miscellaneous)
• Epidemiology
• Experimental and Cognitive Psychology
• Family Practice
• Filtration and Separation
• Fluid Flow and Transfer Processes
• Food Animals
• Food Science
• Fuel Technology
• Fundamentals and Skills
• Gastroenterology
• Gender Studies
• Genetics (clinical)
• Geochemistry and Petrology
• Geography, Planning and Development
• Geometry and Topology
• Geotechnical Engineering and Engineering Geology
• Geriatrics and Gerontology
• Gerontology
• Global and Planetary Change
• Hardware and Architecture
• Health Informatics
• Health Information Management
• Health Policy
• Health Professions (miscellaneous)
• Health (social science)
• Health, Toxicology and Mutagenesis
• History and Philosophy of Science
• Horticulture
• Human-Computer Interaction
• Human Factors and Ergonomics
• Immunology and Allergy
• Immunology and Microbiology (miscellaneous)
• Industrial and Manufacturing Engineering
• Industrial Relations
• Infectious Diseases
• Information Systems
• Information Systems and Management
• Inorganic Chemistry
• Insect Science
• Instrumentation
• Internal Medicine
• Issues, Ethics and Legal Aspects
• Leadership and Management
• Library and Information Sciences
• Life-span and Life-course Studies
• Linguistics and Language
• Literature and Literary Theory
• LPN and LVN
• Management Information Systems
• Management, Monitoring, Policy and Law
• Management of Technology and Innovation
• Management Science and Operations Research
• Materials Chemistry
• Materials Science (miscellaneous)
• Maternity and Midwifery
• Mathematical Physics
• Mathematics (miscellaneous)
• Mechanical Engineering
• Mechanics of Materials
• Media Technology
• Medical and Surgical Nursing
• Medical Assisting and Transcription
• Medical Laboratory Technology
• Medical Terminology
• Medicine (miscellaneous)
• Metals and Alloys
• Microbiology
• Microbiology (medical)
• Modeling and Simulation
• Molecular Biology
• Molecular Medicine
• Nanoscience and Nanotechnology
• Nature and Landscape Conservation
• Neurology (clinical)
• Neuropsychology and Physiological Psychology
• Neuroscience (miscellaneous)
• Nuclear and High Energy Physics
• Nuclear Energy and Engineering
• Numerical Analysis
• Nurse Assisting
• Nursing (miscellaneous)
• Nutrition and Dietetics
• Obstetrics and Gynecology
• Occupational Therapy
• Ocean Engineering
• Oceanography
• Oncology (nursing)
• Ophthalmology
• Oral Surgery
• Organic Chemistry
• Organizational Behavior and Human Resource Management
• Orthodontics
• Orthopedics and Sports Medicine
• Otorhinolaryngology
• Paleontology
• Parasitology
• Pathology and Forensic Medicine
• Pediatrics, Perinatology and Child Health
• Periodontics
• Pharmaceutical Science
• Pharmacology
• Pharmacology (medical)
• Pharmacology (nursing)
• Pharmacology, Toxicology and Pharmaceutics (miscellaneous)
• Physical and Theoretical Chemistry
• Physical Therapy, Sports Therapy and Rehabilitation
• Physics and Astronomy (miscellaneous)
• Physiology (medical)
• Plant Science
• Political Science and International Relations
• Polymers and Plastics
• Process Chemistry and Technology
• Psychiatry and Mental Health
• Psychology (miscellaneous)
• Public Administration
• Public Health, Environmental and Occupational Health
• Pulmonary and Respiratory Medicine
• Radiological and Ultrasound Technology
• Radiology, Nuclear Medicine and Imaging
• Rehabilitation
• Religious Studies
• Renewable Energy, Sustainability and the Environment
• Reproductive Medicine
• Research and Theory
• Respiratory Care
• Review and Exam Preparation
• Reviews and References (medical)
• Rheumatology
• Safety Research
• Safety, Risk, Reliability and Quality
• Sensory Systems
• Signal Processing
• Small Animals
• Social Psychology
• Social Sciences (miscellaneous)
• Social Work
• Sociology and Political Science
• Soil Science
• Space and Planetary Science
• Spectroscopy
• Speech and Hearing
• Sports Science
• Statistical and Nonlinear Physics
• Statistics and Probability
• Statistics, Probability and Uncertainty
• Strategy and Management
• Stratigraphy
• Structural Biology
• Surfaces and Interfaces
• Surfaces, Coatings and Films
• Theoretical Computer Science
• Tourism, Leisure and Hospitality Management
• Transplantation
• Transportation
• Urban Studies
• Veterinary (miscellaneous)
• Visual Arts and Performing Arts
• Waste Management and Disposal
• Water Science and Technology
• All regions / countries
• Asiatic Region
• Eastern Europe
• Latin America
• Middle East
• Northern America
• Pacific Region
• Western Europe
• ARAB COUNTRIES
• IBEROAMERICA
• NORDIC COUNTRIES
• Afghanistan
• Bosnia and Herzegovina
• Brunei Darussalam
• Czech Republic
• Dominican Republic
• Netherlands
• New Caledonia
• New Zealand
• Papua New Guinea
• Philippines
• Puerto Rico
• Russian Federation
• Saudi Arabia
• South Africa
• South Korea
• Switzerland
• Syrian Arab Republic
• Trinidad and Tobago
• United Arab Emirates
• United Kingdom
• United States
• Vatican City State
• Book Series
• Conferences and Proceedings
• Trade Journals

• Citable Docs. (3years)
• Total Cites (3years)

Follow us on @ScimagoJR Scimago Lab , Copyright 2007-2022. Data Source: Scopus®

Cookie settings

Cookie Policy

Legal Notice

Privacy Policy

Literature Reviews

• Getting Started

Selecting a topic

Remember the goal of your literature review, planning your research, popular vs. scholarly, finding scholarly journal articles, finding or requesting full-text articles, why not just use google scholar, finding books, open access resources, manage your own downloads and citations, using your own method, managing your citations, organizing your notes: synthesis matrix.

• Literature Review as a Product: Organizing your Writing
• Finding Examples of Literature Reviews
• General Resources

Need help? Ask a librarian or chat

Ask A Librarian  for help finding sources, narrowing or expanding your topic, and more!​

• Research Desk: 314-246-6950
• Toll-free: 800-985-4279
• Visit us:  Library hours
• Search our FAQs or email us 

One of the most important steps in research is selecting a good research topic.  This page on the Capstone Research Guide will guide you through selecting a topic, writing a thesis statement and/or research questions, selecting keywords, and recommended databases for preliminary research.

The goal of the literature review is to provide an analysis of the literature and research already published on your subject. You want to read as much as possible on your topic to gain a foundation of the information which already exists and analyze that information to understand how it all relates. This gives you the opportunity to identify possible gaps in the research or justify your own research in relation to what already exists. 

This page will walk through the steps in: 

• Planning your research 
• Collecting your research 
• Organizing your notes 

After you have chosen your topic, you will want to make a plan for all of the places you should look for information. This will allow you to organize your search and keep track of information as you find it. In a document, make a list of possible places you might find information on your topic. This list might include: 

• Library databases - See section "Finding scholarly journal articles" below
• Research facilities - Are there organizations or institutions which publish and share research in your field of study?
• Open Access Journals - See section "Open Access Journals" below
• Webster University library catalog - See section "Finding books" below
• The catalogs of other libraries - See section "Finding books" below

Keep in mind that a literature review necessitates the use of scholarly research. These are peer-reviewed articles written by graduate or post-graduate students, educators, researchers, or professionals in the field. These types of articles wil include standard citations for the works they reference in their research. 

What is a scholarly, peer-reviewed journal article?

Scholarly articles are sometimes "peer-reviewed" or "refereed" because they are evaluated by other scholars or experts in the field before being accepted for publication.  A scholarly article is commonly an experimental or research study, or an in-depth theoretical or literature review. It is usually many more pages than a magazine article.

The clearest and most reliable indicator of a scholarly article is the presence of references or citations. Look for a list of works cited, a reference list, and/or numbered footnotes or endnotes. Citations are not merely a check against plagiarism. They set the article in the context of a scholarly discussion and provide useful suggestions for further research. 

Many of our databases allow you to limit your search to just scholarly articles. This is a useful feature, but it is not 100% accurate in terms of what it includes and what it excludes. You should still check to see if the article has references or citations.

The table below compares some of the differences between magazines (e.g. Psychology Today) and journals (e.g Journal of Abnormal Psychology).

How to find scholarly, peer-reviewed articles

• FAQ: How do I find peer-reviewed or scholarly articles?
• FAQ: How can I tell if an article is peer-reviewed?

To get a sense of the available research, you may want to start with a multidisciplinary article database such as  Academic Search Premier  or Business Source Complete (for management and business).  Then, you may want to do a more thorough search in additional specialized sources--see the link below.

• Academic Search Premier (EBSCO) A scholarly multidisciplinary database of periodical articles, most with full-text, through Ebscohost.
• Business Source Complete (EBSCO) A great starting point for articles on business and management topics including peer-reviewed journals and classic publications such Harvard Business Review. Also contains SWOT, industry, and market research reports.
• Additional specialized sources for finding journal articles Select the areas that correspond to your topic or academic program to find specialized journal article databases.

From a library database

When the PDF or HTML full-text is not available in one of our databases, use the "Full Text Finder" button. Full Text Finder will allow you to link to the article in another database. If no full text is available you may request an electronic copy of the article through Interlibrary Loan .

From another source (e.g. online, Google Scholar)

Many articles that you find online may require payment ( aka  paywall) to download the article. In many cases the library can get the article for you for free to keep you from having to pay out of pocket. For more information, please visit our: 

• Request Articles and Books page For information on how to request items depending on your campus or location
• Ask a Librarian Reach out to us in person or via phone, email or 24/7 chat. We are here to help

While  Google Scholar can be a useful source for finding journal articles, there are advantages found in using Webster University Libraries' databases, including:

• Features that let you customize your search
• Access to more full text materials
• Integration with other library services (e.g., chat, delivery services, etc.).

For more information on using Google Scholar, view the FAQ:  How can I connect Google Scholar to the Library?

Do not forget books when you are surveying the literature. They often provide historical information and overviews of current research in a topic area.

• Search for books in the Library Catalog
• Find an electronic book on a topic
• MOBIUS Catalog Here you can borrow from a consortium system of college & university libraries in Missouri and other states. (Some public libraries have also joined MOBIUS.)

Here you can search a large catalog of books and other materials owned by U.S libraries and some worldwide locations. This source shows local libraries where a given book is housed and also indicates if there is an ebook available.  

When something is published as an open access resource, it is published online and can be accessed for free with few or no copyright restrictions. Open access resources allows you to search, download, and cite researchers who have chosen to publish open access without paying for each article.

Searching through open access resources might be a great option for your research once you have exhausted the databases. Please note that sometimes, an open resource repository can be difficult to search through as often there are fewer ways to limit the search. 

• About Open Access guide This research guide provides information about Open Access (OA) and Open Educational Resources (OER) and includes a list of resources by discipline.

As you download articles and begin to identify helpful resources, you will need to develop a method of keeping track of this research. No matter the method you choose to use, make sure that: 

• Any article you've downloaded is saved in a labeled folder which is easy to access
• Any citation you have created is saved and accessible
• EBSCO tools If using an EBSCO database, this document provides information on how to create folders and save links to articles within the database.

You can certainly create a system for organizing your downloads, citations, and other electronic notes. Use the file storage system on your computer, or cloud computing software like  Google Drive  or  Dropbox . Create folders specifically for your project and save everything you think you might use. The benefit of managing your research this way is that these options are often free and allow you to have access to materials beyond your time as a Webster student.

There are a number of software programs available that help students store references and notes, create bibliographies, etc. While not needed for every assignment, they are useful for when you are gathering a large number of articles and other resources for projects such as capstone papers, theses, and dissertations. Some of the main citation management software applications are listed here.

Because the purpose of the literature review is to analyze the research on the topic and find relationships between resources, simply reading the resources and keeping notes may not be enough to help you see the connections between the resources. 

One option is to create a document with a chart used solely to compare the ideas and methods of various scholars and researchers. This is called a synthesis matrix . Before starting a matrix, you may want to identify a couple subtopics or themes to track within the articles you read. Other themes will reveal themselves as you read the literature and can be added to the chart. 

Below, you will find some examples of synthesis matrixes. Use inspiration from any of them to design  your own matrix which works best for your style and ideas. No matter the design of your matrix, some of the items you may want to compare across resources are: 

Publishing Date : Show how an idea has developed over time due to continuous research. 

Research Methods : Discuss findings based upon research type. You might ask yourself: How does the method used to collect the data impact the findings of the study? 

• Themes : Which topics are covered in the article and what does the author believe about that topic? 
• Literature Review: Synthesizing Multiple Sources (Indiana University Bloomington)
• Synthesis Matrix (NC State University)
• Writing a Literature Review and Using a Synthesis Matrix PDF Document created by North Carolina State University and posted by Wayne State University University.
• << Previous: Getting Started
• Next: Literature Review as a Product: Organizing your Writing >>
• Last Updated: Mar 21, 2024 11:13 AM
• URL: https://library.webster.edu/litreview

English and American Literature

• Dictionaries, Encyclopedias, and Histories
• Finding Books
• Historical Periodicals
• Digital Collections (NYU subscription)
• Digital Collections and Digital Humanities projects (open access)
• Literary Criticism
• Biographies
• Publishing and Bibliographies
• Book Reviews
• Citing Sources
• MLA Style and Citation Management Tools
• Special Collections and Archives
• Faculty / PhD resources
• English 101 resources

Using BobCat to Find Literature Journals

One of the best methods of staying current with your field is by browsing the journal literature. Literature journals help you keep up with the latest scholarship and provide potential venues for publishing your own research. The numerous online databases available to you through the library allow you to search through the contents of these journals (and many others), but browsing through an individual journal will allow you the opportunity for more specific topic research.

You can use the advanced search options in BobCat , the NYU Libraries' catalog, to limit the material type to "Journal" while searching for a subject or keyword of your choice. Here are some sample subject searches, limited to journals:

• American literature
• Comparative literature
• English literature -- History and criticism
• Language and literature

Try adding in additional terms to narrow your search down, or use the "Tweak my results" options on the left side of the page if you're getting too many results.

If this method doesn't work for you, you may want to look at a more specialized list of journals (see "Using the MLA Directory of Periodicals" below).

Using the MLA Directory of Periodicals to Find Literature Journals

If you want to look at a list of journals for any topic within literary studies, try the MLA Directory of Periodicals (DOP). Each entry in the DOP provides information about the journal's scope, subject matter, publication schedule, editors, submission guidelines, peer review status, and more. You can access the DoP from within the MLA International Bibliography database, or directly through its own link (below and in the library's Articles & Databases portal).

The video below, from the Modern Language Association, explains the basics of using the DoP.

• MLA Directory of Periodicals This link opens in a new window The MLA Directory of Periodicals provides detailed information on journals and book series that cover literature, literary theory, dramatic arts, folklore, language, linguistics, pedagogy, rhetoric and composition, and the history of printing and publishing. Articles published in works listed in the directory are indexed in the MLA International Bibliography. The directory is a valuable resource for scholars seeking outlets to publish their work

What Is the MLA Directory of Periodicals? (on EBSCO) from Modern Language Association on Vimeo .

Closed captioning is available through the Vimeo platform. No transcript is available.

This video is one of many MLA International Bibliography tutorials available on the MLA's own website.

A Sampling of Literature Journals

Here is a small sampling of the literature journals available to you through the library:

• Book History Book History is devoted to every aspect of the history of the book, broadly defined as the history of the creation, dissemination, and reception of script and print. It publishes research on the social, economic, and cultural history of author- ship, editing, printing, the book arts, publishing, the book trade, periodicals, newspapers, ephemera, copyright, censorship, literary agents, libraries, literary criticism, canon formation, literacy, literary education, reading habits, and reader response.
• Contemporary Literature Contemporary Literature covers the whole range of critical practices, offering new perspectives in contemporary literary studies. CL features in-depth interviews with significant writers, broad-ranging articles written by leaders in the field, and book reviews of important critical studies.
• Critical Inquiry In CI new ideas and reconsideration of those traditional in criticism and culture are granted a voice. The wide interdisciplinary focus creates surprising juxtapositions and linkages of concepts, offering new grounds for theoretical debate. In CI, authors entertain and challenge while illuminating such issues as improvisations, the life of things, Flaubert, and early modern women's writing. CI comes full circle with the electrically charged debates between contributors and their critics.
• ELH Since 1934, ELH has published superior studies that interpret the conditions affecting major works in English and American literature.
• Journal of Victorian Culture Journal of Victorian Culture is essential reading for scholars of the Victorian period. It provides an international forum for discussion and debate on all aspects of Victorian history and culture in a diverse range of formats, including articles, perspectives, roundtables and a section of substantial reviews.
• New Literary History New Literary History focuses on questions of theory, method, interpretation, and literary history. Rather than espousing a single ideology or intellectual framework, it canvasses a wide range of scholarly concerns.
• Publications of the Modern Language Association of America (PMLA) PMLA is the journal of the Modern Language Association of America. Since 1884, PMLA has published members' essays judged to be of interest to scholars and teachers of language and literature.
• SEL: Studies in English Literature 1500-1900 SEL focuses on four fields of British literature in rotating, quarterly issues: English Renaissance, Tudor and Stuart Drama, Restoration and Eighteenth Century, and Nineteenth Century. The editors select learned, readable papers that contribute significantly to the understanding of British literature from 1500 to 1900.

Need to find out if a journal is peer-reviewed? Use Ulrichs

• Ulrichsweb This link opens in a new window Ulrichsweb is an authoritative source of detailed information on periodicals of all types -- academic and scholarly journals, Open Access publications, peer-reviewed titles, popular magazines, newspapers, newsletters, and more from around the world.
• << Previous: Finding Articles via databases
• Next: Historical Periodicals >>
• Last Updated: Mar 25, 2024 2:09 PM
• URL: https://guides.nyu.edu/english-and-american-literature

Academic Integrity and Artificial Intelligence in Higher Education (HE) Contexts: A Rapid Scoping Review

• Helen Pethrick University of Calgary
• K. Alix Hayden University of Calgary
• Jason Wiens University of Calgary
• Brenda McDermott University of Calgary

Artificial Intelligence (AI) developments challenge higher education institutions’ teaching, learning, assessment, and research practices. To contribute timely and evidence-based recommendations for upholding academic integrity, we conducted a rapid scoping review focusing on what is known about academic integrity and AI in higher education. We followed the Updated Reviewer Manual for Scoping Reviews from the Joanna Briggs Institute (JBI) and the Preferred Reporting Items for Systematic reviews Meta-Analysis for Scoping Reviews (PRISMA-ScR) reporting standards. Five databases were searched, and the eligibility criteria included higher education stakeholders of any age and gender engaged with AI in the context of academic integrity from 2007 through November 2022 and available in English. The search retrieved 2223 records, of which 14 publications with mixed methods, qualitative, quantitative, randomized controlled trials, and text and opinion studies met the inclusion criteria. The results showed bounded and unbounded ethical implications of AI. Perspectives included: AI for cheating; AI as legitimate support; an equity, diversity, and inclusion lens into AI; and emerging recommendations to tackle AI implications in higher education. The evidence from the sources provides guidance that can inform educational stakeholders in decision-making processes for AI integration, in the analysis of misconduct cases involving AI, and in the exploration of AI as legitimate assistance. Likewise, this rapid scoping review signals key questions for future research, which we explore in our discussion.

Author Biographies

Beatriz antonieta moya, university of calgary.

Beatriz Moya is a Ph.D. candidate in the Werklund School of Education at the University of Calgary in the Educational Research program, specializing in Leadership. Her primary motivation as a student researcher is to contribute to the continuous transformation of higher education institutions’ cultures to pursue academic and research integrity and social justice. For this reason, Beatriz situates herself at the intersections of the Scholarship of Teaching and Learning (SoTL), Leadership, and Academic Integrity. In this space, Beatriz is currently involved in projects seeking to contribute to a ‘glocal’ scholarly dialogue concerning academic integrity as a teaching and learning imperative. For instance, Beatriz is exploring the situated meanings and experiential insights from academic integrity educational leaders, and is also a contributing author for a chapter on academic integrity policy in Latin America in the Handbook of Academic Integrity (2nd ed.). 

Sarah Elaine Eaton, University of Calgary

Dr. Sarah Elaine Eaton, PhD, is an Associate Professor of Education at the University of Calgary, Canada. She served as the inaugural Educational Leader in Residence, Academic Integrity at Taylor Institute for Teaching and Learning, University of Calgary. Dr. Eaton’s research focuses on academic ethics in higher education. Her work can be found in the British Educational Research Journal, the Journal of Academic Ethics, and the Journal of Educational Thought and Interchange, among other places. She is the Editor-in-Chief of the International Journal for Educational Integrity (Springer Nature) and co-founder and co-editor of Canadian Perspectives on Academic Integrity. In 2020 she received the National Research and Scholarship award from the Canadian Society for the Study of Higher Education (CSSHE) for her contributions to research on academic integrity in Canadian higher education. In 2022, she received the outstanding research award from the European Network for Academic Integrity (ENAI). Her books include Plagiarism in Higher Education: Tackling Tough Topics in Academic Integrity, Academic Integrity in Canada: An Enduring and Essential Challenge (Eaton & Christensen Hughes, eds.), Contract Cheating in Higher Education: Global Perspectives on Theory, Practice, and Policy (Eaton, Curtis, Stoesz, Clare, Rundle, & Seeland, eds.) and Ethics and Integrity in Teacher Education (Eaton & Khan, eds.). 

Helen Pethrick, University of Calgary

Helen Pethrick, MA, is a researcher and educator in Calgary, Alberta, Canada. Helen’s research encompasses academic integrity in higher education, post-secondary student mental health, and mentorship in academia. Helen has interdisciplinary expertise in systematic literature review methodology, qualitative research, project management, and knowledge exchange. With Dr. Sarah Elaine Eaton and Jamie J. Carmichael, Helen is co-Editor of the volume Fake Degrees and Fraudulent Credentials in Higher Education (Eaton, Carmichael, & Pethrick, forthcoming 2023). Currently, Helen’s role is Research Associate, Academic Integrity, which involves acting as Project Manager (administrative) for the Academic Integrity and Artificial Intelligence project at the University of Calgary. 

K. Alix Hayden, University of Calgary

Dr. Alix Hayden is a Librarian at the University of Calgary.

Robert Brennan

Holds a PhD in mechanical engineering from the University of Calgary. He is a Professor of Mechanical and Manufacturing Engineering at the University of Calgary, holds the NSERC Chair in Design Engineering, and has served as President of the Canadian Engineering Education Association (CEEA). His research interests range from engineering education to intelligent automation and control systems.  

Jason Wiens, University of Calgary

Dr. Wiens is a Professor (Teaching) in the Department of English at the University of Calgary. His areas of research interest include contemporary poetry, Canadian literature, archival studies, literary audio, and pedagogy.  He has published articles in numerous journals including LIT: Literature Interpretation Theory, Canadian Literature, Studies on Canadian Writing, and Canadian Poetry. He has recently guest edited a special issue of English Studies in Canada on "Pedagogies of the Archive." He is currently a co-investigator on the SSHRC Partnership project The SpokenWeb, and is a co-investigator on the Academic Integrity and Artificial Intelligence project at the University of Calgary.  

Brenda McDermott, University of Calgary

Dr. McDermott completed her PhD in communication studies at the University of Calgary.  Her research involves looking at ableism embedded in teaching and learning practices, particular assessment. She regularly provided training to faculty to help creating learning environments that reflect the diversity of learners.  

Alonso, A. N. (2022). Online translators in online language assessments. CALL-EJ, 23(3), 115-135. http://callej.org/journal/23-3/Alonso2022.pdf

Anselmo, L., Kendon, T. & Moya, B. (2023). A First Response to Assessment and ChatGPT in your Courses. Taylor Institute for Teaching and Learning. https://taylorinstitute.ucalgary.ca/first-response-assessment-and-chatgpt

Anson, C. M. (2022). AI-based text generation and the social construction of “fraudulent authorship”: A revisitation. Composition Studies, 50(1), 37-46.

Aromataris, E., & Munn, A. (2020). JBI manual for evidence synthesis. JBI. https://doi.org/10.46658/JBIMES-20-01

Australian Academic Integrity Network (AAIN). (2023). AAIN generative artificial intelligence guidelines. https://www.teqsa.gov.au/sites/default/files/2023-04/aain-generative-ai-guidelines.pdf

Barker, T., Stone J. C., Sears K., Klugar M., Tufanaru, C., & Leonardi-Bee, J. (2023). The revised JBI critical appraisal tool for the assessment of risk of bias for randomized controlled trials. JBI Evidence Synthesis, 21(3), 494-506. https://journals.lww.com/jbisrir/Fulltext/2023/03000/The_revised_JBI_critical_appraisal_tool_for_the.5.aspx

Bearman, M., & Luckin, R. (2020). Preparing university assessment for a world with AI: Tasks for human intelligence. In M. Bearman, P. Dawson, R. Ajjawi, J. Tai & D. Boud (Eds.), Re-imagining university assessment in a digital world (pp. 49-63). Springer International Publishing. https://doi.org/https://doi.org/10.1007/978-3-030-41956-1

Brake, J. (2022). Education in the world of ChatGPT. The Absent-Minded Professor https://joshbrake.substack.com/p/education-in-the-world-of-chatgpt?utm_source=direct&utm_campaign=post&utm_medium=web

Bretag, T. (2016). Defining academic integrity: International perspectives – Introduction. In T. Bretag (Ed.), Handbook of Academic Integrity (1st edition). Springer Singapore: 3-5. https://link.springer.com/content/pdf/10.1007/978-981-287-098-8_76.pdf

Brusini, A. (2023). ChatGPT: A brief introduction and considerations for academic integrity. The Innovative Instructor. https://ii.library.jhu.edu/2023/01/30/chatgpt-a-brief-introduction-and-considerations-for-academic-integrity/

Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Tat Lee, Y. Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T., & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with GPT-4. https://arxiv.org/pdf/2303.12712.pdf

Cassidy, C. (2023, January 10). Australian universities to return to ‘pen and paper’ exams after students caught using AI to write essays. The Guardian. https://www.theguardian.com/australia-news/2023/jan/10/universities-to-return-to-pen-and-paper-exams-after-students-caught-using-ai-to-write-essays

Chen, M. H., Huang, S.-T., Chang, J.S., & Liou, H.-C. (2015). Developing a corpus-based paraphrase tool to improve EFL learners' writing skills. Computer Assisted Language Learning, 28(1), 22-40. http://dx.doi.org/10.1080/09588221.2013.783873

Cochrane, T., & Ryan, T. (2023). ChatGPT and academic integrity: Options for adapting. Melbourne Centre for the Study of Higher Education. https://melbourne-cshe.unimelb.edu.au/__data/assets/pdf_file/0008/4533218/ChatGPT-and-Academic-Integrity.pdf

Dans, E. (2019, February 6). Meet Bertie, Heliograf and Cyborg, the new journalists on the block. Forbes. https://www.forbes.com/sites/enriquedans/2019/02/06/meet-bertie-heliograf-and-cyborg-the-new-journalists-on-the-block/?sh=669bf965138d

Dawson, P. (2020). E-Cheating, assessment security and artificial intelligence. In P. Dawson (Ed.), Defending assessment security in a digital world (pp. 83-97). Routledge. http://doi.org/10.4324/9780429324178

Delaney, M. (2023). ChatGPT, AI language bot, can pass business, law and medical exams. Washington Times. https://www.washingtontimes.com/news/2023/jan/29/chatgpt-ai-language-bot-can-pass-business-law-and-/

Delisio, L. A., & Butaky, C. A. (2019). UDL and assistive technology: Utilizing technology beyond mere accessibility. In W. W. Murawski & K. L. Scott (Eds.), What really works with Universal Design for Learning (pp. 157-172). Corwin.

Dignum, V. (2021). The role and challenges of education for responsible AI. London Review of Education, 19(1), 1-11. https://ezproxy.lib.ucalgary.ca/login?url=https://search.ebscohost.com/login.aspx?direct=true&db=a9h&AN=157082792&site=ehost-live

Dinneen, C. (2021). Students’ use of digital translation and paraphrasing tools in written assignments on direct entry English programs. English Australia Journal, 37(1). https://files.eric.ed.gov/fulltext/EJ1341751.pdf

Ducar, C., & Schocket, D. H. (2018). Machine translation and the L2 classroom: Pedagogical solutions for making peace with Google Translate. Foreign Language Annals 51(4), 779-795.

Eaton, S. E. (2022). Student Academic Integrity: A Handbook for Academic Staff and Teaching Assistants. Taylor Institute of Teaching and Learning. https://taylorinstitute.ucalgary.ca/resources/student-academic-integrity-handbook

Eaton, S. E. (2023). Sarah’s Thoughts: Artificial Intelligence and Academic Integrity. Learning, Teaching and Leadership: A blog for educators, researchers and other thinkers. https://drsaraheaton.wordpress.com/2022/12/09/sarahs-thoughts-artificial-intelligence-and-academic-integrity/

Eaton, S. E., & L. Anselmo (2023). Teaching and learning with artificial intelligence apps. Taylor Institute for Teaching and Learning. https://taylorinstitute.ucalgary.ca/teaching-with-AI-apps

Eaton, S. E., Pethrick, H., & Turner, K. L. (2023). Academic integrity and student mental well-being: A rapid review. Canadian Perspectives on Academic Integrity, 5(2). https://doi.org/10.11575/cpai.v5i2.73748

Education USA. (n.d.). The U.S. educational system - Glossary. https://educationusa.state.gov/experience-studying-usa/us-educational-system/glossary#T

European Union. (2022). Ethical guidelines on the use of artificial intelligence (AI) in teaching and learning for educators. https://op.europa.eu/en/publication-detail/-/publication/d81a0d54-5348-11ed-92ed-01aa75ed71a1/language-en

Flinders University (2023). What is artificial intelligence? Flinders University Library. https://library.flinders.edu.au/students/ai

Fyfe, P. (2022). How to cheat on your final paper: Assigning AI for student writing. AI & Society. https://link.springer.com/article/10.1007/s00146-022-01397-z

Foltýnek, T., Bjelobaba, S., Glendinning, I, Khan, Z.R., Santos, R. Pavletic, P., & Kravjar, J. (2023). ENAI Recommendations on the ethical use of Artificial Intelligence in Education. International Journal for Academic Integrity, 19, 12. https://doi.org/10.1007/s40979-023-00133-4

Gero, K. I., et al. (2022). Sparks: Inspiration for science writing using language models. DIS '22: Designing Interactive Systems Conference, Association for Computing Machinery, Inc. https://arxiv.org/abs/2110.07640

Government of Canada. (2022). Education in Canada: Post-secondary. https://www.canada.ca/en/immigration-refugees-citizenship/services/new-immigrants/new-life-canada/education/types-school/post-secondary.html

Groves, M., & Mundt, K. (2021). A ghostwriter in the machine? Attitudes of academic staff towards machine translation use in internationalised Higher Education. Journal of English for Academic Purposes, 50. https://doi.org/10.1016/j.jeap.2021.100957

Hartling, L., Guise, J. M., Hempel, S., Featherstone, R., Mitchell, M. D., Motu’Apuaka, M. L., Robinson, K. A., Schoelles, K., Totten, A., Whitlock, E., Wilt, T. J., Anderson, J., Berliner, E., Gozu, A., Kato, E., Paynter, R., & Umscheid, C. A. (2017). Fit for purpose: Perspectives on rapid reviews from end-user interviews. Systematic Reviews, 6(32). https://doi.org/10.1186/s13643-017-0425-7

Hemsley, B., Power, E., & Given, F. (January 18, 2023). Will AI tech like ChatGPT improve inclusion for people with communication disability. The Conversation. https://theconversation.com/will-ai-tech-like-chatgpt-improve-inclusion-for-people-with-communication-disability-196481

Hiatt, B. (2023, January 28). ChatGPT: Educators hold emergency meetings as AI disrupts schools and universities across Australia. The West Australian. https://thewest.com.au/technology/chatgpt-educators-scramble-as-ai-disrupts-schools-and-universities-across-australia-c-9565061

Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M., & Vedel, I. (2018). Mixed methods appraisal tool (MMAT). McGill University. http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/127916259/MMAT_2018_criteria-manual_2018-08-01_ENG.pdf

Hotson, B., & Bell, S. (2023). Academic writing and ChatGPT: Step back to step forward. Canadian Writing Centers Association. https://cwcaaccr.com/2023/04/09/chatgpt-step-back-to-step-forward/ .

ICAI (2021). The Fundamental Values of Academic Integrity. https://academicintegrity.org/resources/fundamental-values

Keith, T. (2023). Combating Academic Dishonesty, Part 6: ChatGPT, AI, and Academic Integrity. Academic Technology Solutions. https://academictech.uchicago.edu/2023/01/23/combating-academic-dishonesty-part-6-chatgpt-ai-and-academic-integrity/

Khan, Z. R. (2023). Artificial intelligence content generators in education for schools and universities: A good practice guide, European Network for Academic Integrity Working Group Centre for Academic Integrity in the UAE. University of Wollongong in Dubai. https://academicintegrity-uae.com/category/faculty- resources/

Kim, J. (2018, April 25). Are the professions (disciplines?) of educational developer and learning designer merging? Or not? Inside Higher Ed. https://www.insidehighered.com/digital-learning/blogs/technology-and-learning/are-professions-disciplines-educational-developer-and

Kublik, D., & Saboo, S. (2022). GPT-3: Building innovative NLP products using large language models. O’Reilly.

Lesage, J., Brennan, R., Eaton, S. E., Moya, B., McDermott, B., Wiens, J., & Herrero, K. (2023). Exploring natural language processing in mechanical engineering education: Implications for academic integrity. International Journal of Mechanical Engineering Education. https://journals.sagepub.com/doi/full/10.1177/03064190231166665

Lockwood, C., Munn, Z., & Porritt, K. Qualitative research synthesis: Methodological guidance for systematic reviewers utilizing meta-aggregation. International Journal of Evidence Based Healthcare, 13(3), 179–187. https://pubmed.ncbi.nlm.nih.gov/26262565/

Lunny, C., Antony, J., Ríos, P., Williams, C., Ramkissoon, N., Straus, S. E., & Tricco, A. C. (2021). Safety and effectiveness of dose-sparing strategies for intramuscular seasonal influenza vaccine: A rapid scoping review. BMJ Open, 11. https://doi.org/10.1136/bmjopen-2021-050596

McArthur A, Klugarova J, Yan H, & Florescu S. Chapter 4: Systematic reviews of text and opinion. In E. Aromataris & Z. Munn (Eds), JBI Manual for Evidence Synthesis. https://doi.org/10.46658/JBIMES-20-01

McNeill, L., & Chaudhuri, A. (2023). The opportunities of ChatGPT. University of British Columbia. https://academicintegrity.ubc.ca/2023/02/15/the-opportunities-of-chatgpt/

Merine, R., Purkayastha, S. (2022). Risks and benefits of AI-generated text summarization for expert level content in graduate health informatics. 2022 IEEE 10th International Conference on Healthcare Informatics (ICHI), Institute of Electrical and Electronics Engineers Inc. https://ieeexplore.ieee.org/document/9874678/

Mindzak, M. (2020, February 17). What happens when a machine can write as well as an academic? University Affairs. https://www.universityaffairs.ca/opinion/in-my-opinion/what-happens-when-a-machine-can-write-as-well-as-an-academic/

Monash University. (2023). Generative AI and assessment. https://www.monash.edu/learning-teaching/teachhq/Teaching-practices/artificial-intelligence/generative-ai-and-assessment

Moya, B. A., Eaton, S. E., Pethrick, H., Hayden, K. A., Brennan, R., Wiens, J., McDermott, B., & Lesage, J. (2023). Academic integrity and artificial intelligence in higher education contexts: A rapid scoping review protocol. Canadian Perspectives on Academic Integrity, 5(2). https://journalhosting.ucalgary.ca/index.php/ai/article/view/75990

Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18(143). https://doi.org/10.1186/s12874-018-0611-x

Munn, Z., Pollock, D., Khalil, H., Alexander, L., Mclnerney, P., Godfrey, C. M., Peters, M., & Tricco, A. (2022). What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI Evidence Synthesis, 20(4), 950-952. https://pubmed.ncbi.nlm.nih.gov/35249995/

Murphy, S. (2023). ChatGPT passes exams from law and business schools. CNN Business. https://www.cnn.com/2023/01/26/tech/chatgpt-passes-exams/index.html

Nature (2023). Tools such as ChatGPT threaten transparent science. Nature (613). https://www.nature.com/articles/d41586-023-00191-1

OECD. (2002). Education at a glance: OECD indicators 2002. https://www.oecd-ilibrary.org/docserver/eag-2002-en.pdf?expires=1660686371&id=id&accname=guest&checksum=91C5FA8FBA74551E2A1C380F1079D47F

Ouyang, F., Zheng, F., & Jiao, P. (2022). Artificial intelligence in online higher education: A systematic review of empirical research from 2011 to 2020. Education and Information Technologies 27(6): 7893-7925. https://link.springer.com/article/10.1007/s10639-022-10925-9

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hrobjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuiness, L. A., Stewart, L. A., Thomas, J., Tricco, A.C., Welch, V. A, Whiting, P., & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372(71). https://doi.org/10.1136/bmj.n71

Perkins, M. (2023). Academic integrity considerations of AI Large Language Models in the post-pandemic era: ChatGPT and beyond. Journal of University Teaching and Learning Practice, 20(2). https://ro.uow.edu.au/cgi/viewcontent.cgi?article=3071&context=jutlp

Peters, M. D., Godfrey, C., McInerney, P., Munn, Z., Tricco, A., & Khalil, H. (2020). Chapter 11: Scoping reviews. In E. Aromataris & Z. Munn (Eds.), JBI manual for evidence synthesis. JBI. https://doi.org/10.46658/jbimes-20-12

Peters, M. D. J., Godfrey, C., McInerney, P., Khalil, H., Larsen, P., Marnie, C., Pollock, D., Tricco, A. C., & Munn, Z. (2022). Best practice guidance and reporting items for the development of scoping review protocols. JBI Evidence Synthesis, 20(4), 953-968. https://doi.org/10.11124/JBIES-21-00242

Popenici, S. A. D., & Kerr, S. (2017). Exploring the impact of artificial intelligence on teaching and learning in higher education. Research and Practice in Technology Enhanced Learning, 12(22). https://doi.org/10.1186/s41039-017-0062-8

Prentice, F. M., & Kinden, C. E. (2018). Paraphrasing tools, language translation tools and plagiarism: An exploratory study. International Journal for Educational Integrity, 14. https://edintegrity.biomedcentral.com/articles/10.1007/s40979-018-0036-7

Roe, J., & Perkins, M. (2022). What are automated paraphrasing tools and how do we address them? A review of a growing threat to academic integrity. International Journal for Educational Integrity, 18(1). https://edintegrity.biomedcentral.com/articles/10.1007/s40979-022-00109-w

Roe, J., Renandya, W., & Jacobs, G. (2023). A review of AI-powered writing tools and their implications for academic integrity in the language classroom. Journal of English and Applied Linguistics 2(1). https://animorepository.dlsu.edu.ph/jeal/vol2/iss1/3/

Rogerson, A. M., & McCarthy, G. (2017). Using internet based paraphrasing tools: Original work, patchwriting or facilitated plagiarism? International Journal for Educational Integrity, 13(2). https://edintegrity.biomedcentral.com/articles/10.1007/s40979-016-0013-y

Sabzalieva, E., & Valentini, A. (2023). ChatGPT and artificial intelligence in higher education: Quick start guide. UNESCO. https://www.iesalc.unesco.org/wp-content/uploads/2023/04/ChatGPT-and-Artificial-Intelligence-in-higher-education-Quick-Start-guide_EN_FINAL.pdf

Sharples, M. (2022). Automated essay writing: An AIED opinion. International Journal of Artificial Intelligence in Education, 32(4), 1119-1126. https://link.springer.com/article/10.1007/s40593-022-00300-7

Sloan, K. (2023, January 25). ChatGPT passes law school exams despite 'mediocre' performance. Reuters. https://www.reuters.com/legal/transactional/chatgpt-passes-law-school-exams-despite-mediocre-performance-2023-01-25/

Statistics Canada. (2018). National Occupational Classification (NOC) 2011. https://www23.statcan.gc.ca/imdb/p3VD.pl?Function=getVD&TVD=122372&CVD=122375&CPV=511&CST=01012011&CLV=3&MLV=4&D=1

Statistics Canada. (2022). Table 37-10-0076-01 number of full-time teaching staff at Canadian universities, by rank, sex, inactive. https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=3710007601

Sullivan, M., Kelly, A., & McLaughlan, P. (2023). ChatGPT in higher education: Considerations for academic integrity and student learning. Journal of Applied Learning & Teaching 6(1). https://journals.sfu.ca/jalt/index.php/jalt/article/view/731/559

Tauginiené, L., Gaižauskaité, I., Glendinning, I., Kravjar, J., Ojsteršek, M., Ribeiro, L., Odiņeca, T., Marino, F., Cosentino, M., Sivasubramaniam, S., Foltýnek, T. Glossary for Academic Integrity. https://www.academicintegrity.eu/wp/wp-content/uploads/2023/02/EN-Glossary_revised_final_24.02.23.pdf

Tertiary Education Quality and Standards Agency (TEQSA). (2023). Artificial intelligence. https://www.teqsa.gov.au/guides-resources/higher-education-good-practice-hub/artificial-intelligence

Tricco, A. C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J., Perrier, L., Hutton, B., Moher, D., & Straus, S. E. (2015). A scoping review of rapid review methods. BMC Medicine, 13(224). https://doi.org/10.1186/s12916-015-0465-6

UNESCO OECD EUROSTAT. (2001). 2001 Data Collection on Education Systems: Definitions, Explanations and Instructions.

UNESCO. (2021). The ethics of artificial intelligence. https://unesdoc.unesco.org/ark:/48223/pf0000381137

Wollscheid, S., & Tripney, J. (2021). Rapid reviews as an emerging approach to evidence synthesis in education. London Review of Education, 19(1). https://doi.org/10.14324/lre.19.1.32

Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education–where are the educators? International Journal of Educational Technology in Higher Education, 16(1), 1-27. https://doi.org/10.1186/s41239-019-0171-0

How to Cite

• Endnote/Zotero/Mendeley (RIS)

Most read articles by the same author(s)

• Beatriz Antonieta Moya, Sarah Elaine Eaton, Helen Pethrick, K. Alix Hayden, Robert Brennan, Jason Wiens, Brenda McDermott, Jonathan Lesage, Academic Integrity and Artificial Intelligence in Higher Education Contexts: A Rapid Scoping Review Protocol , Canadian Perspectives on Academic Integrity: Vol. 5 No. 2 (2023)
• Sarah Elaine Eaton, Helen Pethrick, Kristal Louise Turner, Academic Integrity and Student Mental Well-Being: A Rapid Review , Canadian Perspectives on Academic Integrity: Vol. 5 No. 2 (2023)
• Alix Hayden, Sarah Elaine Eaton, Katherine Crossman, Lee-Ann Penaluna, Bartlomiej A. Lenart, Text-matching software in post-secondary contexts: A systematic review protocol , Canadian Perspectives on Academic Integrity: Vol. 3 No. 1 (2020): Volume 3, Issue 1
• Helen Pethrick, Sarah Elaine Eaton, Kristal Louise Turner, Academic Integrity and Mental Well-being: Exploring an Unexplored Relationship , Canadian Perspectives on Academic Integrity: Vol. 4 No. 2 (2021): Volume 4, Issue 2
• Beatriz Moya, Sarah Elaine Eaton, Helen Pethrick, Robert Brennan, Jason Wiens, Brenda McDermott, Jonathan Lesage, A Rapid Scoping Review on Academic Integrity and Algorithmic Writing Technologies , Canadian Perspectives on Academic Integrity: Vol. 6 No. 1 (2023)

Developed By

• Français (Canada)

Information

• For Readers
• For Authors
• For Librarians

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Med Libr Assoc
• v.106(4); 2018 Oct

A systematic approach to searching: an efficient and complete method to develop literature searches

Associated data.

Creating search strategies for systematic reviews, finding the best balance between sensitivity and specificity, and translating search strategies between databases is challenging. Several methods describe standards for systematic search strategies, but a consistent approach for creating an exhaustive search strategy has not yet been fully described in enough detail to be fully replicable. The authors have established a method that describes step by step the process of developing a systematic search strategy as needed in the systematic review. This method describes how single-line search strategies can be prepared in a text document by typing search syntax (such as field codes, parentheses, and Boolean operators) before copying and pasting search terms (keywords and free-text synonyms) that are found in the thesaurus. To help ensure term completeness, we developed a novel optimization technique that is mainly based on comparing the results retrieved by thesaurus terms with those retrieved by the free-text search words to identify potentially relevant candidate search terms. Macros in Microsoft Word have been developed to convert syntaxes between databases and interfaces almost automatically. This method helps information specialists in developing librarian-mediated searches for systematic reviews as well as medical and health care practitioners who are searching for evidence to answer clinical questions. The described method can be used to create complex and comprehensive search strategies for different databases and interfaces, such as those that are needed when searching for relevant references for systematic reviews, and will assist both information specialists and practitioners when they are searching the biomedical literature.

INTRODUCTION

Librarians and information specialists are often involved in the process of preparing and completing systematic reviews (SRs), where one of their main tasks is to identify relevant references to include in the review [ 1 ]. Although several recommendations for the process of searching have been published [ 2 – 6 ], none describe the development of a systematic search strategy from start to finish.

Traditional methods of SR search strategy development and execution are highly time consuming, reportedly requiring up to 100 hours or more [ 7 , 8 ]. The authors wanted to develop systematic and exhaustive search strategies more efficiently, while preserving the high sensitivity that SR search strategies necessitate. In this article, we describe the method developed at Erasmus University Medical Center (MC) and demonstrate its use through an example search. The efficiency of the search method and outcome of 73 searches that have resulted in published reviews are described in a separate article [ 9 ].

As we aimed to describe the creation of systematic searches in full detail, the method starts at a basic level with the analysis of the research question and the creation of search terms. Readers who are new to SR searching are advised to follow all steps described. More experienced searchers can consider the basic steps to be existing knowledge that will already be part of their normal workflow, although step 4 probably differs from general practice. Experienced searchers will gain the most from reading about the novelties in the method as described in steps 10–13 and comparing the examples given in the supplementary appendix to their own practice.

CREATING A SYSTEMATIC SEARCH STRATEGY

Our methodology for planning and creating a multi-database search strategy consists of the following steps:

• Determine a clear and focused question
• Describe the articles that can answer the question
• Decide which key concepts address the different elements of the question
• Decide which elements should be used for the best results
• Choose an appropriate database and interface to start with
• Document the search process in a text document
• Identify appropriate index terms in the thesaurus of the first database
• Identify synonyms in the thesaurus
• Add variations in search terms
• Use database-appropriate syntax, with parentheses, Boolean operators, and field codes
• Optimize the search
• Evaluate the initial results
• Check for errors
• Translate to other databases
• Test and reiterate

Each step in the process is reflected by an example search described in the supplementary appendix .

1. Determine a clear and focused question

A systematic search can best be applied to a well-defined and precise research or clinical question. Questions that are too broad or too vague cannot be answered easily in a systematic way and will generally result in an overwhelming number of search results. On the other hand, a question that is too specific will result into too few or even zero search results. Various papers describe this process in more detail [ 10 – 12 ].

2. Describe the articles that can answer the question

Although not all clinical or research questions can be answered in the literature, the next step is to presume that the answer can indeed be found in published studies. A good starting point for a search is hypothesizing what the research that can answer the question would look like. These hypothetical (when possible, combined with known) articles can be used as guidance for constructing the search strategy.

3. Decide which key concepts address the different elements of the question

Key concepts are the topics or components that the desired articles should address, such as diseases or conditions, actions, substances, settings, domains (e.g., therapy, diagnosis, etiology), or study types. Key concepts from the research question can be grouped to create elements in the search strategy.

Elements in a search strategy do not necessarily follow the patient, intervention, comparison, outcome (PICO) structure or any other related structure. Using the PICO or another similar framework as guidance can be helpful to consider, especially in the inclusion and exclusion review stage of the SR, but this is not necessary for good search strategy development [ 13 – 15 ]. Sometimes concepts from different parts of the PICO structure can be grouped together into one search element, such as when the desired outcome is frequently described in a certain study type.

4. Decide which elements should be used for the best results

Not all elements of a research question should necessarily be used in the search strategy. Some elements are less important than others or may unnecessarily complicate or restrict a search strategy. Adding an element to a search strategy increases the chance of missing relevant references. Therefore, the number of elements in a search strategy should remain as low as possible to optimize recall.

Using the schema in Figure 1 , elements can be ordered by their specificity and importance to determine the best search approach. Whether an element is more specific or more general can be measured objectively by the number of hits retrieved in a database when searching for a key term representing that element. Depending on the research question, certain elements are more important than others. If articles (hypothetically or known) exist that can answer the question but lack a certain element in their titles, abstracts, or keywords, that element is unimportant to the question. An element can also be unimportant because of expected bias or an overlap with another element.

Schema for determining the optimal order of elements

Bias in elements

The choice of elements in a search strategy can introduce bias through use of overly specific terminology or terms often associated with positive outcomes. For the question “does prolonged breastfeeding improve intelligence outcomes in children?,” searching specifically for the element of duration will introduce bias, as articles that find a positive effect of prolonged breastfeeding will be much more likely to mention time factors in their titles or abstracts.

Overlapping elements

Elements in a question sometimes overlap in their meaning. Sometimes certain therapies are interventions for one specific disease. The Lichtenstein technique, for example, is a repair method for inguinal hernias. There is no need to include an element of “inguinal hernias” to a search for the effectiveness of the Lichtenstein therapy. Likewise, sometimes certain diseases are only found in certain populations. Adding such an overlapping element could lead to missing relevant references.

The elements to use in a search strategy can be found in the plot of elements in Figure 1 , by following the top row from left to right. For this method, we recommend starting with the most important and specific elements. Then, continue with more general and important elements until the number of results is acceptable for screening. Determining how many results are acceptable for screening is often a matter of negotiation with the SR team.

5. Choose an appropriate database and interface to start with

Important factors for choosing databases to use are the coverage and the presence of a thesaurus. For medically oriented searches, the coverage and recall of Embase, which includes the MEDLINE database, are superior to those of MEDLINE [ 16 ]. Each of these two databases has its own thesaurus with its own unique definitions and structure. Because of the complexity of the Embase thesaurus, Emtree, which contains much more specific thesaurus terms than the MEDLINE Medical Subject Headings (MeSH) thesaurus, translation from Emtree to MeSH is easier than the other way around. Therefore, we recommend starting in Embase.

MEDLINE and Embase are available through many different vendors and interfaces. The choice of an interface and primary database is often determined by the searcher’s accessibility. For our method, an interface that allows searching with proximity operators is desirable, and full functionality of the thesaurus, including explosion of narrower terms, is crucial. We recommend developing a personal workflow that always starts with one specific database and interface.

6. Document the search process in a text document

We advise designing and creating the complete search strategies in a log document, instead of directly in the database itself, to register the steps taken and to make searches accountable and reproducible. The developed search strategies can be copied and pasted into the desired databases from the log document. This way, the searcher is in control of the whole process. Any change to the search strategy should be done in the log document, assuring that the search strategy in the log is always the most recent.

7. Identify appropriate index terms in the thesaurus of the first database

Searches should start by identifying appropriate thesaurus terms for the desired elements. The thesaurus of the database is searched for matching index terms for each key concept. We advise restricting the initial terms to the most important and most relevant terms. Later in the process, more general terms can be added in the optimization process, in which the effect on the number of hits, and thus the desirability of adding these terms, can be evaluated more easily.

Several factors can complicate the identification of thesaurus terms. Sometimes, one thesaurus term is found that exactly describes a specific element. In contrast, especially in more general elements, multiple thesaurus terms can be found to describe one element. If no relevant thesaurus terms have been found for an element, free-text terms can be used, and possible thesaurus terms found in the resulting references can be added later (step 11).

Sometimes, no distinct thesaurus term is available for a specific key concept that describes the concept in enough detail. In Emtree, one thesaurus term often combines two or more elements. The easiest solution for combining these terms for a sensitive search is to use such a thesaurus term in all elements where it is relevant. Examples are given in the supplementary appendix .

8. Identify synonyms in the thesaurus

Most thesauri offer a list of synonyms on their term details page (named Synonyms in Emtree and Entry Terms in MeSH). To create a sensitive search strategy for SRs, these terms need to be searched as free-text keywords in the title and abstract fields, in addition to searching their associated thesaurus terms.

The Emtree thesaurus contains more synonyms (300,000) than MeSH does (220,000) [ 17 ]. The difference in number of terms is even higher considering that many synonyms in MeSH are permuted terms (i.e., inversions of phrases using commas).

Thesaurus terms are ordered in a tree structure. When searching for a more general thesaurus term, the more specific (narrower) terms in the branches below that term will also be searched (this is frequently referred to as “exploding” a thesaurus term). However, to perform a sensitive search, all relevant variations of the narrower terms must be searched as free-text keywords in the title or abstract, in addition to relying on the exploded thesaurus term. Thus, all articles that describe a certain narrower topic in their titles and abstracts will already be retrieved before MeSH terms are added.

9. Add variations in search terms (e.g., truncation, spelling differences, abbreviations, opposites)

Truncation allows a searcher to search for words beginning with the same word stem. A search for therap* will, thus, retrieve therapy, therapies, therapeutic, and all other words starting with “therap.” Do not truncate a word stem that is too short. Also, limitations of interfaces should be taken into account, especially in PubMed, where the number of search term variations that can be found by truncation is limited to 600.

Databases contain references to articles using both standard British and American English spellings. Both need to be searched as free-text terms in the title and abstract. Alternatively, many interfaces offer a certain code to replace zero or one characters, allowing a search for “pediatric” or “paediatric” as “p?ediatric.” Table 1 provides a detailed description of the syntax for different interfaces.

Field codes in five most used interfaces for biomedical literature searching

Searching for abbreviations can identify extra, relevant references and retrieve more irrelevant ones. The search can be more focused by combining the abbreviation with an important word that is relevant to its meaning or by using the Boolean “NOT” to exclude frequently observed, clearly irrelevant results. We advise that searchers do not exclude all possible irrelevant meanings, as it is very time consuming to identify all the variations, it will result in unnecessarily complicated search strategies, and it may lead to erroneously narrowing the search and, thereby, reduce recall.

Searching partial abbreviations can be useful for retrieving relevant references. For example, it is very likely that an article would mention osteoarthritis (OA) early in the abstract, replacing all further occurrences of osteoarthritis with OA . Therefore, it may not contain the phrase “hip osteoarthritis” but only “hip oa.”

It is also important to search for the opposites of search terms to avoid bias. When searching for “disease recurrence,” articles about “disease free” may be relevant as well. When the desired outcome is survival , articles about mortality may be relevant.

10. Use database-appropriate syntax, with parentheses, Boolean operators, and field codes

Different interfaces require different syntaxes, the special set of rules and symbols unique to each database that define how a correctly constructed search operates. Common syntax components include the use of parentheses and Boolean operators such as “AND,” “OR,” and “NOT,” which are available in all major interfaces. An overview of different syntaxes for four major interfaces for bibliographic medical databases (PubMed, Ovid, EBSCOhost, Embase.com, and ProQuest) is shown in Table 1 .

Creating the appropriate syntax for each database, in combination with the selected terms as described in steps 7–9, can be challenging. Following the method outlined below simplifies the process:

• Create single-line queries in a text document (not combining multiple record sets), which allows immediate checking of the relevance of retrieved references and efficient optimization.
• Type the syntax (Boolean operators, parentheses, and field codes) before adding terms, which reduces the chance that errors are made in the syntax, especially in the number of parentheses.
• Use predefined proximity structures including parentheses, such as (() ADJ3 ()) in Ovid, that can be reused in the query when necessary.
• Use thesaurus terms separately from free-text terms of each element. Start an element with all thesaurus terms (using “OR”) and follow with the free-text terms. This allows the unique optimization methods as described in step 11.
• When adding terms to an existing search strategy, pay close attention to the position of the cursor. Make sure to place it appropriately either in the thesaurus terms section, in the title/abstract section, or as an addition (broadening) to an existing proximity search.

The supplementary appendix explains the method of building a query in more detail, step by step for different interfaces: PubMed, Ovid, EBSCOhost, Embase.com, and ProQuest. This method results in a basic search strategy designed to retrieve some relevant references upon which a more thorough search strategy can be built with optimization such as described in step 11.

11. Optimize the search

The most important question when performing a systematic search is whether all (or most) potentially relevant articles have been retrieved by the search strategy. This is also the most difficult question to answer, since it is unknown which and how many articles are relevant. It is, therefore, wise first to broaden the initial search strategy, making the search more sensitive, and then check if new relevant articles are found by comparing the set results (i.e., search for Strategy #2 NOT Strategy #1 to see the unique results).

A search strategy should be tested for completeness. Therefore, it is necessary to identify extra, possibly relevant search terms and add them to the test search in an OR relationship with the already used search terms. A good place to start, and a well-known strategy, is scanning the top retrieved articles when sorted by relevance, looking for additional relevant synonyms that could be added to the search strategy.

We have developed a unique optimization method that has not been described before in the literature. This method often adds valuable extra terms to our search strategy and, therefore, extra, relevant references to our search results. Extra synonyms can be found in articles that have been assigned a certain set of thesaurus terms but that lack synonyms in the title and/or abstract that are already present in the current search strategy. Searching for thesaurus terms NOT free-text terms will help identify missed free-text terms in the title or abstract. Searching for free-text terms NOT thesaurus terms will help identify missed thesaurus terms. If this is done repeatedly for each element, leaving the rest of the query unchanged, this method will help add numerous relevant terms to the query. These steps are explained in detail for five different search platforms in the supplementary appendix .

12. Evaluate the initial results

The results should now contain relevant references. If the interface allows relevance ranking, use that in the evaluation. If you know some relevant references that should be included in the research, search for those references specifically; for example, combine a specific (first) author name with a page number and the publication year. Check whether those references are retrieved by the search. If the known relevant references are not retrieved by the search, adapt the search so that they are. If it is unclear which element should be adapted to retrieve a certain article, combine that article with each element separately.

Different outcomes are desired for different types of research questions. For instance, in the case of clinical question answering, the researcher will not be satisfied with many references that contain a lot of irrelevant references. A clinical search should be rather specific and is allowed to miss a relevant reference. In the case of an SR, the researchers do not want to miss any relevant reference and are willing to handle many irrelevant references to do so. The search for references to include in an SR should be very sensitive: no included reference should be missed. A search that is too specific or too sensitive for the intended goal can be adapted to become more sensitive or specific. Steps to increase sensitivity or specificity of a search strategy can be found in the supplementary appendix .

13. Check for errors

Errors might not be easily detected. Sometimes clues can be found in the number of results, either when the number of results is much higher or lower than expected or when many retrieved references are not relevant. However, the number expected is often unknown, and very sensitive search strategies will always retrieve many irrelevant articles. Each query should, therefore, be checked for errors.

One of the most frequently occurring errors is missing the Boolean operator “OR.” When no “OR” is added between two search terms, many interfaces automatically add an “AND,” which unintentionally reduces the number of results and likely misses relevant references. One good strategy to identify missing “OR”s is to go to the web page containing the full search strategy, as translated by the database, and using Ctrl-F search for “AND.” Check whether the occurrences of the “AND” operator are deliberate.

Ideally, search strategies should be checked by other information specialists [ 18 ]. The Peer Review of Electronic Search Strategies (PRESS) checklist offers good guidance for this process [ 4 ]. Apart from the syntax (especially Boolean operators and field codes) of the search strategy, it is wise to have the search terms checked by the clinician or researcher familiar with the topic. At Erasmus MC, researchers and clinicians are involved during the complete process of structuring and optimizing the search strategy. Each word is added after the combined decision of the searcher and the researcher, with the possibility of directly comparing results with and without the new term.

14. Translate to other databases

To retrieve as many relevant references as possible, one has to search multiple databases. Translation of complex and exhaustive queries between different databases can be very time consuming and cumbersome. The single-line search strategy approach detailed above allows quick translations using the find and replace method in Microsoft Word (<Ctrl-H>).

At Erasmus MC, macros based on the find-and-replace method in Microsoft Word have been developed for easy and fast translation between the most used databases for biomedical and health sciences questions. The schema that is followed for the translation between databases is shown in Figure 2 . Most databases simply follow the structure set by the Embase.com search strategy. The translation from Emtree terms to MeSH terms for MEDLINE in Ovid often identifies new terms that need to be added to the Embase.com search strategy before the translation to other databases.

Schematic representation of translation between databases used at Erasmus University Medical Center

Dotted lines represent databases that are used in less than 80% of the searches.

Using five different macros, a thoroughly optimized query in Embase.com can be relatively quickly translated into eight major databases. Basic search strategies will be created to use in many, mostly smaller, databases, because such niche databases often do not have extensive thesauri or advanced syntax options. Also, there is not much need to use extensive syntax because the number of hits and, therefore, the amount of noise in these databases is generally low. In MEDLINE (Ovid), PsycINFO (Ovid), and CINAHL (EBSCOhost), the thesaurus terms must be adapted manually, as each database has its own custom thesaurus. These macros and instructions for their installation, use, and adaptation are available at bit.ly/databasemacros.

15. Test and reiterate

Ideally, exhaustive search strategies should retrieve all references that are covered in a specific database. For SR search strategies, checking searches for their recall is advised. This can be done after included references have been determined by the authors of the systematic review. If additional papers have been identified through other non-database methods (i.e., checking references in included studies), results that were not identified by the database searches should be examined. If these results were available in the databases but not located by the search strategy, the search strategy should be adapted to try to retrieve these results, as they may contain terms that were omitted in the original search strategies. This may enable the identification of additional relevant results.

A methodology for creating exhaustive search strategies has been created that describes all steps of the search process, starting with a question and resulting in thorough search strategies in multiple databases. Many of the steps described are not new, but together, they form a strong method creating high-quality, robust searches in a relatively short time frame.

Our methodology is intended to create thoroughness for literature searches. The optimization method, as described in step 11, will identify missed synonyms or thesaurus terms, unlike any other method that largely depends on predetermined keywords and synonyms. Using this method results in a much quicker search process, compared to traditional methods, especially because of the easier translation between databases and interfaces (step 13). The method is not a guarantee for speed, since speed depends on many factors, including experience. However, by following the steps and using the tools as described above, searchers can gain confidence first and increase speed through practice.

What is new?

This method encourages searchers to start their search development process using empty syntax first and later adding the thesaurus terms and free-text synonyms. We feel this helps the searcher to focus on the search terms, instead of on the structure of the search query. The optimization method in which new terms are found in the already retrieved articles is used in some other institutes as well but has to our knowledge not been described in the literature. The macros to translate search strategies between interfaces are unique in this method.

What is different compared to common practice?

Traditionally, librarians and information specialists have focused on creating complex, multi-line (also called line-by-line) search strategies, consisting of multiple record sets, and this method is frequently advised in the literature and handbooks [ 2 , 19 – 21 ]. Our method, instead, uses single-line searches, which is critical to its success. Single-line search strategies can be easily adapted by adding or dropping a term without having to recode numbers of record sets, which would be necessary in multi-line searches. They can easily be saved in a text document and repeated by copying and pasting for search updates. Single-line search strategies also allow easy translation to other syntaxes using find-and-replace technology to update field codes and other syntax elements or using macros (step 13).

When constructing a search strategy, the searcher might experience that certain parentheses in the syntax are unnecessary, such as parentheses around all search terms in the title/abstract portion, if there is only one such term, there are double parentheses in the proximity statement, or one of the word groups exists for only one word. One might be tempted to omit those parentheses for ease of reading and management. However, during the optimization process, the searcher is likely to find extra synonyms that might consist of one word. To add those terms to the first query (with reduced parentheses) requires adding extra parentheses (meticulously placing and counting them), whereas, in the latter search, it only requires proper placement of those terms.

Many search methods highly depend on the PICO framework. Research states that often PICO or PICOS is not suitable for every question [ 22 , 23 ]. There are other acronyms than PICO—such as sample, phenomenon of interest, design, evaluation, research type (SPIDER) [ 24 ]—but each is just a variant. In our method, the most important and specific elements of a question are being analyzed for building the best search strategy.

Though it is generally recommended that searchers search both MEDLINE and Embase, most use MEDLINE as the starting point. It is considered the gold standard for biomedical searching, partially due to historical reasons, since it was the first of its kind, and more so now that it is freely available via the PubMed interface. Our method can be used with any database as a starting point, but we use Embase instead of MEDLINE or another database for a number of reasons. First, Embase provides both unique content and the complete content of MEDLINE. Therefore, searching Embase will be, by definition, more complete than searching MEDLINE only. Second, the number of terms in Emtree (the Embase thesaurus) is three times as high as that of MeSH (the MEDLINE thesaurus). It is easier to find MeSH terms after all relevant Emtree terms have been identified than to start with MeSH and translate to Emtree.

At Erasmus MC, the researchers sit next to the information specialist during most of the search strategy design process. This way, the researchers can deliver immediate feedback on the relevance of proposed search terms and retrieved references. The search team then combines knowledge about databases with knowledge about the research topic, which is an important condition to create the highest quality searches.

Limitations of the method

One disadvantage of single-line searches compared to multi-line search strategies is that errors are harder to recognize. However, with the methods for optimization as described (step 11), errors are recognized easily because missed synonyms and spelling errors will be identified during the process. Also problematic is that more parentheses are needed, making it more difficult for the searcher and others to assess the logic of the search strategy. However, as parentheses and field codes are typed before the search terms are added (step 10), errors in parentheses can be prevented.

Our methodology works best if used in an interface that allows proximity searching. It is recommended that searchers with access to an interface with proximity searching capabilities select one of those as the initial database to develop and optimize the search strategy. Because the PubMed interface does not allow proximity searches, phrases or Boolean “AND” combinations are required. Phrase searching complicates the process and is more specific, with the higher risk of missing relevant articles, and using Boolean “AND” combinations increases sensitivity but at an often high loss of specificity. Due to some searchers’ lack of access to expensive databases or interfaces, the freely available PubMed interface may be necessary to use, though it should never be the sole database used for an SR [ 2 , 16 , 25 ]. A limitation of our method is that it works best with subscription-based and licensed resources.

Another limitation is the customization of the macros to a specific institution’s resources. The macros for the translation between different database interfaces only work between the interfaces as described. To mitigate this, we recommend using the find-and-replace functionality of text editors like Microsoft Word to ease the translation of syntaxes between other databases. Depending on one’s institutional resources, custom macros can be developed using similar methods.

Results of the method

Whether this method results in exhaustive searches where no important article is missed is difficult to determine, because the number of relevant articles is unknown for any topic. A comparison of several parameters of 73 published reviews that were based on a search developed with this method to 258 reviews that acknowledged information specialists from other Dutch academic hospitals shows that the performance of the searches following our method is comparable to those performed in other institutes but that the time needed to develop the search strategies was much shorter than the time reported for the other reviews [ 9 ].

CONCLUSIONS

With the described method, searchers can gain confidence in their search strategies by finding many relevant words and creating exhaustive search strategies quickly. The approach can be used when performing SR searches or for other purposes such as answering clinical questions, with different expectations of the search’s precision and recall. This method, with practice, provides a stepwise approach that facilitates the search strategy development process from question clarification to final iteration and beyond.

SUPPLEMENTAL FILE

Acknowledgments.

We highly appreciate the work that was done by our former colleague Louis Volkers, who in his twenty years as an information specialist in Erasmus MC laid the basis for our method. We thank Professor Oscar Franco for reviewing earlier drafts of this article.

• DEGREES & PROGRAMS
• Programs & Pathways
• Courses & Catalogs
• Online Learning
• Academic Calendar
• Short-Term Training
• ADMISSIONS & AID
• Financial Aid
• Get Started
• Paying for College
• Michigan Reconnect
• SERVICES & SUPPORT
• Mentoring & Advising
• Career Center
• Registration & Records
• Library & Learning Services
• Student Wellness
• Technology Services
• Mid Foundation
• Facilities & Features
• Lifelong Learning
• Safety & Security
• Current Students
• Area Businesses
• Future Students
• K-12 Counselors
• Faculty & Staff

Academic vs Non-Academic Articles

• Find Articles
• Research Assistance

Academic vs. Non-Academic: What's the Difference?

The majority of your research will require academic and scholarly articles. Many students struggle with trying to determine what an academic source, or article, is.

Academic articles   are written by professionals in a given field. They are edited by the author's peers and often take years to publish. Their language is formal and will contain words and terms typical to the field. The author's name will be present, as will their credentials. There will be a list of references that indicate where the author obtained the information they are using in the article.

Academic articles can be found in periodicals similar to the Journal of Psychology, Childhood Education, or The American Journal of Public Health.

The following link is an example of an academic article.  Experimental educational networking on open research issues; Studying PSS applicability and development in emerging contexts .

This article is considered academic because the language is very formal and genre-specific, there are two authors and their credentials are listed (these are found at the end of the article), and most importantly there is a list of references.

Non-academic articles are written for the mass public. They are published quickly and can be written by anyone. Their language is informal, and casual and may contain slang. The author may not be provided and will not have any credentials listed. There will be no reference list. Non-academic articles can be found in periodicals similar to Time, Newsweek, or Rolling Stone.

As a general rule religious texts and newspapers are not considered academic sources. Do not use Wikipedia as an academic source. This website can be altered by anyone so any information found within its pages cannot be considered credible or academic.

The following link is an example of a non-academic article.  Marketing News's Writers Rules

This article is non-academic because the language is very casual and includes some examples of slang, there is an author, but they chose to write anonymously so there are no credentials provided for the author, and no references were included to show where the author obtained their information.

• YouTube Icon
• Facebook Icon
• Twitter Icon
• LinkedIn Icon
• Instagram Icon

The Scholarly Kitchen

What’s Hot and Cooking In Scholarly Publishing

The Latest “Crisis” — Is the Research Literature Overrun with ChatGPT- and LLM-generated Articles?

• Artificial Intelligence
• Metrics and Analytics
• Peer Review
• Research Integrity

Elsevier has been under the spotlight this month for publishing a paper that contains a clearly ChatGPT-written portion of its introduction. The first sentence of the paper’s Introduction reads, “Certainly, here is a possible introduction for your topic:…” To date, the article remains unchanged, and unretracted. A second paper , containing the phrase “I’m very sorry, but I don’t have access to real-time information or patient-specific data, as I am an AI language model” was subsequently found , and similarly remains unchanged. This has led to a spate of amateur bibliometricians scanning the literature for similar common AI-generated phrases, with some alarming results . But it’s worth digging a little deeper into these results to get a sense of whether this is indeed a widespread problem, and where such papers have made it through to publication, where the errors are occurring.

Several of the investigations into AI-pollution of the literature that I’ve seen employ Google Scholar for data collection (the link above, and another here ). But when you start looking at the Google Scholar search results, you notice that a lot of what’s listed, at least on the first few pages, are either preprints, items on ResearchGate, book chapters, or often something posted to a website you’ve never heard of with a Russian domain URL. The problem here is that Google Scholar is deliberately a largely non-gated index. It scans the internet for things that look like research papers (does it have an Abstract, does it have References), rather than limiting results to a carefully curated list of reputable publications. Basically, it grabs anything that looks “scholarly”. This is a feature, not a bug, and one of the important values that Google Scholar offers is that it can reach beyond the more limiting inclusion criteria (and often English language and Global North biased) content of indexes like the Web of Science.

But what happens when one does similar searches on a more curated database, one that is indeed limited to what most would consider a more accurate picture of the reputable scholarly literature? Here I’ve chosen Dimensions , an inter-linked research information system provided by Digital Science, as its content inclusion is broader than the Web of Science, but not as unlimited as Google Scholar. With the caveat that all bibliometrics indexes are lagging, and take some time to bring in the most recently published articles (the two Elsevier papers mentioned above are dated as being from March and June of 2024 and so aren’t yet indexed as far as I can tell), my results are perhaps less worrying. All searches below were limited to research articles (no preprints, book chapters, or meeting abstracts) published after November 2022, when ChatGPT was publicly released.

A search for “Certainly, here is” brings up a total of ten articles published over that time period. Of those ten articles, eight are about ChatGPT, so the inclusion of the phrase is likely not suspect. A search for “as of my last knowledge update” gives a total of six articles, again with four of those articles focused on ChatGPT itself. A search for “I don’t have access to real-time data” brings up only three articles, all of which cover ChatGPT or AI. During this same period, Dimensions lists nearly 5.7M research articles and review articles published, putting the error rate for these three phrases to slip through into publications at 0.00007%.

Retraction Watch has a larger list of 77 items (as of this writing), using a more comprehensive set of criteria to spot problematic, likely AI-generated text which includes journal articles from Elsevier, Springer Nature, MDPI, PLOS, Frontiers, Wiley, IEEE, and Sage. Again, this list needs further sorting, as it also includes some five book chapters, eleven preprints, and at least sixteen conference proceedings pieces. Removing these 32 items from the list suggests a failure rate of 0.00056%.

While many would argue that this does not constitute a “crisis”, it is likely that such errors will continue to rise, and frankly, there’s not really any excuse for allowing even a single paper with such an obvious tell to make it through to publication. While this has led many to question the peer review process at the journals where these failures occurred, it’s worth considering other points in the publication workflow where such errors might happen. As Lisa Hinchliffe recently pointed out , it’s possible these sections are being added at the revision stage or even post-acceptance. Peer reviewers and editors looking at a revision may only be looking at the specific sections where they requested changes, and may miss other additions an author has put into the new version of the article. Angela Cochran wrote about how this has been exploited by unscrupulous authors adding in hundreds of citations in order to juice their own metrics. Also possible, the LLM-generated language may have been added at the pageproof stage (whether deliberately or not). Most journals outsource typesetting to third party vendors, and how carefully a journal scrutinizes the final, typeset version of the paper varies widely. As always, time spent by human editorial staff is the most expensive part of the publishing process, so many journals assume their vendors have done their jobs, and don’t go over each paper with a fine toothed comb unless a problem is raised.

Two other important conclusions can be drawn from this uproar. The first is that despite preprints having been around for decades, those both within and adjacent to the research community clearly do not understand their nature and why they’re different from the peer reviewed literature, so more educational effort is needed. It should not be surprising to anyone that there are a lot of rough early drafts of papers or unpublishable manuscripts in SSRN (founded in 1994) or arXiv (launched in 1991). We’ve heard a lot of concern about journalists not being able to recognize that preprints aren’t peer reviewed, but maybe there’s as big a problem much closer to home. The second conclusion is that there seems to be a perception that appearing in Google Scholar search results offers some assurance of credibility or validation. This is absolutely not the case, and perhaps the fault here lies with t he lack of differentiation between the profile service offered by Google Scholar , which is personally curated by individuals and its search results which are far less discriminating.

Going forward, I would hope that at the journals where the small number of papers have slipped through, an audit is underway to better understand where the language was introduced and how it managed to get all the way to publication. Automated checks should be able to weed out common AI language like this, but they likely need to be run at multiple points in the publication process, rather than just on initial submissions. While the systems in place seem to be performing pretty well overall, there’s no room for complacency, and research integrity vigilance will only become more and more demanding.

David Crotty

David Crotty is a Senior Consultant at Clarke & Esposito, a boutique management consulting firm focused on strategic issues related to professional and academic publishing and information services. Previously, David was the Editorial Director, Journals Policy for Oxford University Press. He oversaw journal policy across OUP’s journals program, drove technological innovation, and served as an information officer. David acquired and managed a suite of research society-owned journals with OUP, and before that was the Executive Editor for Cold Spring Harbor Laboratory Press, where he created and edited new science books and journals, along with serving as a journal Editor-in-Chief. He has served on the Board of Directors for the STM Association, the Society for Scholarly Publishing and CHOR, Inc., as well as The AAP-PSP Executive Council. David received his PhD in Genetics from Columbia University and did developmental neuroscience research at Caltech before moving from the bench to publishing.

26 Thoughts on "The Latest “Crisis” — Is the Research Literature Overrun with ChatGPT- and LLM-generated Articles?"

There is a huge difference between an entire article being written by AI, as the title of this post suggests, and having a few paragraphs being written by AI, as the actual evidence in most of this post suggests. I think the most worrisome part of this is that the journal editors are doing such a poor job that they aren’t catching those obvious “certainly…” type phrases, not because of what they imply about authorship, but just because they don’t belong in the final text at all.

I find the debate about using AI and academic integrity is bringing out an inconsistency in our very reasons for opposing “plagiarism”. That is, is the problem that the ideas aren’t yours per se, or that you are “stealing” another person’s ideas? When the “other person” is not a human, suddenly this distinction is in sharp relief. If you aren’t stealing from someone else, is there still a bad thing happening here, or is this just a much more “humanities” version of using R or SPSS to do your quantitative analysis? Usually I find it frustrating when people mix up copyright law with plagiarism, but in this situation I think copyright law has something useful to inform the plagiarism discussion. Copyright law in the US at least is very clear that non-humans (eg monkeys, elephants, and computers) can’t be credited with authorship/creatorship.

• By Melissa Belvadi
• Mar 20, 2024, 7:59 AM
• Reply to Comment

“Also possible, the LLM-generated language may have been added at the pageproof stage…”. That seems a bit far fetched. More likely some manuscripts don’t get carefully reviewed. More interesting to me than including LLM text in low quality articles is the sophistication of AI generated cell biology images. They’re getting to the point that sleuths like Elisabeth Bik can’t tell them from the real thing.

• By Chris Mebane
• Mar 20, 2024, 8:55 AM

The authors of one of the suspect papers have stated that the inclusion of the text was a cut-and-paste error. Why would such an error be “far fetched” during corrections but less so in other parts of the publication process?

• By David Crotty
• Mar 20, 2024, 9:27 AM

This may be but what’s the excuse for the failure to include the required disclosure statement? Everyone is focusing on the sloppy editing. But, what this signals to me is that there’s probably a bigger crises of non-disclosure, which is a different issue and – given Elsevier doesn’t prohibit the use of generative AI for editorial assistance but does require disclosure – to me is the bigger ethical question. The disclosure should have been made even if their was no copy/paste sloppiness. So, how many authors are using AI and not disclosing – and not tipping their hand by sloppiness?

• By Lisa Janicke Hinchliffe
• Mar 21, 2024, 10:51 AM

In this case, I believe the authors claimed that they had asked ChatGPT for a summary, but found it lacking and decided not to use it. Then it got “accidentally” pasted into some version of the paper. If accurate and believable, then no disclosure was necessary (as they did not deliberately use an AI).

All that said, you do get to the heart of the issue, and why the fuss over a tiny number of papers found with obvious tells is less important than the bigger picture that many are less sloppy and probably using these tools. While I do think it’s reasonable to require disclosure (if publishing is part of the scientific process, then you should record the tools used as you do in the Materials and Methods section for your experiments), I personally don’t think the use of AI in writing is all that problematic. What I care about is whether the experiments/research were/was done correctly, described accurately, and that the conclusions drawn are supported by the data presented. In the end, I don’t really care who (or what) wrote the story about the research, I’m interested in the research and ensuring it’s valid. We know that ghostwriting is fairly rampant in the medical world, and that industry research companies often have publication planners and writers on staff who put together publications. Is this any different ethically, as long as the author who puts their name on the paper vouches for its contents? Is that what matters (I’m staking my reputation on what’s in this document) rather than who/what chose the actual words and put them in that order?

Of course this is a different matter from fraudulent papers where the research wasn’t actually done, and yes, AI does simplify the process of creating fraudulent work, but that’s a separate issue. It does strike me as interesting that the two papers linked to above had AI text in an Introduction and an Abstract, rather than in the Results or the Conclusion sections of the papers.

• Mar 21, 2024, 11:04 AM

I’ve not seen any author statement on either of these examples. I’ve seen one person say they are reporting what one author told him — but that didn’t include that the AI text was found wanting. I did see that the same intermediary say one of the authors says they had emailed Elsevier about this issue post-publication. Everyone decrying how long this is taking to fix and none of the authors just getting the right version out there by posting it up on a preprint server. I mean, for goodness say, it’s not like we need to wait for libraries to tip in a new page of the printed journal. Any way, somehow the authors also turned back page proofs without noticing the inclusion either. So, while I like you am not too concerned about how words get generated in this context (I have a statement in my syllabus this semester that students can use gen AI if they disclosure and explain why they decided it was the best choice), it matters if authors don’t take responsibility for their work and don’t make required disclosures. What other corners are they cutting?

• Mar 21, 2024, 11:14 AM

I remember seeing something from the authors on Twitter, will try to dig it out. Regardless, I agree 100%. When you publish a paper, you are building/risking your reputation and your career. From my researcher days, the notion that I wouldn’t have carefully pored over every single character of every single draft of every paper I published escapes me. Those authors will now always be known as the ChatGPT people, I guess better than those permanently known as the gigantic rat genitals people, but still. If you are so sloppy in your writing, why wouldn’t I expect you to be similarly sloppy in your experiments? What did you accidentally “cut and paste” there?

• Mar 21, 2024, 11:18 AM

In case anyone else is following this convo in the comments, here’s the thread not from the authors I referenced: https://twitter.com/GuyCurtis10/status/1768786985253319040

• Mar 21, 2024, 11:41 AM

This is an interesting discussion; thanks for advancing it. Given the challenges facing scientific and medical publishing, which is a bigger crisis? 1) That overworked, underpaid, stressed, researchers and academics who don’t directly benefit financially from their content are now leveraging AI to speed time to publication, or 2) That the majority of entrenched publishers continue to act as if human friction and delays in getting novel science and groundbreaking medicine to patients and doctors who are trying to improve human suffering, which could be improved significantly with AI, is itself a problem to be dealt with? Full disclosure, I serve as Chief AI Officer of Inizio Medical and am a Founding Board Member of the Society for Artificial Intelligence and Health, but my comment is mine and mine alone.

• By Matt Lewis
• Mar 20, 2024, 9:52 AM

It is alarming that this slipped through after all the work done to tighten up publishing integrity over the last year or two. It suggests more sophisticated fake/fraudulent AI content is also getting through. More generally, it is another embarrassing, public black eye for scholarly publishing and science.

• By Curtis Brundy
• Mar 20, 2024, 10:42 AM

It’s clear from this post that the crisis is not about generative AI, which simply reveals the underlying problem. The real issue is the faulty editorial workflow, which Angela Cochran identified as far back as 2017. The human editor should have the last check of the paper. If you allow authors to make changes, then those changes should be monitored by the editor, and a system such as track changes applied to ensure that any changes the author makes, whether or not requested by the editor, are identifiable for checking.

What this reveals is that scholarly publishing still retains a culture of trust, in this case, trust that the author will not make extensive changes at proof stage. It is no longer possible to run scholarly publishing with the assumption that the author will behave responsibly.

• By Michael Upshall
• Mar 20, 2024, 10:57 AM

I think you’re largely right, but those needs are directly in opposition to the increasing pressure for journals to 1) publish faster, and 2) publish cheaper. As noted in the post, the sorts of human interventions you suggest are the most expensive parts of the process. Would the community accept slower publication and higher APCs/subscription prices in order to ensure this level of scrutiny?

• Mar 20, 2024, 11:00 AM

Well, that’s for the scholarly community to decide, but personally, I don’t think there is a choice, to preserve the credibility of academic publishing.

• Mar 20, 2024, 11:10 AM

What Automated Checks systems do you recommend be used by journals?

• By Deb Whippen
• Mar 20, 2024, 11:24 AM

“Automated” is a loaded word here — every check that I know of requires at least some level of human interpretation and intervention.

That said, there are good recommendations coming out of the STM Research Integrity Hub ( https://www.stm-assoc.org/stm-integrity-hub/ ) and tools include plagiarism checkers, image integrity checkers, and paper mill checkers. As far as I know, there are no reliable automated tools for determining whether text or images were AI-generated. But screening for phrases like the ones mentioned above seems a reasonable step.

And as noted, this makes publishing slower and more expensive.

• Mar 20, 2024, 11:47 AM

I think that screening for phrases, as you say, could work. A mechanism that detects hallucinated content, including fake references/citations, could also help. But then, AI will also evolve…

• By Ron Martinez
• Mar 21, 2024, 5:03 PM

As the EIC of a journal, I read every manuscript that is submitted. Lately, I have detected a handful of manuscripts that have the hallmarks of being generated (all or in part) by an LLM. It is fairly obvious (to me) to identify such manuscripts, because the text reads like “word salad”, that is, bland, general sentences that do not seem to converge on a clear meaning and lack specific details. A superficial reading is not good enough in such cases. As pointed out in the post, having actual humans read the manuscripts or proofs is necessary, but expensive.

• By Constance Senior
• Mar 20, 2024, 2:15 PM

Same here. I had one the other day that was not really in the scope of the journal, had that “word salad” feel you mentioned, and had only 5 total references. (And two of those were fake.)

• Mar 21, 2024, 5:01 PM

Interesting, another question is what percent of peer review reports/peer reviewers are using AI, and is incorrect use of AI in this area able to be detected, I heard this is potentially a growing, and yet somewhat hidden problem as well.

• By Adrian Stanley
• Mar 20, 2024, 4:29 PM

This preprint might be of interest – https://arxiv.org/abs/2403.07183 , “ Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews”, https://doi.org/10.48550/arXiv.2403.07183

• By Dugald McGlashan
• Mar 21, 2024, 2:53 AM

“time spent by human editorial staff is the most expensive part of the publishing process”

It turns out more and more that the most *expensive* part of the publishing process is editorial and other negligence actively encouraged by those who never tire of finding more “cost-effective” and “streamlined/frictionless” ways to “improve the author experience.” It’s a sure way to steer scholarly publishing towards obsolescence because however happy authors (or their administrators) might be for having their work published, fewer and fewer people will be inclined or able to read this work (and AI reading AI-generated or “enhanced” texts is like an empty house of mirrors).

The cost of not having enough human eyes (i.e., editorial staff and peer reviewers) will be (and already is) exorbitant for both science and the public. High-quality, sustainable science and scholarship is slow and tedious and does not respond to exhortations for speed and efficiency. It’s a non-negotiable feature, not a bug.

• Mar 20, 2024, 4:44 PM

The one thing that is absolutely certain is that no copyeditors have been retained for the journals where these articles have slipped through. No proper copyeditor — one who edits for grammar, spelling, clarity, logic, and flow– is going to allow these kinds of things to pass through.

The big platforms do not do proper copyediting; the provide spelling and grammar checking from international outsourcing firms (packagers) and pay them literally one quarter of what I am paid, or they use their production team (layout, not copyeditors) to do the copyedits.

Copyeditors know that the vast majority of the time, we are the first ones to read a manuscript all the way through, after R&R or when there is no peer review. These instances prove that when we aren’t being hired, no one is reading a manuscript from start to finish.

• By Rudy Leon
• Mar 20, 2024, 6:20 PM

Importantly, the stance of MDPI is relaxed and predictable: https://www.mdpi.com/2072-6651/15/6/399

Also, I notice a certain trend of affiliates of a certain ’boutique’ subtly promoting Dimensions

• By Ivan Sterligov
• Mar 21, 2024, 2:44 AM

Do you feel that Dimensions is an inappropriate tool to use for these purposes? Which database would have been better?

• Mar 21, 2024, 7:14 AM

Thanks David, you brought some level-headedness to the conversation. Your deep dive into the Digital Science data has me feeling much better about this issue being overblown than I would have originally thought. Also, you make a good point about trying to understand why and how these things happen. An author using GPT to help them with language and accidentally inserting a GPT-phrase is very different than AI writing an article for them.

• By Avi Staiman
• Mar 21, 2024, 9:27 AM

The main question that needs to be answered: Is whether the papers passed through peer-reviewed or not. If so, what about their comments.

• By Raymond Adegboyega R
• Mar 31, 2024, 4:23 PM

Leave a Comment Cancel reply

Notify me of follow-up comments by email.

Related Articles:

Next Article:

The problem with making all academic research free

A new funding model for journals could deprive the world of valuable research in the humanities and social sciences..

There has been an earthquake in my corner of academia that will affect who teaches in prestigious universities and what ideas circulate among educated people around the world.

And it all happened because a concept rooted in good intentions — that academic research should be “open access,” free for everyone to read — has started to go too far.

The premise of open-access publishing is simple and attractive. It can cost libraries thousands of dollars a year to subscribe to academic journals, which sometimes means only academics affiliated with wealthy colleges and universities may access that research. But under open-access publishing, nearly anyone with an internet connection can find and read those articles for free. Authors win, because they find more readers. Academics around the world benefit, because they can access the latest scholarship. And the world wins, because scientific and intellectual progress is facilitated by the free exchange of ideas.

By now this model has taken hold in the natural sciences, especially in biology and biomedicine; during the pandemic many publishers removed paywalls from articles about vaccines and treatments. The Biden administration requires federally funded scholarly publications to be made freely available without any delay.

Advertisement

However, there is no such thing as a free academic article. Even with digital distribution, the expenses of running a journal are considerable. These costs include hosting the websites where people submit, peer-review, and edit articles; copyediting; advertising; preserving journal archives; and maintaining continuity as editors come and go.

As a result, unless journals have a source of revenue other than subscription fees, any move toward open access raises the question of who will cover the costs of publication.

One answer is that the money will come from authors themselves or their academic institutions or other backers. This works well enough in the natural sciences, because those researchers are often funded by grants, and some of that money can be set aside to cover a journal’s fees for publishing scientific articles. The Bill and Melinda Gates Foundation demands that all research funded by the foundation, including the underlying data, be published open access.

According to a paper published in Quantitative Science Studies , however, only a small fraction of scholars in the humanities publish their articles on an open-access basis. Unlike biologists and biomedical engineers, humanities scholars such as philosophers and historians do not get grants that can cover the publishing costs.

This means that if open access is to take hold in those fields as well — as many publishers and academics are advocating — the costs will have to be covered by some foundation or other sponsor, by the scholars’ institutions, or even by the scholars themselves. And all these models have serious downsides.

I’m a political philosopher. The earthquake in my field that I mentioned earlier shook one of our most prominent journals: the Journal of Political Philosophy.

Publishing an article in this journal has long made the difference between whether a candidate gets hired, tenured, or promoted at an elite institution of higher education. The high quality has stemmed in large part from the rigorous approach of the founding editor, Robert Goodin.

At the end of 2023, the publisher, Wiley, terminated its contract with Goodin. The reasons were not immediately clear, and over 1,000 academics, including me, signed a petition stating that we would not serve on the editorial board or write or review for the journal until Wiley reinstates Goodin. I recently attended a panel at an American Philosophical Association conference where philosophers voiced their anger and puzzlement about the situation.

One source of the problem appears to be that Wiley now charges the authors of an article or their institutions $3,840 to get published open access in the journal.

The Journal of Political Philosophy is actually hybrid open access, which means it waives the article processing charges for authors who permit their work to appear behind a subscription-only paywall. Nonetheless, Goodin and Anna Stilz , a Princeton professor and Journal of Political Philosophy editorial board member, point out that publishers like Wiley now have a strong incentive to favor open-access articles.

In the old model, in which university libraries subscribed to journals, editors were mainly incentivized to publish first-rate material that would increase subscriptions. In the open-access model, however, now that authors or their universities must cover the costs of processing articles, publishers of humanities journals seem to be incentivized to boost revenue by accepting as many articles as possible. According to Goodin , open access has “been the death knell of quality academic publishing.” The reason that Goodin lost his job, Goodin and Stilz imply, is that Wiley pressured Goodin to accept more articles to increase Wiley’s profits, and he said no. (Wiley representatives say that lines of communication had collapsed with Goodin.)

Early this year, Goodin cofounded a new journal titled simply Political Philosophy . The journal will be published by the Open Library of Humanities, which is subsidized by libraries and institutions around the world. But this version of open-access publishing does not have the financial stability of the old subscription model. Scholars affiliated with the Open Library of Humanities have pointed out that the project has substantial overhead costs, and it relied on a grant from the Andrew W. Mellon Foundation that has already ended. The Open Library of Humanities is an experiment, and I hope that it works, but as of now it publishes only 30 journals , compared with the 1,600 journals that Wiley publishes.

The fact remains that no one has satisfactorily explained how open access could work in the humanities and social sciences.

In his 2023 book “ Athena Unbound : Why and How Scholarly Knowledge Should Be Free for All,” UCLA history professor Peter Baldwin attempts an answer. He points to Latin America, where some national governments cover all expenses of academic publishing. But this proposal ignores the fact that the governments of the United States and other nations probably do not want to pay for humanities and social sciences journals.

Baldwin also floats the idea of preprint depositories where academics could share documents on the cloud before they have undergone the (somewhat expensive) process of peer review. But this means that academics would lose the benefits that come from getting double-blind feedback from one’s peers. This idea would reduce the costs of publishing a journal article, but it would turn much academic writing into fancy blogging.

Ultimately, Baldwin’s solution is that authors might “have to participate directly, giving them skin in the game and helping contain costs.” This means academics might ask their employers to pay the article processing charges, ask a journal for the processing fees to be waived, or dig into their own pockets to pay to publish.

And it might mean less gets published overall. The journal Government and Opposition, published by Cambridge University Press, is entirely open access and charges $3,450 for an article to be published. I’d have to apply for a discount or a waiver to publish there. Or I could do what political philosophers in Japan and Bosnia and Herzegovina have told me they do: avoid submitting to open-access journals. Their universities will not cover their article processing charges except maybe in the top journals, and even the reduced fees can run into hundreds of dollars that these professors do not have.

In “Athena Unbound,” Baldwin notes that Harvard subscribes to 10 times as many periodicals as India’s Institute of Science. One can bemoan this fact, but one may also appreciate that Harvard’s largesse spreads enough subscription revenue around to reputable journals to enable academics to avoid paying to publish in them, no matter whether they teach at regional state schools, non-elite private schools, or institutions of higher education in poor countries. For all its flaws, the old model meant that when rich alumni donated to their alma maters, it increased library budgets and thereby made it possible for scholars of poetry and state politics to run and publish in academic journals.

Until we have more evidence that open-access journals in the humanities and social sciences can thrive in the long run, academics need to appreciate the advantages of the subscription model.

This article was updated on March 28 to correct the reference to the paper published in Quantitative Science Studies.

Nicholas Tampio is a professor of political science at Fordham University in New York City.

Globe Ideas